Compare commits
72 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9785cda39d | |||
| 0cea99cd5e | |||
| 71bc754feb | |||
| cabbfed730 | |||
| 8b6ef7778a | |||
| 06319655b0 | |||
| f636169783 | |||
| 0f8f38ff52 | |||
| 142998afb3 | |||
| 665d7d2bb6 | |||
| 1746d47412 | |||
| 9bb2f7bcae | |||
| 1f2720463b | |||
| 8bb530e9b4 | |||
| 72df965a9c | |||
| f688325170 | |||
| c711213dcc | |||
| 11ecdccd65 | |||
| 27ab84d198 | |||
| 28e4da3879 | |||
| 31c0a83e51 | |||
| 34fddc3bf5 | |||
| 13c3dac554 | |||
| f7ed053e8e | |||
| 2e8feb56dd | |||
| 793178b345 | |||
| ee0359efd5 | |||
| 1dc57a85c2 | |||
| e7e8960fec | |||
| 79e278064d | |||
| 474b894d1c | |||
| 0c76b74afe | |||
| b608ed2c50 | |||
| f9d4e6bc74 | |||
| 223f11b989 | |||
| 5095be0b0d | |||
| 2aa64cf0a4 | |||
| 90c266d68a | |||
| 8410b2f4ef | |||
| ba69a5ad20 | |||
| 07b9b20eb5 | |||
| 901a0515b7 | |||
| 50c01a19c9 | |||
| 3a8fa87acb | |||
| df1dbbb8fb | |||
| 531ca851c4 | |||
| b3fd74529d | |||
| 00b376d234 | |||
| fa720d0054 | |||
| 91438e8214 | |||
| 08e9a34183 | |||
| 85937b8cbb | |||
| 107986bf85 | |||
| a8cabbebd5 | |||
| 636a0c5f43 | |||
| bf46782db3 | |||
| 02919e959e | |||
| d231eba8be | |||
| 637382f364 | |||
| bf6ef5b426 | |||
| 822e8fcd86 | |||
| 63e1a866a3 | |||
| 22c88c449d | |||
| ed6c0ac10c | |||
| 355c5d1e9a | |||
| 61d8b1dee2 | |||
| 0db32a865c | |||
| a4061ee1e1 | |||
| ec64be340e | |||
| f1c383c392 | |||
| 984c327162 | |||
| e2689d1b04 |
40
CHANGELOG.md
40
CHANGELOG.md
@@ -1,8 +1,46 @@
|
|||||||
<!-- file: CHANGELOG.md -->
|
<!-- file: CHANGELOG.md -->
|
||||||
<!-- version: 30 -->
|
<!-- version: 33 -->
|
||||||
|
|
||||||
# Changelog KSP
|
# Changelog KSP
|
||||||
|
|
||||||
|
## 0.3.13 — convergence live multi-source du Worker RawTransaction — 2026-09-10
|
||||||
|
|
||||||
|
`0.3.13` généralise `ksp-worker-raw-transaction-ingest-lib` de la verticale Yellowstone de `0.3.12` vers une composition caller-owned de `1..32` sources logiques appartenant au même réseau. Cinq familles live convergent désormais vers le même pipeline Common RAW / admission / Store : Yellowstone, Standard Logs avec hydration `getTransaction observed`, Standard Block RAW-direct, Helius `transactionSubscribe` avec hydration et HTTP Block Polling run-local RAW-direct. Le Worker reste sans Config, Job Backfill, backend Store physique, client réseau direct ou SDK provider parallèle.
|
||||||
|
|
||||||
|
La convergence partage un registre global d'hydration entre Yellowstone, Standard Logs et Helius afin de coalescer `(network, signature, commitment)` avant fan-out HTTP. Après canonicalisation, les acquisitions de même `(network, signature)` sont sérialisées de manière bornée : un contenu identique conserve les observations/provenances distinctes, tandis qu'une divergence devient un `content_conflict` terminal explicite. Aucune majorité provider, préférence silencieuse ou stratégie `first-provider-wins` n'est introduite. Standard Block et HTTP Block Polling admettent directement les transactions Legacy/V0/V1 qualifiées avec `maxSupportedTransactionVersion = 1`.
|
||||||
|
|
||||||
|
Le supervisor multi-source possède toutes les tâches et applique des bornes source-neutral sur admission, pending signals, hydrations et persistence avec une fairness minimale entre sources reference-bearing. Les snapshots agrègent activity/health/backpressure/reconnect/replay/gap sans exposer le matériau provider. La politique reste conservative : une source configurée qui échoue est terminale faute d'équivalence de coverage prouvée ; le gap repair, les rôles degraded/failover et la preuve de couverture restent donc `0.3.14`. Le shutdown stop/fault/drain/abort+join est durci contre les races, leaders d'hydration abandonnés, Store lent et publications tardives.
|
||||||
|
|
||||||
|
Le gate déterministe final passe `cargo fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy workspace/all-targets/all-features avec `-D warnings`, `cargo test --workspace --all-targets --all-features` et les graphes Cargo : `1 850` tests passent, `0` échoue et `15` restent ignored opt-in/operator-only. Les smokes keyless Solana HTTP Devnet et WebSocket Devnet passent `1/1` chacun. Yellowstone provider-gated, Helius `transactionSubscribe` live, `blockSubscribe` provider dédié, HTTP polling Worker end-to-end et Worker multi-source vers Store restent explicitement non exécutés faute de gate provisionné ; aucune fixture ne les requalifie en PASS.
|
||||||
|
|
||||||
|
`prompts/033-V0_3_14_START_PROMPT.md` ouvre `0.3.14` exclusivement depuis le futur tag stable `v0.3.13`. Cette release doit fermer le gap repair/hardening multi-source limité au run actif — replay adressable, coverage redondante, scan HTTP borné, hydration de réparation, unresolved gaps et health policy — sans transformer le Worker en moteur historique ni créer de dépendance vers `ksp-job-backfill-lib`.
|
||||||
|
|
||||||
|
## 0.3.12 — Yellowstone + hydration HTTP + continuité de run du Worker RawTransaction — 2026-09-09
|
||||||
|
|
||||||
|
`0.3.12` ouvre la première source réseau productive de `ksp-worker-raw-transaction-ingest-lib` sans remettre en cause la fondation source-neutral de `0.3.11`. Le Worker dépend désormais de la façade `ksp-onchain-transport-lib` mais reste sans Config, Job Backfill, backend Store physique, `reqwest`, `tonic` ou proto provider direct. `RawTransactionIngestRuntimeResources` reçoit une source Yellowstone caller-composed et un pool HTTP cohérent ; `start` conserve la fondation sans source, tandis que `start_with_runtime_resources` lance la verticale live supervisée.
|
||||||
|
|
||||||
|
Les updates Yellowstone `Transaction`, `TransactionStatus` et les transactions incluses dans `Block` sont projetées vers des signaux source-neutral, coalescées de manière bornée par `(network, signature, commitment)`, hydratées via `getTransaction` observed puis converties par `ksp-raw-transaction-lib` avant admission/persistence Store. `BlockMeta` et `Slot` restent continuity-only. La provenance composite conserve les identités logiques sûres de la source et de la route d’hydration, sans URL, secret ni payload distant. Les gardes réseau/signature/slot/index/commitment empêchent les croisements de matériau avant admission.
|
||||||
|
|
||||||
|
La release matérialise une processing frontier strictement run-local : le Worker distingue travail pending/settled, n’avance jamais au travers d’un pending plus ancien et compacte ses états sans revendiquer une preuve de complétude durable. Le reconnect/replay reste Transport-owned : `from_slot` et `SubscribeReplayInfo` ne sont pas pilotés par le Worker. Le Worker projette un état source-neutral `Active/Reconnecting/Closing/Closed/Failed`, compte reconnect/replay-attempt/continuity-gap et fault avec `source_failed` lorsqu’une hausse du gap Transport prouve que la rétention ne permet plus la reprise demandée. Aucun Backfill ou repair automatique n’est déclenché.
|
||||||
|
|
||||||
|
Le hardening final ferme les duplicate storms, saturation/coalescence, Store lent/fautif, stop/fault pendant hydration/reconnect, overflow de compteurs, abandon des pending sans faux `settled`, abort/join des tâches privées et absence de retry HTTP possédé par le Worker. `pre.010` ajoute les canaris cross-layer Transport -> Worker -> Common RAW -> Store et verrouille les dependency firewalls, fixtures Legacy/V0, redaction et inventaires publics.
|
||||||
|
|
||||||
|
Le gate déterministe `pre.011` passe rustfmt check, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, `cargo test --workspace --all-targets --all-features`, les suites ciblées et les trois inspections Cargo demandées. Les smokes live keyless HTTP Devnet et WebSocket Devnet sont exécutés séparément et passent ; les smokes Yellowstone token-gated et le Worker end-to-end Yellowstone + HTTP + Store restent explicitement non exécutés faute de ressources fournies, sans faux PASS. `pre.012` réconcilie ensuite README/USAGE et les architectures concernées avec cette verticale réellement matérialisée.
|
||||||
|
|
||||||
|
La suite reste découpée : `0.3.13` ajoute WS standard, Helius `transactionSubscribe`, HTTP live polling et la convergence multi-source ; `0.3.14` ferme le gap repair/hardening multi-source. `prompts/032-V0_3_13_START_PROMPT.md` ouvre `0.3.13` uniquement depuis le tag stable `v0.3.12` et impose un `pre.001` d’audit/sizing avant toute extension productive.
|
||||||
|
|
||||||
|
## 0.3.11 — Fondation runtime source-neutral du Worker RawTransaction ingest — 2026-09-08
|
||||||
|
|
||||||
|
`0.3.11` introduit `ksp-worker-raw-transaction-ingest-lib` comme premier Worker concret KSP pour l’acquisition continue de `RawTransaction`, tout en fermant volontairement la release **sans source réseau productive**. La crate consomme `ksp-worker-api`, `ksp-raw-transaction-lib` et la seule façade `ksp-store-lib`; elle ne dépend ni de Config, ni d’un Job, ni d’un backend Store physique, ni encore de `ksp-onchain-transport-lib`. Son identité stable est `raw_transaction_ingest`, ses settings bornent la capacité d’admission, la concurrence de persistence et la deadline de drain, et son `start` utilise exclusivement le runtime Tokio actif fourni par le caller.
|
||||||
|
|
||||||
|
Le runtime possède un supervisor privé, des tâches source/persistence détenues, une admission centrale `mpsc` bornée avec backpressure, la canonicalisation/assembly Common RAW, une observation key déterministe et la persistence atomique en mode `Normal`. Les outcomes distinguent insert, idempotence et tombstone purgée ; un contenu divergent devient `worker_raw_transaction_ingest.content_conflict`, les erreurs Store restent classifiées sans texte distant, et aucun provider gagnant n’est choisi implicitement. Les snapshots concrets latest-value se projettent directement sur `WorkerSnapshotSource`, utilisent séquences/compteurs checked et conservent le terminal pour les lecteurs tardifs.
|
||||||
|
|
||||||
|
Le hardening final ferme les races stop/fault, les source failures, la saturation observable, le Store lent/fautif et le shutdown borné. À l’expiration de `shutdown_drain_timeout`, le Worker ferme les admissions, abort les tâches restantes, les rejoint toutes puis publie `drain_timeout`; aucune tâche privée ne survit au terminal. Les tests externes de hardening/release verrouillent les sept codes d’erreur publics, les huit modules de production, les vingt-quatre exports crate-root, la redaction, le dependency firewall et l’absence de surface historique/Backfill/Transport/Config/backend direct.
|
||||||
|
|
||||||
|
Le gate technique final `pre.011` passe rustfmt check, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 56 tests Worker, `cargo test --workspace --all-targets --all-features`, les arbres Cargo Worker normal/features et `cargo tree --duplicates`. La réconciliation `pre.012` ajoute ensuite les README/USAGE durables du Worker et réaligne l’architecture sur l’état matériel : la fondation `0.3.11` n’a aucun adapter HTTP/WS/Yellowstone productif, aucune API publique d’enqueue/source registration et aucun replay/gap-repair live.
|
||||||
|
|
||||||
|
La trajectoire live reste découpée : `0.3.12` ouvre Yellowstone transactions/blocks/status + hydration HTTP + continuité de run, `0.3.13` ajoute WS standard/Helius/HTTP live et la convergence multi-source, puis `0.3.14` ferme le gap repair/hardening multi-source. `prompts/031-V0_3_12_START_PROMPT.md` ouvre `0.3.12` uniquement depuis le tag stable `v0.3.11`; son `pre.001` doit réauditer les surfaces Yellowstone/Solana/provider actuelles, l’injection Transport, l’hydration, le frontier, `from_slot`/ReplayInfo, les smokes accessibles et le sizing avant toute implémentation lourde.
|
||||||
|
|
||||||
## 0.3.10 — Lower-layer RAW Transaction commune + qualification cross-source — 2026-09-08
|
## 0.3.10 — Lower-layer RAW Transaction commune + qualification cross-source — 2026-09-08
|
||||||
|
|
||||||
`0.3.10` introduit `ksp-raw-transaction-lib` comme lower-layer source-neutral commune aux producteurs `RawTransaction`. La canonicalisation RAW v1 auparavant locale à `ksp-job-backfill-lib` est extraite sans modifier les golden bytes/hash existants ; le Backfill consomme désormais cette common crate sans créer de dépendance vers un Worker. La bibliothèque construit les modèles `RawTransaction`/`RawTransactionObservation` de `ksp-store-api`, conserve l'identité canonique `(network, signature)`, borne et redacted les entrées hostiles, et reste volontairement sans Transport, Config, runtime async, Job, Worker ou backend Store physique.
|
`0.3.10` introduit `ksp-raw-transaction-lib` comme lower-layer source-neutral commune aux producteurs `RawTransaction`. La canonicalisation RAW v1 auparavant locale à `ksp-job-backfill-lib` est extraite sans modifier les golden bytes/hash existants ; le Backfill consomme désormais cette common crate sans créer de dépendance vers un Worker. La bibliothèque construit les modèles `RawTransaction`/`RawTransactionObservation` de `ksp-store-api`, conserve l'identité canonique `(network, signature)`, borne et redacted les entrées hostiles, et reste volontairement sans Transport, Config, runtime async, Job, Worker ou backend Store physique.
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
# file: Cargo.toml
|
# file: Cargo.toml
|
||||||
# version: 505
|
# version: 565
|
||||||
|
|
||||||
[workspace]
|
[workspace]
|
||||||
resolver = "3"
|
resolver = "3"
|
||||||
members = ["crates/ksp-app-backfill-desk", "crates/ksp-app-config-desk", "crates/ksp-app-solprices-desk", "crates/ksp-app-store-desk", "crates/ksp-app-wallet-desk", "crates/ksp-config-lib", "crates/ksp-core-lib", "crates/ksp-interface-lib", "crates/ksp-job-api", "crates/ksp-job-backfill-lib", "crates/ksp-logging-lib", "crates/ksp-offchain-transport-lib", "crates/ksp-onchain-transport-lib", "crates/ksp-program-api", "crates/ksp-raw-transaction-lib", "crates/ksp-store-api", "crates/ksp-store-lib", "crates/ksp-store-postgres-lib", "crates/ksp-wallet-lib", "crates/ksp-worker-api"]
|
members = ["crates/ksp-app-backfill-desk", "crates/ksp-app-config-desk", "crates/ksp-app-solprices-desk", "crates/ksp-app-store-desk", "crates/ksp-app-wallet-desk", "crates/ksp-config-lib", "crates/ksp-core-lib", "crates/ksp-interface-lib", "crates/ksp-job-api", "crates/ksp-job-backfill-lib", "crates/ksp-logging-lib", "crates/ksp-offchain-transport-lib", "crates/ksp-onchain-transport-lib", "crates/ksp-program-api", "crates/ksp-raw-transaction-lib", "crates/ksp-store-api", "crates/ksp-store-lib", "crates/ksp-store-postgres-lib", "crates/ksp-wallet-lib", "crates/ksp-worker-api", "crates/ksp-worker-raw-transaction-ingest-lib"]
|
||||||
|
|
||||||
[workspace.package]
|
[workspace.package]
|
||||||
version = "0.3.10"
|
version = "0.3.13"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
license = "MIT"
|
license = "MIT"
|
||||||
repository = "https://git.sasedev.com/Sasedev/khadhroony-solana-project"
|
repository = "https://git.sasedev.com/Sasedev/khadhroony-solana-project"
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: README.md -->
|
<!-- file: README.md -->
|
||||||
<!-- version: 11 -->
|
<!-- version: 14 -->
|
||||||
|
|
||||||
# Khadhroony Solana Project
|
# Khadhroony Solana Project
|
||||||
|
|
||||||
@@ -57,7 +57,7 @@ La couche RAW dispose d'une façade Store backend-neutral et d'un premier job hi
|
|||||||
|
|
||||||
`ksp-worker-api` fournit désormais la fondation générique des services continus : identité Worker, lifecycle borné, health/activity, intention de stop coopératif et observation latest-value. Cette API reste Core-only, runtime-neutral et sans connaissance Solana, Transport, Store ou Job. Elle ne démarre ni n'arrête elle-même un runtime concret.
|
`ksp-worker-api` fournit désormais la fondation générique des services continus : identité Worker, lifecycle borné, health/activity, intention de stop coopératif et observation latest-value. Cette API reste Core-only, runtime-neutral et sans connaissance Solana, Transport, Store ou Job. Elle ne démarre ni n'arrête elle-même un runtime concret.
|
||||||
|
|
||||||
Le futur `ksp-worker-raw-transaction-ingest-lib` sera un consumer concret distinct du Job Backfill : il fonctionnera en continu entre Start et Stop, sans scope historique métier, afin d'acquérir et persister des `RawTransaction` selon les sources/capabilities configurées. `ksp-job-backfill-lib` conserve de son côté son rôle historique paramétré et borné. Les deux producteurs restent indépendants et convergent uniquement vers les mêmes contrats RAW/Store.
|
`ksp-worker-raw-transaction-ingest-lib` est le premier Worker concret distinct du Job Backfill. Il conserve sa fondation Start/Stop, admission bornée, Common RAW, Store backend-neutral, shutdown borné et snapshots latest-value. Ses sources productives couvrent maintenant Yellowstone + hydration HTTP `getTransaction`, Standard WS `logsSubscribe` + hydration, Standard WS `blockSubscribe` RAW-direct, Helius `transactionSubscribe` + hydration et Solana HTTP live block polling. Le polling HTTP démarre au `getSlot` observé du run, découvre uniquement les blocs live via `getBlocksWithLimit` puis matérialise les slots listés via `getBlock observed`; il n'est jamais utilisé comme Backfill implicite. Les signaux hydratés de même `(network, signature, commitment)` convergent vers le même coordinator, tandis que les voies block RAW-direct passent par la même admission Common RAW. Le reconnect/replay reste propriétaire de Transport, la frontier de traitement reste run-local et un gap de rétention prouvé provoque un fault au lieu de lancer une campagne historique. `ksp-job-backfill-lib` conserve son rôle historique paramétré et borné ; les deux producteurs restent indépendants et convergent uniquement vers les mêmes contrats RAW/Store.
|
||||||
|
|
||||||
## Points d'entrée
|
## Points d'entrée
|
||||||
|
|
||||||
|
|||||||
10
ROADMAP.md
10
ROADMAP.md
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: ROADMAP.md -->
|
<!-- file: ROADMAP.md -->
|
||||||
<!-- version: 108 -->
|
<!-- version: 111 -->
|
||||||
|
|
||||||
# Roadmap KSP
|
# Roadmap KSP
|
||||||
|
|
||||||
@@ -103,9 +103,9 @@ RAW -> STRUCTURAL -> DECODED -> DOMAIN
|
|||||||
- [X] `0.3.8` — `ksp-app-store-desk` V1 RAW livrée sur le gabarit KSP courant comme application Tauri read-only backend-neutral. La Desk compose Config + Logging + `ksp-store-lib`, expose health/runtime sûrs, DataTables `serverSide` pour `RawTransaction`, `RawAccountState` et leurs observations, détails bornés, provenance sûre et états de rétention/tombstone, sans SQL/backend physique/Transport dans l'application. La nouvelle inspection random-access `offset + limit + counts exacts` reste distincte de la pagination machine cursor/keyset conservée pour workers/backfills/replays. Le gate final et les builds Linux `.deb`/`.rpm` sont verts.
|
- [X] `0.3.8` — `ksp-app-store-desk` V1 RAW livrée sur le gabarit KSP courant comme application Tauri read-only backend-neutral. La Desk compose Config + Logging + `ksp-store-lib`, expose health/runtime sûrs, DataTables `serverSide` pour `RawTransaction`, `RawAccountState` et leurs observations, détails bornés, provenance sûre et états de rétention/tombstone, sans SQL/backend physique/Transport dans l'application. La nouvelle inspection random-access `offset + limit + counts exacts` reste distincte de la pagination machine cursor/keyset conservée pour workers/backfills/replays. Le gate final et les builds Linux `.deb`/`.rpm` sont verts.
|
||||||
- [X] `0.3.9` — `ksp-worker-api` stable comme API générique et volontairement courte pour services continus : identité/kind bornés, lifecycle explicite, health/activity sûrs, stop token partagé, séquence/snapshot latest-value et `WorkerSnapshotSource` object-safe, avec dépendance Core-only et sans runtime/start-stop universel, Solana, Transport, Store, Config ou Tauri. Après freeze Worker API, l'audit `RawTransaction` a été consolidé dans une synthèse Store-centrique exhaustive séparant possibilités/support/preuve et classant HTTP, WS standard/provider, Yellowstone gRPC, blocs/slots, discovery + hydration, replay de continuité, archives et sources EARLY. Le Job Backfill historique paramétré et le Worker Ingest live start/stop sont deux producteurs indépendants du même Store ; `mainnet` est désormais l'identité KSP canonique et `mainnet-beta` un alias legacy/externe. Le handoff retient `ksp-raw-transaction-lib` comme lower-layer commune de canonicalisation RAW v1 pour `0.3.10`, sans edge Job ↔ Worker.
|
- [X] `0.3.9` — `ksp-worker-api` stable comme API générique et volontairement courte pour services continus : identité/kind bornés, lifecycle explicite, health/activity sûrs, stop token partagé, séquence/snapshot latest-value et `WorkerSnapshotSource` object-safe, avec dépendance Core-only et sans runtime/start-stop universel, Solana, Transport, Store, Config ou Tauri. Après freeze Worker API, l'audit `RawTransaction` a été consolidé dans une synthèse Store-centrique exhaustive séparant possibilités/support/preuve et classant HTTP, WS standard/provider, Yellowstone gRPC, blocs/slots, discovery + hydration, replay de continuité, archives et sources EARLY. Le Job Backfill historique paramétré et le Worker Ingest live start/stop sont deux producteurs indépendants du même Store ; `mainnet` est désormais l'identité KSP canonique et `mainnet-beta` un alias legacy/externe. Le handoff retient `ksp-raw-transaction-lib` comme lower-layer commune de canonicalisation RAW v1 pour `0.3.10`, sans edge Job ↔ Worker.
|
||||||
- [X] `0.3.10` — Lower-layer RAW Transaction commune stabilisée : `ksp-raw-transaction-lib` possède la canonicalisation RAW v1 source-neutral partagée, le Backfill est migré sans changement des golden bytes/hash, Transport expose `get_block_observed`, le wire Solana Legacy/V0/V1 est matérialisé et les canaris HTTP/WS standard/Helius/Yellowstone qualifient précisément les voies directes ou nécessitant hydration. La release se ferme sans runtime Worker concret ; celui-ci commence en `0.3.11` selon le redécoupage `0.3.11`–`0.3.14`.
|
- [X] `0.3.10` — Lower-layer RAW Transaction commune stabilisée : `ksp-raw-transaction-lib` possède la canonicalisation RAW v1 source-neutral partagée, le Backfill est migré sans changement des golden bytes/hash, Transport expose `get_block_observed`, le wire Solana Legacy/V0/V1 est matérialisé et les canaris HTTP/WS standard/Helius/Yellowstone qualifient précisément les voies directes ou nécessitant hydration. La release se ferme sans runtime Worker concret ; celui-ci commence en `0.3.11` selon le redécoupage `0.3.11`–`0.3.14`.
|
||||||
- [ ] `0.3.11` — Introduire la **fondation runtime** de `ksp-worker-raw-transaction-ingest-lib` sans source live complexe : crate/dependency firewall, settings source-neutral, handle/start-stop, lifecycle, snapshots, supervisor, bounded channels et pipeline de persistence/déduplication déterministe. Aucun protocole live n'est encore requis pour fermer cette release.
|
- [X] `0.3.11` — Fondation runtime de `ksp-worker-raw-transaction-ingest-lib` livrée : crate/dependency firewall, settings source-neutral, handle/start-stop, lifecycle, snapshots, supervisor, admission bornée, canonicalisation Common RAW, persistence/déduplication Store déterministe, backpressure et shutdown/fault hardening. La release se ferme volontairement sans source réseau productive ; Yellowstone + hydration HTTP commencent en `0.3.12`.
|
||||||
- [ ] `0.3.12` — Ajouter au Worker la voie **Yellowstone + hydration HTTP** : transactions/blocks/status, projection vers la common RAW, provenance sûre, `from_slot`, replay info, frontier de run et continuité propre au run, avec fixtures déterministes et smokes accessibles sans prétendre à un RAW-direct lorsque les métadonnées restent incomplètes.
|
- [X] `0.3.12` — Première source live productive du Worker livrée : Yellowstone `Transaction`/`TransactionStatus`/`Block` vers signaux source-neutral, coalescence bornée puis hydration HTTP `getTransaction` observed avant Common RAW/Store ; `BlockMeta`/`Slot` restent continuity-only. Processing frontier run-local, reconnect/from_slot/ReplayInfo Transport-owned, compteurs source-neutral et fault sur gap de rétention prouvé sans Backfill automatique. Hardening duplicate/backpressure/stop/fault et canaris cross-layer fermés ; gate workspace complet vert, smokes live HTTP Devnet + WebSocket Devnet verts, Yellowstone token-gated/Worker end-to-end laissés explicitement non exécutés.
|
||||||
- [ ] `0.3.13` — Ajouter les voies **WS standard / Helius / HTTP live** et leur convergence multi-source : `logsSubscribe + getTransaction`, `blockSubscribe`, `transactionSubscribe + hydration`, polling blocs HTTP, observations multiples, content conflict explicite, coalescence et backpressure sans second actor Transport.
|
- [X] `0.3.13` — Convergence live multi-source du Worker RAW livrée : composition caller-owned `1..32` sources d’un même réseau, cinq familles Yellowstone / Standard Logs / Standard Block / Helius Transaction / HTTP Block Polling, hydration globale des trois voies reference-bearing, RAW-direct Legacy/V0/V1 pour les deux voies blocs, observations multiples, content conflict explicite, coalescence cross-source, fairness/backpressure/health source-neutral et shutdown hardening sans second actor Transport. Gate workspace complet vert (`1 850` PASS / `0` échec / `15` ignored) et smokes keyless HTTP + WebSocket Devnet verts ; les preuves provider/resource-gated non exécutées restent explicitement non revendiquées.
|
||||||
- [ ] `0.3.14` — Fermer le Worker par le **gap repair/hardening multi-source** : replay natif, source redondante, scan HTTP blocs, hydration de réparation, unresolved gaps visibles, politiques degraded/unhealthy/faulted, shutdown pendant repair, smokes provider accessibles et adapter EARLY seulement si réellement prouvé.
|
- [ ] `0.3.14` — Fermer le Worker par le **gap repair/hardening multi-source** : replay natif, source redondante, scan HTTP blocs, hydration de réparation, unresolved gaps visibles, politiques degraded/unhealthy/faulted, shutdown pendant repair, smokes provider accessibles et adapter EARLY seulement si réellement prouvé.
|
||||||
- [ ] `0.3.15` — Introduire `ksp-app-raw-transaction-ingest-desk`, Desk Tauri KSP spécialisée qui choisit et supervise une ou plusieurs sources/méthodes réellement admises par le Worker finalisé : lifecycle, health, rates, backpressure, reconnect/recovery, gap state et compteurs sûrs. La Desk ne réimplémente ni discovery, hydration, déduplication, reprise ni persistence.
|
- [ ] `0.3.15` — Introduire `ksp-app-raw-transaction-ingest-desk`, Desk Tauri KSP spécialisée qui choisit et supervise une ou plusieurs sources/méthodes réellement admises par le Worker finalisé : lifecycle, health, rates, backpressure, reconnect/recovery, gap state et compteurs sûrs. La Desk ne réimplémente ni discovery, hydration, déduplication, reprise ni persistence.
|
||||||
- [ ] `0.3.16` — Étendre `ksp-job-backfill-lib` et `ksp-app-backfill-desk` au **backfill multi-source/multi-stratégie** à partir de la même matrice d'acquisition auditée en `0.3.9`. Conserver la stratégie actuelle `getSignaturesForAddress + getTransaction` comme première voie HTTP valide, puis ajouter seulement les voies historiques/catch-up/gap-repair réellement pertinentes et sûres, sans supposer qu'une source live WS constitue un historique universel.
|
- [ ] `0.3.16` — Étendre `ksp-job-backfill-lib` et `ksp-app-backfill-desk` au **backfill multi-source/multi-stratégie** à partir de la même matrice d'acquisition auditée en `0.3.9`. Conserver la stratégie actuelle `getSignaturesForAddress + getTransaction` comme première voie HTTP valide, puis ajouter seulement les voies historiques/catch-up/gap-repair réellement pertinentes et sûres, sans supposer qu'une source live WS constitue un historique universel.
|
||||||
@@ -119,7 +119,7 @@ RAW -> STRUCTURAL -> DECODED -> DOMAIN
|
|||||||
|
|
||||||
- [ ] **TODO** — maintenir la matrice d’admission HTTP/WS/gRPC/provider lors de toute nouvelle famille D1 : plusieurs sources ne convergent vers un même struct que si elles satisfont la même sémantique sans perte.
|
- [ ] **TODO** — maintenir la matrice d’admission HTTP/WS/gRPC/provider lors de toute nouvelle famille D1 : plusieurs sources ne convergent vers un même struct que si elles satisfont la même sémantique sans perte.
|
||||||
- [X] **TODO `0.3.9`** — matrice RAW Transaction unique produite et consolidée dans `docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md`, avec séparation possibilités/support/preuve, capabilities orthogonales aux producteurs, provenance/déduplication/continuité et handoffs désormais scindés `0.3.11`–`0.3.14` Worker live / `0.3.16` Backfill historique.
|
- [X] **TODO `0.3.9`** — matrice RAW Transaction unique produite et consolidée dans `docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md`, avec séparation possibilités/support/preuve, capabilities orthogonales aux producteurs, provenance/déduplication/continuité et handoffs désormais scindés `0.3.11`–`0.3.14` Worker live / `0.3.16` Backfill historique.
|
||||||
- [ ] **TODO Helius `0.3.13`** — ajouter uniquement les profils/capabilities Helius réellement nécessaires au Worker live après revalidation des surfaces/tier courants ; réutiliser exclusivement `KSP_SECRET_HELIUS_API_KEY` via Config et la surface `transactionSubscribe` déjà Transport-owned, sans SDK provider ni second client parallèle.
|
- [X] **TODO Helius `0.3.13`** — la voie Worker Helius `transactionSubscribe` réutilise la surface Transport et les profils Config Helius déjà existants avec `KSP_SECRET_HELIUS_API_KEY`; aucun nouveau profil, secret, SDK provider, tier codé ou second client parallèle n’a été nécessaire. Le payload riche reste Transport-owned et la voie Worker hydrate via `getTransaction observed` tant que le RAW complet n’est pas prouvé directement.
|
||||||
- [X] **TODO réseau** — identité KSP canonique fixée à `mainnet` dans Config/Store/Transport/tests ; `mainnet-beta` reste uniquement un alias legacy/externe lorsque la frontière provider l'exige. Les données D1 RAW antérieures restent non autoritaires pendant cette phase et aucune compatibilité de base de test n'impose l'ancien libellé.
|
- [X] **TODO réseau** — identité KSP canonique fixée à `mainnet` dans Config/Store/Transport/tests ; `mainnet-beta` reste uniquement un alias legacy/externe lorsque la frontière provider l'exige. Les données D1 RAW antérieures restent non autoritaires pendant cette phase et aucune compatibilité de base de test n'impose l'ancien libellé.
|
||||||
- [X] `RawAccountState` + observation — contrat commun stabilisé en `0.3.1` avec bytes complets + slot, provenance séparée et enrichissements source-specific optionnels ; la persistence PostgreSQL physique est complétée en `0.3.4` avec les quatre capabilities account et la conformance RAW 10/10.
|
- [X] `RawAccountState` + observation — contrat commun stabilisé en `0.3.1` avec bytes complets + slot, provenance séparée et enrichissements source-specific optionnels ; la persistence PostgreSQL physique est complétée en `0.3.4` avec les quatre capabilities account et la conformance RAW 10/10.
|
||||||
- [ ] **TODO** — statut/commitment transactionnel restant : `0.3.5` couvre uniquement le fait passif d’exécution `slot + signature + outcome`; réauditer séparément `signatureSubscribe` et `getSignatureStatuses` lorsqu’un consumer de commitment/snapshot réel apparaît, sans fusionner snapshot, transition et execution update dans un modèle Option-soup.
|
- [ ] **TODO** — statut/commitment transactionnel restant : `0.3.5` couvre uniquement le fait passif d’exécution `slot + signature + outcome`; réauditer séparément `signatureSubscribe` et `getSignatureStatuses` lorsqu’un consumer de commitment/snapshot réel apparaît, sans fusionner snapshot, transition et execution update dans un modèle Option-soup.
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/src/grpc_stream.rs
|
// file: crates/ksp-onchain-transport-lib/src/grpc_stream.rs
|
||||||
// version: 3
|
// version: 4
|
||||||
|
|
||||||
use tonic_prost::prost::Message; // rust-rules: trait-import
|
use tonic_prost::prost::Message; // rust-rules: trait-import
|
||||||
|
|
||||||
@@ -39,6 +39,43 @@ pub struct YellowstoneGrpcSubscribeSnapshot {
|
|||||||
terminal_error_code: std::option::Option<ksp_core_lib::ErrorCode>,
|
terminal_error_code: std::option::Option<ksp_core_lib::ErrorCode>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Cloneable latest-value observer for one standard Yellowstone Subscribe session snapshot.
|
||||||
|
///
|
||||||
|
/// The observer exposes only the already-safe transport snapshot and keeps the internal Tokio watch channel private. Cloning the observer does not duplicate
|
||||||
|
/// the gRPC stream, request state or reconnect actor.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct YellowstoneGrpcSubscribeSnapshotSource {
|
||||||
|
receiver: tokio::sync::watch::Receiver<crate::YellowstoneGrpcSubscribeSnapshot>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::YellowstoneGrpcSubscribeSnapshotSource {
|
||||||
|
fn new(receiver: tokio::sync::watch::Receiver<crate::YellowstoneGrpcSubscribeSnapshot>) -> Self {
|
||||||
|
return Self { receiver };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the current safe reconnect/replay snapshot without waiting for another actor transition.
|
||||||
|
#[must_use]
|
||||||
|
pub fn current(&self) -> crate::YellowstoneGrpcSubscribeSnapshot {
|
||||||
|
return *self.receiver.borrow();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Waits for one newer safe reconnect/replay snapshot.
|
||||||
|
///
|
||||||
|
/// `None` means the owning Subscribe actor dropped the latest-value publisher and no further snapshot can arrive.
|
||||||
|
pub async fn wait_for_change(&mut self) -> std::option::Option<crate::YellowstoneGrpcSubscribeSnapshot> {
|
||||||
|
return match self.receiver.changed().await {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::Some(*self.receiver.borrow_and_update()),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::YellowstoneGrpcSubscribeSnapshotSource {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter.debug_struct("YellowstoneGrpcSubscribeSnapshotSource").field("current", &self.current()).finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
impl YellowstoneGrpcSubscribeSnapshot {
|
impl YellowstoneGrpcSubscribeSnapshot {
|
||||||
const fn new(initial_from_slot: std::option::Option<u64>) -> Self {
|
const fn new(initial_from_slot: std::option::Option<u64>) -> Self {
|
||||||
return Self {
|
return Self {
|
||||||
@@ -154,6 +191,12 @@ impl SolanaYellowstoneGrpcSubscribeSession {
|
|||||||
return *self.snapshot_rx.borrow();
|
return *self.snapshot_rx.borrow();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Returns one cloneable latest-value observer for reconnect/replay snapshot transitions.
|
||||||
|
#[must_use]
|
||||||
|
pub fn snapshot_source(&self) -> crate::YellowstoneGrpcSubscribeSnapshotSource {
|
||||||
|
return crate::YellowstoneGrpcSubscribeSnapshotSource::new(self.snapshot_rx.clone());
|
||||||
|
}
|
||||||
|
|
||||||
/// Queues one complete standard Yellowstone request mutation without waiting for network dispatch.
|
/// Queues one complete standard Yellowstone request mutation without waiting for network dispatch.
|
||||||
///
|
///
|
||||||
/// The mutation is rejected synchronously when local validation fails, the encoded request exceeds the configured outbound bound, the bounded request
|
/// The mutation is rejected synchronously when local validation fails, the encoded request exceeds the configured outbound bound, the bounded request
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/src/grpc_subscribe.rs
|
// file: crates/ksp-onchain-transport-lib/src/grpc_subscribe.rs
|
||||||
// version: 8
|
// version: 9
|
||||||
|
|
||||||
const MAX_GRPC_BLOCKHASH_TEXT_LENGTH_BYTES: usize = 128;
|
const MAX_GRPC_BLOCKHASH_TEXT_LENGTH_BYTES: usize = 128;
|
||||||
const MAX_GRPC_BLOCK_VECTOR_COUNT: usize = 65_536;
|
const MAX_GRPC_BLOCK_VECTOR_COUNT: usize = 65_536;
|
||||||
@@ -2529,13 +2529,34 @@ impl YellowstoneSubscribeEntryFilter {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Opaque deterministic identity material for one complete Yellowstone Subscribe request.
|
||||||
|
///
|
||||||
|
/// The encoded bytes remain private. `Hash` feeds those canonical bytes to a caller-provided hasher while `Debug` exposes only their length.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct YellowstoneSubscribeRequestIdentity {
|
||||||
|
bytes: std::vec::Vec<u8>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::hash::Hash for crate::YellowstoneSubscribeRequestIdentity {
|
||||||
|
fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
|
||||||
|
state.write(&(self.bytes.len() as u64).to_be_bytes());
|
||||||
|
state.write(self.bytes.as_slice());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::YellowstoneSubscribeRequestIdentity {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter.debug_struct("YellowstoneSubscribeRequestIdentity").field("byte_len", &self.bytes.len()).finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Provider-neutral standard Yellowstone subscribe request owned by KSP.
|
/// Provider-neutral standard Yellowstone subscribe request owned by KSP.
|
||||||
///
|
///
|
||||||
/// The seven upstream maps are represented independently and retain named empty entries. An entirely empty map is the logical KSP representation of no active
|
/// The seven upstream maps are represented independently and retain named empty entries. An entirely empty map is the logical KSP representation of no active
|
||||||
/// filter in that family; protobuf map encoding does not distinguish an omitted map from an empty map. Filter-group names are globally unique across all seven
|
/// filter in that family; protobuf map encoding does not distinguish an omitted map from an empty map. Filter-group names are globally unique across all seven
|
||||||
/// maps so the names echoed by `SubscribeUpdate.filters` remain unambiguous. `Debug` exposes only counts and common scalar options, never filter names or
|
/// maps so the names echoed by `SubscribeUpdate.filters` remain unambiguous. `Debug` exposes only counts and common scalar options, never filter names or
|
||||||
/// future
|
/// future filter payloads.
|
||||||
/// filter payloads.
|
|
||||||
#[derive(Clone, Eq, PartialEq)]
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
pub struct YellowstoneSubscribeRequest {
|
pub struct YellowstoneSubscribeRequest {
|
||||||
accounts: std::collections::BTreeMap<crate::YellowstoneSubscribeFilterName, crate::YellowstoneSubscribeAccountFilter>,
|
accounts: std::collections::BTreeMap<crate::YellowstoneSubscribeFilterName, crate::YellowstoneSubscribeAccountFilter>,
|
||||||
@@ -2551,7 +2572,7 @@ pub struct YellowstoneSubscribeRequest {
|
|||||||
from_slot: std::option::Option<u64>,
|
from_slot: std::option::Option<u64>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl YellowstoneSubscribeRequest {
|
impl crate::YellowstoneSubscribeRequest {
|
||||||
/// Creates an empty subscribe request. Empty requests are valid because later bidi lifecycle code uses request mutations to clear filters or carry ping
|
/// Creates an empty subscribe request. Empty requests are valid because later bidi lifecycle code uses request mutations to clear filters or carry ping
|
||||||
/// state.
|
/// state.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
@@ -2762,6 +2783,40 @@ impl YellowstoneSubscribeRequest {
|
|||||||
return self.from_slot;
|
return self.from_slot;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Builds one opaque deterministic identity for the complete logical Subscribe request.
|
||||||
|
///
|
||||||
|
/// The identity preserves filter-family separation, globally sorted filter names, exact filter wire payloads and common request options. Its Debug surface
|
||||||
|
/// exposes only the encoded byte length. Callers may hash the opaque value but cannot recover the underlying identity bytes through this API.
|
||||||
|
pub fn identity(&self) -> ksp_core_lib::Result<crate::YellowstoneSubscribeRequestIdentity> {
|
||||||
|
let validation = self.validate();
|
||||||
|
if let std::result::Result::Err(error) = validation {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let mut bytes = std::vec::Vec::new();
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"accounts", &self.accounts, |filter| return filter.to_wire());
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"blocks", &self.blocks, |filter| return filter.to_wire());
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"blocks_meta", &self.blocks_meta, |filter| return filter.to_wire());
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"entry", &self.entry, |filter| return filter.to_wire());
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"slots", &self.slots, |filter| return filter.to_wire());
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"transactions", &self.transactions, |filter| return filter.to_wire());
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"transactions_status", &self.transactions_status, |filter| return filter.to_wire());
|
||||||
|
let mut common = match self.to_wire() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
common.accounts.clear();
|
||||||
|
common.blocks.clear();
|
||||||
|
common.blocks_meta.clear();
|
||||||
|
common.entry.clear();
|
||||||
|
common.slots.clear();
|
||||||
|
common.transactions.clear();
|
||||||
|
common.transactions_status.clear();
|
||||||
|
let common = yellowstone_grpc_proto::prost::Message::encode_to_vec(&common);
|
||||||
|
append_subscribe_identity_component(&mut bytes, b"common");
|
||||||
|
append_subscribe_identity_component(&mut bytes, common.as_slice());
|
||||||
|
return std::result::Result::Ok(crate::YellowstoneSubscribeRequestIdentity { bytes });
|
||||||
|
}
|
||||||
|
|
||||||
/// Validates all deterministic common subscribe-request bounds before any network I/O.
|
/// Validates all deterministic common subscribe-request bounds before any network I/O.
|
||||||
pub fn validate(&self) -> ksp_core_lib::Result<()> {
|
pub fn validate(&self) -> ksp_core_lib::Result<()> {
|
||||||
if self.total_filter_count() > MAX_GRPC_SUBSCRIBE_FILTER_GROUP_COUNT {
|
if self.total_filter_count() > MAX_GRPC_SUBSCRIBE_FILTER_GROUP_COUNT {
|
||||||
@@ -3710,13 +3765,38 @@ fn commitment_to_wire(commitment: std::option::Option<crate::SolanaCommitment>)
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
impl std::default::Default for YellowstoneSubscribeRequest {
|
fn append_subscribe_identity_component(output: &mut std::vec::Vec<u8>, value: &[u8]) {
|
||||||
|
output.extend_from_slice(&(value.len() as u64).to_be_bytes());
|
||||||
|
output.extend_from_slice(value);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn append_subscribe_identity_map<V, W, F>(
|
||||||
|
output: &mut std::vec::Vec<u8>,
|
||||||
|
family: &[u8],
|
||||||
|
values: &std::collections::BTreeMap<crate::YellowstoneSubscribeFilterName, V>,
|
||||||
|
mut to_wire: F,
|
||||||
|
) where
|
||||||
|
W: yellowstone_grpc_proto::prost::Message,
|
||||||
|
F: FnMut(&V) -> W,
|
||||||
|
{
|
||||||
|
append_subscribe_identity_component(output, family);
|
||||||
|
output.extend_from_slice(&(values.len() as u64).to_be_bytes());
|
||||||
|
for (name, filter) in values {
|
||||||
|
append_subscribe_identity_component(output, name.as_str().as_bytes());
|
||||||
|
let wire = yellowstone_grpc_proto::prost::Message::encode_to_vec(&to_wire(filter));
|
||||||
|
append_subscribe_identity_component(output, wire.as_slice());
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::default::Default for crate::YellowstoneSubscribeRequest {
|
||||||
fn default() -> Self {
|
fn default() -> Self {
|
||||||
return Self::new();
|
return Self::new();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl std::fmt::Debug for YellowstoneSubscribeRequest {
|
impl std::fmt::Debug for crate::YellowstoneSubscribeRequest {
|
||||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
return formatter
|
return formatter
|
||||||
.debug_struct("YellowstoneSubscribeRequest")
|
.debug_struct("YellowstoneSubscribeRequest")
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/src/lib.rs
|
// file: crates/ksp-onchain-transport-lib/src/lib.rs
|
||||||
// version: 45
|
// version: 47
|
||||||
|
|
||||||
#![warn(missing_docs)]
|
#![warn(missing_docs)]
|
||||||
#![deny(unreachable_pub)]
|
#![deny(unreachable_pub)]
|
||||||
@@ -143,6 +143,8 @@ pub use self::grpc_settings::YellowstoneGrpcTransportSettings;
|
|||||||
pub use self::grpc_stream::SolanaYellowstoneGrpcSubscribeSession;
|
pub use self::grpc_stream::SolanaYellowstoneGrpcSubscribeSession;
|
||||||
/// Safe Yellowstone reconnect/replay continuity snapshot.
|
/// Safe Yellowstone reconnect/replay continuity snapshot.
|
||||||
pub use self::grpc_stream::YellowstoneGrpcSubscribeSnapshot;
|
pub use self::grpc_stream::YellowstoneGrpcSubscribeSnapshot;
|
||||||
|
/// Cloneable latest-value observer for one standard Yellowstone Subscribe session snapshot.
|
||||||
|
pub use self::grpc_stream::YellowstoneGrpcSubscribeSnapshotSource;
|
||||||
/// Safe Yellowstone bidirectional Subscribe lifecycle state.
|
/// Safe Yellowstone bidirectional Subscribe lifecycle state.
|
||||||
pub use self::grpc_stream::YellowstoneGrpcSubscribeState;
|
pub use self::grpc_stream::YellowstoneGrpcSubscribeState;
|
||||||
/// One validated standard Yellowstone account predicate.
|
/// One validated standard Yellowstone account predicate.
|
||||||
@@ -213,6 +215,8 @@ pub use self::grpc_subscribe::YellowstoneSubscribePingUpdate;
|
|||||||
pub use self::grpc_subscribe::YellowstoneSubscribePongUpdate;
|
pub use self::grpc_subscribe::YellowstoneSubscribePongUpdate;
|
||||||
/// Provider-neutral standard Yellowstone Subscribe request.
|
/// Provider-neutral standard Yellowstone Subscribe request.
|
||||||
pub use self::grpc_subscribe::YellowstoneSubscribeRequest;
|
pub use self::grpc_subscribe::YellowstoneSubscribeRequest;
|
||||||
|
/// Opaque deterministic identity for one complete Yellowstone Subscribe request.
|
||||||
|
pub use self::grpc_subscribe::YellowstoneSubscribeRequestIdentity;
|
||||||
/// Complete slot-family filter group for standard Yellowstone Subscribe.
|
/// Complete slot-family filter group for standard Yellowstone Subscribe.
|
||||||
pub use self::grpc_subscribe::YellowstoneSubscribeSlotFilter;
|
pub use self::grpc_subscribe::YellowstoneSubscribeSlotFilter;
|
||||||
/// Complete transaction-family filter shared by transactions and transaction-status maps.
|
/// Complete transaction-family filter shared by transactions and transaction-status maps.
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/tests/public_api.rs
|
// file: crates/ksp-onchain-transport-lib/tests/public_api.rs
|
||||||
// version: 50
|
// version: 52
|
||||||
|
|
||||||
//! Integration tests for the public `ksp-onchain-transport-lib` consumer contract.
|
//! Integration tests for the public `ksp-onchain-transport-lib` consumer contract.
|
||||||
|
|
||||||
@@ -1013,6 +1013,10 @@ fn public_v0_2_9_pre_010_yellowstone_reconnect_snapshot_is_available_from_crate_
|
|||||||
let _snapshot = std::any::type_name::<ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot>();
|
let _snapshot = std::any::type_name::<ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot>();
|
||||||
let _state = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeState::Reconnecting;
|
let _state = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeState::Reconnecting;
|
||||||
let _session_snapshot = ksp_onchain_transport_lib::SolanaYellowstoneGrpcSubscribeSession::snapshot;
|
let _session_snapshot = ksp_onchain_transport_lib::SolanaYellowstoneGrpcSubscribeSession::snapshot;
|
||||||
|
let _snapshot_source = ksp_onchain_transport_lib::SolanaYellowstoneGrpcSubscribeSession::snapshot_source;
|
||||||
|
let _source_type = std::any::type_name::<ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshotSource>();
|
||||||
|
let _source_current = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshotSource::current;
|
||||||
|
let _source_wait = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshotSource::wait_for_change;
|
||||||
let _reconnect_count = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::reconnect_count;
|
let _reconnect_count = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::reconnect_count;
|
||||||
let _gap_count = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::continuity_gap_count;
|
let _gap_count = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::continuity_gap_count;
|
||||||
let _duplicate_count = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::duplicate_update_count;
|
let _duplicate_count = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::duplicate_update_count;
|
||||||
@@ -1028,3 +1032,15 @@ fn public_v0_3_10_pre_004_observed_get_block_surface_is_available_from_crate_roo
|
|||||||
let _ = method;
|
let _ = method;
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn public_v0_3_13_pre_002_yellowstone_subscribe_identity_is_opaque_and_available_from_crate_root() {
|
||||||
|
let request = ksp_onchain_transport_lib::YellowstoneSubscribeRequest::new();
|
||||||
|
let identity = request.identity().expect("empty validated request identity must build");
|
||||||
|
let _identity_type = std::any::type_name::<ksp_onchain_transport_lib::YellowstoneSubscribeRequestIdentity>();
|
||||||
|
let debug = std::format!("{identity:?}");
|
||||||
|
assert!(debug.contains("YellowstoneSubscribeRequestIdentity"));
|
||||||
|
assert!(debug.contains("byte_len"));
|
||||||
|
assert!(!debug.contains("bytes:"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/tests/release_completeness.rs
|
// file: crates/ksp-onchain-transport-lib/tests/release_completeness.rs
|
||||||
// version: 42
|
// version: 44
|
||||||
|
|
||||||
//! Release-level completeness canaries for staged HTTP and WebSocket Transport coverage.
|
//! Release-level completeness canaries for staged HTTP and WebSocket Transport coverage.
|
||||||
|
|
||||||
@@ -1307,6 +1307,9 @@ fn release_v0_2_9_pre_010_adds_bounded_reconnect_replay_and_conservative_continu
|
|||||||
for required in [
|
for required in [
|
||||||
"YellowstoneGrpcSubscribeState::Reconnecting",
|
"YellowstoneGrpcSubscribeState::Reconnecting",
|
||||||
"YellowstoneGrpcSubscribeSnapshot",
|
"YellowstoneGrpcSubscribeSnapshot",
|
||||||
|
"YellowstoneGrpcSubscribeSnapshotSource",
|
||||||
|
"snapshot_source",
|
||||||
|
"wait_for_change",
|
||||||
"reconnect_subscribe_stream",
|
"reconnect_subscribe_stream",
|
||||||
"replay_first_available",
|
"replay_first_available",
|
||||||
"SubscribeReplayInfo",
|
"SubscribeReplayInfo",
|
||||||
@@ -1336,3 +1339,18 @@ fn release_v0_2_9_pre_010_adds_bounded_reconnect_replay_and_conservative_continu
|
|||||||
let _snapshot = std::any::type_name::<ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot>();
|
let _snapshot = std::any::type_name::<ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot>();
|
||||||
let _state = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeState::Reconnecting;
|
let _state = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeState::Reconnecting;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn release_v0_3_13_pre_002_yellowstone_subscribe_identity_remains_opaque_and_dependency_neutral() {
|
||||||
|
let source = include_str!("../src/grpc_subscribe.rs");
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let identity_start = source.find("pub struct YellowstoneSubscribeRequestIdentity").expect("identity struct must remain present");
|
||||||
|
let request_start = source.find("/// Provider-neutral standard Yellowstone subscribe request").expect("request contract marker must remain present");
|
||||||
|
let identity_surface = &source[identity_start..request_start];
|
||||||
|
assert!(source.contains("pub fn identity(&self)"));
|
||||||
|
assert!(root.contains("pub use self::grpc_subscribe::YellowstoneSubscribeRequestIdentity;"));
|
||||||
|
assert!(!identity_surface.contains("pub fn bytes("));
|
||||||
|
assert!(!identity_surface.contains("pub fn as_bytes("));
|
||||||
|
assert!(!root.contains("pub use yellowstone_grpc_proto"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/unit_tests/grpc_stream.rs
|
// file: crates/ksp-onchain-transport-lib/unit_tests/grpc_stream.rs
|
||||||
// version: 4
|
// version: 5
|
||||||
|
|
||||||
#[derive(Clone, Copy)]
|
#[derive(Clone, Copy)]
|
||||||
enum FixtureMode {
|
enum FixtureMode {
|
||||||
@@ -159,6 +159,9 @@ impl yellowstone_grpc_proto::geyser::geyser_server::Geyser for FixtureGeyser {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
if matches!(mode, FixtureMode::ReconnectReplay) && subscribe_call == 2 {
|
||||||
|
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
|
||||||
|
}
|
||||||
return std::result::Result::Ok(tonic::Response::new(super::MpscStream::new(outbound_rx)));
|
return std::result::Result::Ok(tonic::Response::new(super::MpscStream::new(outbound_rx)));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -662,6 +665,45 @@ async fn yellowstone_reconnect_budget_exhaustion_is_terminal_and_safe() {
|
|||||||
server.stop().await;
|
server.stop().await;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn yellowstone_snapshot_source_observes_replay_attempt_before_successful_reconnect() {
|
||||||
|
let server = FixtureServer::start(FixtureMode::ReconnectReplay).await;
|
||||||
|
let defaults = crate::YellowstoneGrpcSessionSettings::default();
|
||||||
|
let settings = fixture_settings_with_reconnect(
|
||||||
|
server.endpoint_url.as_str(),
|
||||||
|
8,
|
||||||
|
8,
|
||||||
|
defaults.max_inbound_message_size_bytes(),
|
||||||
|
defaults.max_outbound_message_size_bytes(),
|
||||||
|
crate::YellowstoneGrpcReconnectSettings::new(3, std::time::Duration::from_millis(5), std::time::Duration::from_millis(20)),
|
||||||
|
);
|
||||||
|
let channel = crate::YellowstoneGrpcChannel::connect(&settings).await.expect("fixture channel must connect");
|
||||||
|
let mut session = channel.open_standard_subscribe(initial_request()).await.expect("fixture Subscribe stream must open");
|
||||||
|
let mut snapshots = session.snapshot_source();
|
||||||
|
assert_eq!(snapshots.current().state(), crate::YellowstoneGrpcSubscribeState::Active);
|
||||||
|
let _ = session.next_update().await.expect("first slot must decode").expect("first slot must be present");
|
||||||
|
let replaying = tokio::time::timeout(std::time::Duration::from_secs(1), async {
|
||||||
|
loop {
|
||||||
|
let snapshot = match snapshots.wait_for_change().await {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
if snapshot.replay_attempt_count() > 0 && snapshot.reconnect_count() == 0 {
|
||||||
|
return std::option::Option::Some(snapshot);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.expect("snapshot source must observe reconnect progress")
|
||||||
|
.expect("snapshot source must stay open during reconnect");
|
||||||
|
assert_eq!(replaying.state(), crate::YellowstoneGrpcSubscribeState::Reconnecting);
|
||||||
|
assert_eq!(replaying.replay_attempt_count(), 1);
|
||||||
|
assert_eq!(replaying.reconnect_count(), 0);
|
||||||
|
let _ = session.next_update().await.expect("replayed duplicate must decode").expect("replayed duplicate must be present");
|
||||||
|
session.close().await.expect("reconnected stream must close cleanly");
|
||||||
|
server.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
#[tokio::test(flavor = "current_thread")]
|
#[tokio::test(flavor = "current_thread")]
|
||||||
async fn yellowstone_shutdown_interrupts_reconnect_backoff_and_mutation_is_rejected_during_reconnect() {
|
async fn yellowstone_shutdown_interrupts_reconnect_backoff_and_mutation_is_rejected_during_reconnect() {
|
||||||
let server = FixtureServer::start(FixtureMode::ReconnectReplay).await;
|
let server = FixtureServer::start(FixtureMode::ReconnectReplay).await;
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/unit_tests/grpc_subscribe.rs
|
// file: crates/ksp-onchain-transport-lib/unit_tests/grpc_subscribe.rs
|
||||||
// version: 5
|
// version: 6
|
||||||
|
|
||||||
fn filter_name(value: &str) -> crate::YellowstoneSubscribeFilterName {
|
fn filter_name(value: &str) -> crate::YellowstoneSubscribeFilterName {
|
||||||
return crate::YellowstoneSubscribeFilterName::new(value).expect("fixture filter name must validate");
|
return crate::YellowstoneSubscribeFilterName::new(value).expect("fixture filter name must validate");
|
||||||
@@ -138,6 +138,37 @@ fn yellowstone_subscribe_common_bounds_reject_before_wire_conversion() {
|
|||||||
assert!(filters.insert_account_filter(excess_name, crate::YellowstoneSubscribeAccountFilter::new()).is_err());
|
assert!(filters.insert_account_filter(excess_name, crate::YellowstoneSubscribeAccountFilter::new()).is_err());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn yellowstone_subscribe_request_identity_is_order_stable_exact_and_debug_redacted() {
|
||||||
|
let mut first = crate::YellowstoneSubscribeRequest::new();
|
||||||
|
let mut second = crate::YellowstoneSubscribeRequest::new();
|
||||||
|
let mut changed = crate::YellowstoneSubscribeRequest::new();
|
||||||
|
let mut filter_a = crate::YellowstoneSubscribeTransactionFilter::new();
|
||||||
|
filter_a.set_failed(std::option::Option::Some(false));
|
||||||
|
let mut filter_b = crate::YellowstoneSubscribeTransactionFilter::new();
|
||||||
|
filter_b.set_vote(std::option::Option::Some(false));
|
||||||
|
assert!(first.insert_transaction_filter(filter_name("identity-filter-beta-canary"), filter_b.clone()).is_ok());
|
||||||
|
assert!(first.insert_transaction_filter(filter_name("identity-filter-alpha-canary"), filter_a.clone()).is_ok());
|
||||||
|
assert!(second.insert_transaction_filter(filter_name("identity-filter-alpha-canary"), filter_a.clone()).is_ok());
|
||||||
|
assert!(second.insert_transaction_filter(filter_name("identity-filter-beta-canary"), filter_b.clone()).is_ok());
|
||||||
|
filter_a.set_failed(std::option::Option::Some(true));
|
||||||
|
assert!(changed.insert_transaction_filter(filter_name("identity-filter-alpha-canary"), filter_a).is_ok());
|
||||||
|
assert!(changed.insert_transaction_filter(filter_name("identity-filter-beta-canary"), filter_b).is_ok());
|
||||||
|
first.set_commitment(std::option::Option::Some(crate::SolanaCommitment::Confirmed));
|
||||||
|
second.set_commitment(std::option::Option::Some(crate::SolanaCommitment::Confirmed));
|
||||||
|
changed.set_commitment(std::option::Option::Some(crate::SolanaCommitment::Confirmed));
|
||||||
|
let first_identity = first.identity().expect("first request identity must build");
|
||||||
|
let second_identity = second.identity().expect("second request identity must build");
|
||||||
|
let changed_identity = changed.identity().expect("changed request identity must build");
|
||||||
|
assert_eq!(first_identity, second_identity);
|
||||||
|
assert_ne!(first_identity, changed_identity);
|
||||||
|
let debug = std::format!("{first_identity:?}");
|
||||||
|
assert!(debug.contains("byte_len"));
|
||||||
|
assert!(!debug.contains("identity-filter-alpha-canary"));
|
||||||
|
assert!(!debug.contains("identity-filter-beta-canary"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn yellowstone_subscribe_debug_omits_filter_names_and_future_payloads() {
|
fn yellowstone_subscribe_debug_omits_filter_names_and_future_payloads() {
|
||||||
let mut request = crate::YellowstoneSubscribeRequest::new();
|
let mut request = crate::YellowstoneSubscribeRequest::new();
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: crates/ksp-raw-transaction-lib/README.md -->
|
<!-- file: crates/ksp-raw-transaction-lib/README.md -->
|
||||||
<!-- version: 1 -->
|
<!-- version: 2 -->
|
||||||
|
|
||||||
# ksp-raw-transaction-lib
|
# ksp-raw-transaction-lib
|
||||||
|
|
||||||
@@ -26,6 +26,7 @@ La canonicalisation produit un `ksp_store_api::RawTransaction` avec payload KSP
|
|||||||
### Signature Solana
|
### Signature Solana
|
||||||
|
|
||||||
- `parse_raw_transaction_signature` valide et décode une signature Base58 bornée vers exactement 64 bytes ;
|
- `parse_raw_transaction_signature` valide et décode une signature Base58 bornée vers exactement 64 bytes ;
|
||||||
|
- `format_raw_transaction_signature` encode exactement 64 bytes canoniques vers la forme Base58 bornée correspondante ;
|
||||||
- `extract_raw_transaction_signature_from_binary_base64` extrait la première signature canonique d'un wire transactionnel Base64 complet ;
|
- `extract_raw_transaction_signature_from_binary_base64` extrait la première signature canonique d'un wire transactionnel Base64 complet ;
|
||||||
- les bornes textuelles publiques permettent l'admission avant décodage.
|
- les bornes textuelles publiques permettent l'admission avant décodage.
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: crates/ksp-raw-transaction-lib/USAGE.md -->
|
<!-- file: crates/ksp-raw-transaction-lib/USAGE.md -->
|
||||||
<!-- version: 1 -->
|
<!-- version: 2 -->
|
||||||
|
|
||||||
# Utilisation de ksp-raw-transaction-lib
|
# Utilisation de ksp-raw-transaction-lib
|
||||||
|
|
||||||
@@ -17,6 +17,16 @@ fn parse_signature(value: &str) -> ksp_core_lib::Result<ksp_store_api::RawTransa
|
|||||||
|
|
||||||
Le résultat contient exactement 64 bytes canoniques. Les entrées vides, hors bornes, Base58 invalides ou de longueur décodée incorrecte sont rejetées sans recopier la valeur hostile dans l'erreur.
|
Le résultat contient exactement 64 bytes canoniques. Les entrées vides, hors bornes, Base58 invalides ou de longueur décodée incorrecte sont rejetées sans recopier la valeur hostile dans l'erreur.
|
||||||
|
|
||||||
|
Lorsqu'un consumer possède déjà les 64 bytes canoniques et doit appeler une API textuelle Solana, utiliser l'encodeur commun plutôt que d'implémenter Base58 localement :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn format_signature(value: &ksp_store_api::RawTransactionSignature) -> std::string::String {
|
||||||
|
return ksp_raw_transaction_lib::format_raw_transaction_signature(value);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le texte produit respecte les bornes publiques de signature et effectue un round-trip exact avec `parse_raw_transaction_signature`.
|
||||||
|
|
||||||
Lorsqu'un wire Base64 complet contient déjà son tableau de signatures, utiliser :
|
Lorsqu'un wire Base64 complet contient déjà son tableau de signatures, utiliser :
|
||||||
|
|
||||||
```rust
|
```rust
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-raw-transaction-lib/src/lib.rs
|
// file: crates/ksp-raw-transaction-lib/src/lib.rs
|
||||||
// version: 3
|
// version: 4
|
||||||
|
|
||||||
#![warn(missing_docs)]
|
#![warn(missing_docs)]
|
||||||
#![deny(unreachable_pub)]
|
#![deny(unreachable_pub)]
|
||||||
@@ -7,7 +7,7 @@
|
|||||||
|
|
||||||
//! Source-neutral canonicalization of complete Solana transactions into KSP RAW transaction v1.
|
//! Source-neutral canonicalization of complete Solana transactions into KSP RAW transaction v1.
|
||||||
//!
|
//!
|
||||||
//! This lower layer owns the frozen RAW transaction format, textual signature parsing, canonical
|
//! This lower layer owns the frozen RAW transaction format, textual signature parsing/formatting, canonical
|
||||||
//! JSON construction, content hashing and assembly of one canonical transaction with one
|
//! JSON construction, content hashing and assembly of one canonical transaction with one
|
||||||
//! producer-owned observation. It owns no Transport, Config, Job, Worker, async runtime or Store
|
//! producer-owned observation. It owns no Transport, Config, Job, Worker, async runtime or Store
|
||||||
//! backend behavior.
|
//! backend behavior.
|
||||||
@@ -46,6 +46,8 @@ pub use self::signature::MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES;
|
|||||||
pub use self::signature::MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES;
|
pub use self::signature::MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES;
|
||||||
/// Extracts the first canonical 64-byte Solana signature from one complete Base64 transaction wire.
|
/// Extracts the first canonical 64-byte Solana signature from one complete Base64 transaction wire.
|
||||||
pub use self::signature::extract_raw_transaction_signature_from_binary_base64;
|
pub use self::signature::extract_raw_transaction_signature_from_binary_base64;
|
||||||
|
/// Encodes one canonical 64-byte Solana transaction signature as bounded Base58 text.
|
||||||
|
pub use self::signature::format_raw_transaction_signature;
|
||||||
/// Parses one bounded Base58 Solana transaction signature to exactly 64 canonical bytes.
|
/// Parses one bounded Base58 Solana transaction signature to exactly 64 canonical bytes.
|
||||||
pub use self::signature::parse_raw_transaction_signature;
|
pub use self::signature::parse_raw_transaction_signature;
|
||||||
/// One source-neutral v0 address-table lookup.
|
/// One source-neutral v0 address-table lookup.
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-raw-transaction-lib/src/signature.rs
|
// file: crates/ksp-raw-transaction-lib/src/signature.rs
|
||||||
// version: 3
|
// version: 4
|
||||||
|
|
||||||
use base64::Engine; // rust-rules: trait-import
|
use base64::Engine; // rust-rules: trait-import
|
||||||
|
|
||||||
@@ -8,6 +8,35 @@ pub const MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES: usize = 88;
|
|||||||
/// Minimum UTF-8 byte length admitted for one textual Base58 Solana transaction signature.
|
/// Minimum UTF-8 byte length admitted for one textual Base58 Solana transaction signature.
|
||||||
pub const MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES: usize = 64;
|
pub const MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES: usize = 64;
|
||||||
|
|
||||||
|
/// Encodes one canonical 64-byte Solana transaction signature as bounded Base58 text.
|
||||||
|
#[must_use]
|
||||||
|
pub fn format_raw_transaction_signature(signature: &ksp_store_api::RawTransactionSignature) -> std::string::String {
|
||||||
|
const ALPHABET: &[u8; 58] = b"123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz";
|
||||||
|
let bytes = signature.as_bytes();
|
||||||
|
let leading_zeroes = bytes.iter().take_while(|byte| return **byte == 0).count();
|
||||||
|
let mut digits = std::vec::Vec::with_capacity(crate::MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES);
|
||||||
|
for byte in bytes {
|
||||||
|
let mut carry = u32::from(*byte);
|
||||||
|
for digit in &mut digits {
|
||||||
|
let expanded = (u32::from(*digit) * 256) + carry;
|
||||||
|
*digit = (expanded % 58) as u8;
|
||||||
|
carry = expanded / 58;
|
||||||
|
}
|
||||||
|
while carry != 0 {
|
||||||
|
digits.push((carry % 58) as u8);
|
||||||
|
carry /= 58;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let mut output = std::string::String::with_capacity(leading_zeroes + digits.len());
|
||||||
|
for _ in 0..leading_zeroes {
|
||||||
|
output.push('1');
|
||||||
|
}
|
||||||
|
for digit in digits.iter().rev() {
|
||||||
|
output.push(char::from(ALPHABET[usize::from(*digit)]));
|
||||||
|
}
|
||||||
|
return output;
|
||||||
|
}
|
||||||
|
|
||||||
/// Parses one bounded Base58 Solana transaction signature to exactly 64 canonical bytes.
|
/// Parses one bounded Base58 Solana transaction signature to exactly 64 canonical bytes.
|
||||||
pub fn parse_raw_transaction_signature(value: &str) -> ksp_core_lib::Result<ksp_store_api::RawTransactionSignature> {
|
pub fn parse_raw_transaction_signature(value: &str) -> ksp_core_lib::Result<ksp_store_api::RawTransactionSignature> {
|
||||||
let text = value.as_bytes();
|
let text = value.as_bytes();
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-raw-transaction-lib/tests/public_api.rs
|
// file: crates/ksp-raw-transaction-lib/tests/public_api.rs
|
||||||
// version: 4
|
// version: 5
|
||||||
|
|
||||||
//! Integration canaries for the public common RAW Transaction crate-root surface.
|
//! Integration canaries for the public common RAW Transaction crate-root surface.
|
||||||
|
|
||||||
@@ -120,3 +120,16 @@ fn pre_006_source_neutral_wire_contract_is_available_from_crate_root() {
|
|||||||
}
|
}
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_signature_formatter_is_consumable_from_crate_root() {
|
||||||
|
let signature = ksp_store_api::RawTransactionSignature::new([0_u8; 64]);
|
||||||
|
let text = ksp_raw_transaction_lib::format_raw_transaction_signature(&signature);
|
||||||
|
assert_eq!(text, "1".repeat(64));
|
||||||
|
let parsed = ksp_raw_transaction_lib::parse_raw_transaction_signature(text.as_str());
|
||||||
|
assert!(parsed.is_ok());
|
||||||
|
if let std::result::Result::Ok(parsed) = parsed {
|
||||||
|
assert_eq!(parsed, signature);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-raw-transaction-lib/unit_tests/signature.rs
|
// file: crates/ksp-raw-transaction-lib/unit_tests/signature.rs
|
||||||
// version: 3
|
// version: 4
|
||||||
|
|
||||||
use base64::Engine; // rust-rules: trait-import
|
use base64::Engine; // rust-rules: trait-import
|
||||||
|
|
||||||
@@ -79,3 +79,19 @@ fn pre_006_embedded_signature_extracts_terminal_v1_signature_and_rejects_trailin
|
|||||||
assert!(crate::extract_raw_transaction_signature_from_binary_base64(trailing.as_str()).is_err());
|
assert!(crate::extract_raw_transaction_signature_from_binary_base64(trailing.as_str()).is_err());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_base58_formatter_round_trips_canonical_signatures() {
|
||||||
|
for bytes in [[0_u8; 64], [1_u8; 64], [0xAB_u8; 64]] {
|
||||||
|
let signature = ksp_store_api::RawTransactionSignature::new(bytes);
|
||||||
|
let text = crate::format_raw_transaction_signature(&signature);
|
||||||
|
assert!((crate::MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES..=crate::MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES).contains(&text.len()));
|
||||||
|
let parsed = crate::parse_raw_transaction_signature(text.as_str());
|
||||||
|
assert!(parsed.is_ok());
|
||||||
|
if let std::result::Result::Ok(parsed) = parsed {
|
||||||
|
assert_eq!(parsed, signature);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert_eq!(crate::format_raw_transaction_signature(&ksp_store_api::RawTransactionSignature::new([0_u8; 64])), "1".repeat(64));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|||||||
21
crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
Normal file
21
crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
# file: crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
|
||||||
|
# version: 2
|
||||||
|
|
||||||
|
[package]
|
||||||
|
name = "ksp-worker-raw-transaction-ingest-lib"
|
||||||
|
version.workspace = true
|
||||||
|
edition.workspace = true
|
||||||
|
repository.workspace = true
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
ksp-core-lib = { path = "../ksp-core-lib" }
|
||||||
|
ksp-logging-lib = { path = "../ksp-logging-lib" }
|
||||||
|
ksp-onchain-transport-lib = { path = "../ksp-onchain-transport-lib" }
|
||||||
|
ksp-raw-transaction-lib = { path = "../ksp-raw-transaction-lib" }
|
||||||
|
ksp-store-lib = { path = "../ksp-store-lib", default-features = false }
|
||||||
|
ksp-worker-api = { path = "../ksp-worker-api" }
|
||||||
|
sha2 = { workspace = true }
|
||||||
|
tokio = { workspace = true, features = ["macros", "rt", "sync", "time"] }
|
||||||
|
|
||||||
|
[lints]
|
||||||
|
workspace = true
|
||||||
319
crates/ksp-worker-raw-transaction-ingest-lib/README.md
Normal file
319
crates/ksp-worker-raw-transaction-ingest-lib/README.md
Normal file
@@ -0,0 +1,319 @@
|
|||||||
|
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/README.md -->
|
||||||
|
<!-- version: 11 -->
|
||||||
|
|
||||||
|
# ksp-worker-raw-transaction-ingest-lib
|
||||||
|
|
||||||
|
`ksp-worker-raw-transaction-ingest-lib` est le Worker concret KSP chargé de l'alimentation continue de la couche RAW Transaction.
|
||||||
|
|
||||||
|
La crate possède deux niveaux publics complémentaires :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestWorker::start
|
||||||
|
-> fondation source-neutral, sans source productive
|
||||||
|
|
||||||
|
RawTransactionIngestWorker::start_with_runtime_resources
|
||||||
|
-> même runtime + 1..32 sources productives supervisées simultanément
|
||||||
|
Yellowstone, WS standard logsSubscribe, WS standard blockSubscribe, Helius transactionSubscribe et/ou HTTP block polling
|
||||||
|
+ hydration HTTP getTransaction lorsque la source produit une référence
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker reste indépendant de Config et de tout backend Store physique. Le caller compose les ressources Transport et Store, puis les remet à la crate par ses façades publiques.
|
||||||
|
|
||||||
|
## Pipeline productif actuel
|
||||||
|
|
||||||
|
Les verticales live productives sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Yellowstone standard subscribe
|
||||||
|
-> Transaction / TransactionStatus / Block
|
||||||
|
-> signal source-neutral (network, signature, slot, commitment, provenance sûre)
|
||||||
|
|
||||||
|
Solana standard WS logsSubscribe
|
||||||
|
-> context.slot + signature
|
||||||
|
-> signal source-neutral (network, signature, slot, commitment, provenance sûre)
|
||||||
|
-> coalescence bornée par (network, signature, commitment)
|
||||||
|
-> HTTP getTransaction observed, Base64, maxSupportedTransactionVersion=1
|
||||||
|
|
||||||
|
Solana standard WS blockSubscribe
|
||||||
|
-> Full + Base64 + maxSupportedTransactionVersion=1 + showRewards=false
|
||||||
|
-> qualification explicite Legacy / V0 / V1
|
||||||
|
-> matériau Common RAW direct par transaction du bloc
|
||||||
|
|
||||||
|
Helius transactionSubscribe
|
||||||
|
-> Full + Base64 + maxSupportedTransactionVersion=1 + showRewards=false
|
||||||
|
-> signature + slot + transactionIndex uniquement dans le signal Worker
|
||||||
|
-> coalescence bornée par (network, signature, commitment)
|
||||||
|
-> HTTP getTransaction observed avant Common RAW
|
||||||
|
|
||||||
|
Solana HTTP live block polling
|
||||||
|
-> getSlot borne le run courant
|
||||||
|
-> getBlocksWithLimit découvre les blocs live
|
||||||
|
-> getBlock observed matérialise Full/Base64 Legacy/V0/V1
|
||||||
|
-> matériau Common RAW direct par transaction du bloc
|
||||||
|
|
||||||
|
les chemins productifs
|
||||||
|
-> ksp-raw-transaction-lib
|
||||||
|
-> admission centrale bornée
|
||||||
|
-> ksp-store-lib
|
||||||
|
-> RawTransaction + RawTransactionObservation
|
||||||
|
```
|
||||||
|
|
||||||
|
`BlockMeta` et `Slot` ne produisent pas de RAW directement. Ils servent uniquement à la projection de continuité du run.
|
||||||
|
|
||||||
|
La qualification reste conservative : même lorsqu'une update Yellowstone contient une transaction complète côté protobuf, le Worker hydrate actuellement les signaux transactionnels par HTTP `getTransaction` avant de construire le RAW canonique. Il n'existe donc pas de second canonicaliseur Yellowstone.
|
||||||
|
|
||||||
|
## Contrat de source Yellowstone + HTTP
|
||||||
|
|
||||||
|
`RawTransactionIngestYellowstoneSource::new` reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
YellowstoneGrpcChannel
|
||||||
|
YellowstoneSubscribeRequest
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName d'hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction est sans I/O et refuse notamment :
|
||||||
|
|
||||||
|
- une requête Yellowstone invalide ;
|
||||||
|
- l'absence de famille porteuse d'ingestion ;
|
||||||
|
- un commitment `Processed` ou implicite ;
|
||||||
|
- un réseau Yellowstone non représentable ;
|
||||||
|
- une provenance provider/endpoint non représentable ;
|
||||||
|
- l'absence d'une route HTTP compatible pour `getTransaction` sur le même réseau.
|
||||||
|
|
||||||
|
Le runtime-resource aggregate public accepte une collection validée de 1 à 32 sources logiques et les démarre simultanément sous un supervisor privé. La collection est validée entièrement avant spawn ; aucun sous-ensemble silencieux, source primaire implicite ou standby n'est choisi. La collection interne, les `source_key`, les URLs, les filtres et les clients inférieurs ne sont pas exposés.
|
||||||
|
|
||||||
|
Le supervisor possède toutes les tâches source. Une source qui échoue est terminale pour le Worker et déclenche l'arrêt coopératif puis le join des autres sources, car cette release ne suppose aucune équivalence de coverage. Une fermeture propre d'une source alors que le Worker n'est pas en arrêt est également traitée comme une perte de source configurée et devient terminale.
|
||||||
|
|
||||||
|
Un inventaire privé `source_key -> latest processing/source state`, borné à 32 entrées, agrège la projection run-local. La frontier agrégée reste conservative : elle n'expose un `processing_frontier_slot` que lorsque toutes les sources en possèdent un, choisit le minimum des frontiers connus et le plus ancien pending. Les sources reference-bearing partagent en plus un registre global d'hydration borné : une même clé `(network, signature, commitment)` ne déclenche qu'un leader HTTP, puis chaque signal source conserve sa propre provenance lors de la finalisation.
|
||||||
|
|
||||||
|
## Contrat de source Standard Logs + HTTP
|
||||||
|
|
||||||
|
`RawTransactionIngestStandardLogsSource::new` reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WsEndpointSettings kind solana_standard
|
||||||
|
SolanaLogsSubscribeFilter
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName d'hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction est sans I/O. Elle refuse un endpoint WS invalide ou non standard, `Processed`, un réseau/provenance non représentable et l'absence de route HTTP `getTransaction` compatible sur le même réseau. Le filtre `All`, `AllWithVotes` ou `Mentions(pubkey)` participe uniquement à une empreinte privée ; le pubkey d'un filtre `Mentions` n'est pas recopié dans `Debug`, snapshot ou provenance textuelle.
|
||||||
|
|
||||||
|
Au runtime, le Worker ouvre `SolanaStandardWsSession::connect`, puis `logs_subscribe`. Les lignes de logs et `err` restent dans Transport et ne sont jamais stockées dans le signal Worker. Seuls `context.slot` et `signature` sont projetés vers l'hydration commune. Reconnect, resubscribe et backpressure de la subscription restent possédés par Transport.
|
||||||
|
|
||||||
|
## Contrat de source Standard Block direct RAW
|
||||||
|
|
||||||
|
`RawTransactionIngestStandardBlockSource::new` reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WsEndpointSettings kind solana_standard
|
||||||
|
SolanaBlockSubscribeFilter
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
```
|
||||||
|
|
||||||
|
Le runtime ouvre la `SolanaStandardWsSession` existante et demande exactement `Base64`, `Full`, `maxSupportedTransactionVersion = 1` et `showRewards = false`. Aucun `HttpTransportPool` n'est attaché à cette source : les transactions dont la version est explicitement `Legacy`, `0` ou `1` sont transformées directement en `RawTransactionMaterial` à partir du wire Base64 du bloc, avec signature embarquée, slot, block time, meta, version et index de transaction.
|
||||||
|
|
||||||
|
La qualification est fermée : une version omise/nulle ou supérieure à `1`, une transaction non Base64, `block: null`, une erreur distante de notification, un slot de contexte incohérent ou un champ transactions absent/nul termine la source par une erreur sûre. Aucun de ces cas n'est converti en bloc vide, en succès silencieux ou en progression artificielle de frontier.
|
||||||
|
|
||||||
|
Pour un bloc multi-transaction, le slot n'est projeté settled qu'après l'admission réussie de toutes ses transactions. Cette voie RAW-direct n'incrémente pas `hydration_pending`; un stop ou une erreur au milieu du bloc ne produit aucune progression artificielle. Reconnect, resubscribe et backpressure WebSocket restent possédés par Transport.
|
||||||
|
|
||||||
|
## Contrat de source Helius Transaction + HTTP
|
||||||
|
|
||||||
|
`RawTransactionIngestHeliusTransactionSource::new` reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WsEndpointSettings kind helius_laserstream
|
||||||
|
HeliusTransactionSubscribeFilter
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName d'hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction est sans I/O. Elle réutilise le contrat Transport existant et impose `Full`, `Base64`, `showRewards = false` et `maxSupportedTransactionVersion = 1`. Le Worker ne lit pas Config, ne lit pas `KSP_SECRET_HELIUS_API_KEY` et ne code aucun tier provider ; l'URL résolue et le credential restent dans l'endpoint Transport fourni par le caller.
|
||||||
|
|
||||||
|
Au runtime, `HeliusLaserStreamWsSession::connect` puis `transaction_subscribe` sont utilisés. Seule une notification `Full` conforme au mode demandé est admise ; une forme `Signature`, `Unknown` ou future devient une faute source sûre. Le payload Helius `transaction` n'est jamais copié dans l'état Worker : la projection conserve uniquement signature, slot et transaction index, puis réutilise le coordinateur d'hydration commun `getTransaction observed`.
|
||||||
|
|
||||||
|
La clé logique Helius inclut réseau, identités provider/endpoint sûres, commitment et empreinte privée du filtre. Les listes de pubkeys du filtre sont normalisées avant hash afin que leur ordre ne crée pas artificiellement deux sources logiques ; le rôle HTTP d'hydration reste exclu de l'identité live.
|
||||||
|
|
||||||
|
## Contrat de source HTTP Block Polling
|
||||||
|
|
||||||
|
`RawTransactionIngestHttpBlockPollingSource::new` reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName de polling
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction est sans I/O et vérifie que le rôle HTTP possède, sur un seul réseau, les capacités `getSlot`, `getBlocksWithLimit` et `getBlock`. La cadence est bornée entre 100 ms et 30 s, avec 1 s par défaut ; la découverte est bornée entre 1 et 1024 blocs par cycle, avec 128 par défaut. Ces réglages de cadence ne font pas partie de l'identité logique de la source.
|
||||||
|
|
||||||
|
Au démarrage, le premier `getSlot` fixe la borne inférieure du run. Le Worker ne demande aucun slot antérieur. Chaque cycle relit le tip, découvre les blocs disponibles avec `getBlocksWithLimit`, puis matérialise chaque slot listé par `getBlock observed` en `Full + Base64 + maxSupportedTransactionVersion = 1 + showRewards = false`. Seules les transactions Legacy/V0/V1 explicitement qualifiées entrent directement dans Common RAW.
|
||||||
|
|
||||||
|
Un slot listé dont `getBlock` retourne `null` reste la tête de reprise du cycle suivant ; il n'est ni considéré vide ni marqué settled. Les slots absents de la liste de découverte sont traités comme non produits/skipped pour ce run. Le slot n'est settled qu'après admission réussie de toutes ses transactions. Le polling reste run-local : aucun checkpoint durable, aucun scan avant la borne initiale et aucun Backfill implicite ne sont créés. Les retries/reroutages HTTP restent possédés par `ksp-onchain-transport-lib`.
|
||||||
|
|
||||||
|
## Runtime et lifecycle
|
||||||
|
|
||||||
|
Le Worker s'exécute sur le runtime Tokio courant du caller. Il ne crée pas de runtime global et n'expose aucun `JoinHandle` public.
|
||||||
|
|
||||||
|
`RawTransactionIngestHandle` permet de :
|
||||||
|
|
||||||
|
- demander un stop coopératif et idempotent ;
|
||||||
|
- lire une source de snapshots concrets latest-value ;
|
||||||
|
- utiliser la même source via `WorkerSnapshotSource` ;
|
||||||
|
- attendre le terminal après drain et join des tâches possédées.
|
||||||
|
|
||||||
|
Le shutdown est borné par `shutdown_drain_timeout`. Le supervisor multi-source relaie le stop à toutes les sources et les rejoint avant de rendre son résultat au supervisor Worker ; les tâches source, hydration et persistence possédées sont ensuite drainées ou abort+join avant publication terminale. Si la deadline expire, l'abort du wrapper source détruit aussi son `JoinSet` interne et annule ses tâches imbriquées avant le terminal. Une faute déjà observée n'est pas remplacée par un stop concurrent, sauf le `drain_timeout` terminal lorsqu'une récupération bornée dépasse sa deadline. L'abandon terminal d'une hydration retire son pending run-local sans le convertir artificiellement en travail `settled`.
|
||||||
|
|
||||||
|
## Admission, coalescence et backpressure
|
||||||
|
|
||||||
|
La queue centrale est un `tokio::sync::mpsc` privé borné par `admission_queue_capacity`. Les sources internes subissent la backpressure ; aucune queue non bornée ni silent drop n'est autorisé.
|
||||||
|
|
||||||
|
Le Worker possède des coordinateurs source-neutral Yellowstone, Standard Logs et Helius Transaction branchés sur un registre global d'hydration partagé. Standard Block et HTTP Block Polling n'entrent pas dans ce registre lorsqu'une transaction est direct-qualified.
|
||||||
|
|
||||||
|
Les sources reference-bearing reçoivent des quotas déterministes dont la somme reste exactement dans les bornes techniques configurées :
|
||||||
|
|
||||||
|
```text
|
||||||
|
somme pending source signals <= admission_queue_capacity
|
||||||
|
somme hydration tasks in flight <= persistence_concurrency
|
||||||
|
source reference-bearing active => quota pending >= 1 et quota in-flight >= 1
|
||||||
|
```
|
||||||
|
|
||||||
|
Pour garantir simultanément ces bornes et l'absence de starvation structurelle, le démarrage échoue avant spawn si le nombre de sources reference-bearing dépasse `admission_queue_capacity` ou `persistence_concurrency`. Un sémaphore global protège en plus l'ouverture effective des hydrations HTTP.
|
||||||
|
|
||||||
|
Les signaux partageant le même `(network, signature, commitment)` sont coalescés cross-source avant le fan-out HTTP. La publication du résultat partagé notifie les followers sous le verrou de registry avant de retirer la clé : une nouvelle génération de leader ne peut donc pas s'intercaler entre retrait et notification. Après canonicalisation, une cache run-local bornée sérialise les acquisitions de même `(network, signature)` : la première passe par l'écriture atomique entity + observation, les suivantes de contenu canonique identique ajoutent uniquement leur observation déterministe. Une divergence de slot, block time, format ou hash canonique devient un content conflict terminal ; aucune majorité, préférence provider ou overwrite n'est appliqué. Le Store conserve son guard durable final.
|
||||||
|
|
||||||
|
Les retries/reroutages HTTP appartiennent à `ksp-onchain-transport-lib`. Le Worker ne possède pas une seconde boucle de retry autour de `getTransaction`.
|
||||||
|
|
||||||
|
## Processing frontier run-local
|
||||||
|
|
||||||
|
Le snapshot expose :
|
||||||
|
|
||||||
|
```text
|
||||||
|
hydration_pending
|
||||||
|
processing_frontier_slot
|
||||||
|
oldest_pending_slot
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette frontier mesure uniquement le traitement des signaux réellement observés pendant le run courant. Elle n'est ni un checkpoint durable, ni une preuve de complétude blockchain, ni un curseur de Backfill.
|
||||||
|
|
||||||
|
Un signal transactionnel devient pending après validation de sa clé d'hydration et insertion dans le coordinateur. Il devient settled pour la source lorsque :
|
||||||
|
|
||||||
|
```text
|
||||||
|
getTransaction -> Missing
|
||||||
|
ou
|
||||||
|
getTransaction -> Available puis ingress envoyé avec succès vers l'admission centrale
|
||||||
|
```
|
||||||
|
|
||||||
|
Un `BlockMeta` ou `Slot` continuity-only est settled localement sans produire de RAW. La frontier n'avance jamais à travers le plus ancien pending connu.
|
||||||
|
|
||||||
|
## Reconnect, replay et continuité
|
||||||
|
|
||||||
|
Le reconnect/replay Yellowstone appartient à Transport. Le Worker n'écrit pas `from_slot` et n'interprète pas directement `SubscribeReplayInfo`.
|
||||||
|
|
||||||
|
Le snapshot Worker projette uniquement des informations sûres et source-neutral :
|
||||||
|
|
||||||
|
```text
|
||||||
|
source_total
|
||||||
|
source_active
|
||||||
|
source_reconnecting
|
||||||
|
source_failed
|
||||||
|
source_state
|
||||||
|
source_failure_total
|
||||||
|
backpressure_wait_total
|
||||||
|
source_reconnect_total
|
||||||
|
source_replay_attempt_total
|
||||||
|
source_continuity_gap_total
|
||||||
|
```
|
||||||
|
|
||||||
|
Les quatre comptes de sources sont des gauges latest-value et ne contiennent aucune identité de source. `source_total` reste le nombre de sources logiques configurées pour le run. Tant que toutes les sources attendues ne sont pas `Active`, la health publique reste conservative : un reconnect transitoire projette `Degraded`, une source `Failed` projette `Unhealthy`, et le retour de toutes les sources attendues à `Active` permet de revenir à `Healthy`.
|
||||||
|
|
||||||
|
`RawTransactionIngestSourceState` distingue :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Active
|
||||||
|
Reconnecting
|
||||||
|
Closing
|
||||||
|
Closed
|
||||||
|
Failed
|
||||||
|
```
|
||||||
|
|
||||||
|
Un replay attempt n'est pas une preuve de replay réussi ni de continuité parfaite. Si Transport augmente son compteur de continuity gap parce que la borne de rétention prouve que le slot demandé n'est plus rejouable, le Worker classe la source en failure et termine avec `worker_raw_transaction_ingest.source_failed`.
|
||||||
|
|
||||||
|
Le Worker ne lance alors ni Job Backfill ni campagne historique automatique.
|
||||||
|
|
||||||
|
## Persistence Store
|
||||||
|
|
||||||
|
La persistance passe exclusivement par `ksp-store-lib` avec `default-features = false`. Aucun backend physique n'est importé directement.
|
||||||
|
|
||||||
|
L'écriture utilise le mode normal atomique `RawTransaction + RawTransactionObservation`. Les outcomes distingués incluent :
|
||||||
|
|
||||||
|
```text
|
||||||
|
entity inserted
|
||||||
|
entity already present
|
||||||
|
entity skipped purged
|
||||||
|
observation inserted
|
||||||
|
observation already present
|
||||||
|
observation not recorded for purged entity
|
||||||
|
content conflict
|
||||||
|
store failure
|
||||||
|
```
|
||||||
|
|
||||||
|
Un content conflict est terminal et n'est jamais converti en succès idempotent.
|
||||||
|
|
||||||
|
## Snapshots et erreurs
|
||||||
|
|
||||||
|
`RawTransactionIngestSnapshotSource` est latest-value : les lecteurs peuvent rater des transitions intermédiaires mais récupèrent toujours la dernière projection complète et monotone.
|
||||||
|
|
||||||
|
Les codes Worker publics sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.settings_invalid
|
||||||
|
worker_raw_transaction_ingest.runtime_invalid
|
||||||
|
worker_raw_transaction_ingest.store_failed
|
||||||
|
worker_raw_transaction_ingest.content_conflict
|
||||||
|
worker_raw_transaction_ingest.counter_exhausted
|
||||||
|
worker_raw_transaction_ingest.source_failed
|
||||||
|
worker_raw_transaction_ingest.drain_timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
Les diagnostics et `Debug` ne recopient pas de payload RAW, signature, URL, credential, filtre provider, texte backend/provider arbitraire ou client inférieur. Les agrégations de compteurs de continuité multi-source utilisent une arithmétique vérifiée ; un overflow devient `counter_exhausted` au lieu d'être saturé silencieusement.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
Les dépendances normales sont exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-core-lib
|
||||||
|
ksp-logging-lib
|
||||||
|
ksp-onchain-transport-lib
|
||||||
|
ksp-raw-transaction-lib
|
||||||
|
ksp-store-lib (default-features = false)
|
||||||
|
ksp-worker-api
|
||||||
|
sha2
|
||||||
|
tokio (macros, rt, sync, time)
|
||||||
|
```
|
||||||
|
|
||||||
|
La crate ne dépend pas de Config, Job, `ksp-store-api` directement, backend Store concret, `reqwest`, `tonic`, `yellowstone-grpc-proto` ou Tauri.
|
||||||
|
|
||||||
|
## Hors périmètre
|
||||||
|
|
||||||
|
La verticale actuelle ne possède pas :
|
||||||
|
|
||||||
|
- sélection Config interne, reconfiguration dynamique ou policy de failover entre sources ;
|
||||||
|
- checkpoint persistent de processing frontier ;
|
||||||
|
- campagne de réparation historique automatique ;
|
||||||
|
- application Desk ou process autonome ;
|
||||||
|
- décodage STRUCTURAL/DECODED/DOMAIN.
|
||||||
|
|
||||||
|
Les futures extensions doivent conserver la séparation avec `ksp-job-backfill-lib` et réutiliser les mêmes contrats Common RAW/Store.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- [`USAGE.md`](USAGE.md) — utilisation de la façade publique ;
|
||||||
|
- [`../ksp-worker-api/README.md`](../ksp-worker-api/README.md) — contrats Worker génériques ;
|
||||||
|
- [`../ksp-raw-transaction-lib/README.md`](../ksp-raw-transaction-lib/README.md) — canonicalisation RAW commune ;
|
||||||
|
- [`../../docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md`](../../docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md) — séparation Worker/Job ;
|
||||||
|
- [`../../docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md`](../../docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md) — architecture d'acquisition RawTransaction.
|
||||||
422
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
Normal file
422
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
Normal file
@@ -0,0 +1,422 @@
|
|||||||
|
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md -->
|
||||||
|
<!-- version: 11 -->
|
||||||
|
|
||||||
|
# Utilisation de ksp-worker-raw-transaction-ingest-lib
|
||||||
|
|
||||||
|
Cette page décrit la façade publique de `ksp-worker-raw-transaction-ingest-lib`. Le caller possède la composition du runtime Tokio, du `Store` et des ressources Transport ; le Worker ne lit pas Config, ne construit pas un backend physique et ne lit pas de secret depuis l'environnement.
|
||||||
|
|
||||||
|
## Construire les settings
|
||||||
|
|
||||||
|
Les identités sont validées par leurs couches propriétaires :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn worker_settings() -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-mainnet-0001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::with_defaults(network, worker_id));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker kind est fixe :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE, "raw_transaction_ingest");
|
||||||
|
```
|
||||||
|
|
||||||
|
Pour des limites explicites :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let settings = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::new(
|
||||||
|
network,
|
||||||
|
worker_id,
|
||||||
|
512,
|
||||||
|
16,
|
||||||
|
std::time::Duration::from_secs(10),
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
Bornes publiques :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission_queue_capacity 1 ..= 65_536 défaut 256
|
||||||
|
persistence_concurrency 1 ..= 64 défaut 8
|
||||||
|
shutdown_drain_timeout 100 ms ..= 30 s défaut 5 s
|
||||||
|
```
|
||||||
|
|
||||||
|
Une valeur hors borne retourne `worker_raw_transaction_ingest.settings_invalid` avec uniquement le nom stable du champ invalide.
|
||||||
|
|
||||||
|
## Choisir le mode de démarrage
|
||||||
|
|
||||||
|
### Fondation sans source productive
|
||||||
|
|
||||||
|
`RawTransactionIngestWorker::start(settings, store)` démarre la fondation runtime sans source Transport. Ce mode reste utile aux tests/compositions qui veulent uniquement le lifecycle, les snapshots et le contrat de shutdown.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let handle = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWorker::start(settings, store) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
### Source Yellowstone productive
|
||||||
|
|
||||||
|
Pour l'ingestion live, le caller compose d'abord les ressources via les crates propriétaires, puis construit :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn runtime_resources(
|
||||||
|
yellowstone_channel: ksp_onchain_transport_lib::YellowstoneGrpcChannel,
|
||||||
|
subscribe_request: ksp_onchain_transport_lib::YellowstoneSubscribeRequest,
|
||||||
|
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources> {
|
||||||
|
let source = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestYellowstoneSource::new(
|
||||||
|
yellowstone_channel,
|
||||||
|
subscribe_request,
|
||||||
|
http_pool,
|
||||||
|
hydration_role,
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::new(source));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Puis :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let handle = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWorker::start_with_runtime_resources(
|
||||||
|
settings,
|
||||||
|
store,
|
||||||
|
runtime_resources,
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Les deux entrées exigent un runtime Tokio courant et un `Store` portant exactement le même `RawNetworkId` que les settings. Le démarrage avec ressources exige également que chaque source composée cible ce même réseau. L'agrégat accepte 1 à 32 sources logiques et les lance simultanément ; les doublons d'identité et les mélanges de réseaux sont refusés avant spawn. Il n'existe pas de source primaire, standby ou fallback implicite : toute source configurée fait partie du run.
|
||||||
|
|
||||||
|
### Composer plusieurs sources simultanées
|
||||||
|
|
||||||
|
Une fois un premier `RawTransactionIngestRuntimeResources` construit, le caller ajoute les autres sources avec les méthodes `try_push_*` correspondant à leur capability. Exemple conceptuel :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let mut resources = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::new(yellowstone_source);
|
||||||
|
if let std::result::Result::Err(error) = resources.try_push_standard_logs_source(standard_logs_source) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = resources.try_push_http_block_polling_source(http_polling_source) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
La validation est transactionnelle à chaque ajout : limite globale 32, réseau unique et `source_key` logique unique. Au démarrage, toutes les sources présentes sont supervisées ensemble. La défaillance d'une source est terminale pour le Worker ; les autres sources sont arrêtées et jointes, car le Worker ne suppose pas qu'elles couvrent les mêmes filtres ou le même univers de transactions.
|
||||||
|
|
||||||
|
### Source Standard Logs productive
|
||||||
|
|
||||||
|
Une source standard Solana WS se compose ainsi :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn standard_logs_runtime_resources(
|
||||||
|
ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
filter: ksp_onchain_transport_lib::SolanaLogsSubscribeFilter,
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources> {
|
||||||
|
let source = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource::new(
|
||||||
|
ws_endpoint,
|
||||||
|
filter,
|
||||||
|
commitment,
|
||||||
|
http_pool,
|
||||||
|
hydration_role,
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_standard_logs_source(source));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`ws_endpoint` doit être un endpoint Transport valide de kind `solana_standard`. Le commitment doit être explicitement `Confirmed` ou `Finalized`. Le pool HTTP doit exposer `getTransaction` via le rôle indiqué sur le même réseau. `All`, `AllWithVotes` et `Mentions(pubkey)` sont acceptés par le contrat Transport ; la valeur du filtre reste privée dans le Worker.
|
||||||
|
|
||||||
|
### Source Standard Block productive
|
||||||
|
|
||||||
|
Une source `blockSubscribe` standard se compose sans pool HTTP :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn standard_block_runtime_resources(
|
||||||
|
ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
filter: ksp_onchain_transport_lib::SolanaBlockSubscribeFilter,
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources> {
|
||||||
|
let source = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardBlockSource::new(ws_endpoint, filter, commitment) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_standard_block_source(source));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker demande `Base64`, `Full`, `maxSupportedTransactionVersion = 1` et `showRewards = false`. Une transaction n'est RAW-direct que si sa version est explicitement `Legacy`, `0` ou `1`. Une version omise/nulle ou supérieure, `block: null`, une erreur de bloc, un champ transactions absent/nul ou une transaction non Base64 provoque une faute sûre ; ces cas ne sont jamais assimilés à une progression vide.
|
||||||
|
|
||||||
|
### Source Helius Transaction productive
|
||||||
|
|
||||||
|
Une source Helius LaserStream `transactionSubscribe` se compose avec le même pattern caller-owned :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn helius_transaction_runtime_resources(
|
||||||
|
ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
filter: ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter,
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources> {
|
||||||
|
let source = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource::new(
|
||||||
|
ws_endpoint,
|
||||||
|
filter,
|
||||||
|
commitment,
|
||||||
|
http_pool,
|
||||||
|
hydration_role,
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_helius_transaction_source(source));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`ws_endpoint` doit être un endpoint Transport de kind `helius_laserstream`. Le caller supérieur résout éventuellement `KSP_SECRET_HELIUS_API_KEY` via Config avant de construire l'endpoint ; le Worker ne lit jamais l'environnement ni Config. Le commitment est limité à `Confirmed`/`Finalized` et la route HTTP doit supporter `getTransaction` sur le même réseau.
|
||||||
|
|
||||||
|
Le Worker demande la forme Helius `Full` avec `Base64`, `showRewards = false` et `maxSupportedTransactionVersion = 1`, mais ne fait pas confiance au nested payload pour construire directement le Common RAW. Il conserve seulement signature/slot/index et hydrate par `getTransaction observed`. Une notification d'une autre forme est fail-closed.
|
||||||
|
|
||||||
|
### Source HTTP Block Polling productive
|
||||||
|
|
||||||
|
Une source HTTP live peut être composée sans WebSocket ni gRPC :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn http_block_polling_runtime_resources(
|
||||||
|
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
polling_role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources> {
|
||||||
|
let source = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource::new(
|
||||||
|
http_pool,
|
||||||
|
polling_role,
|
||||||
|
commitment,
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_http_block_polling_source(source));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le rôle HTTP doit disposer, sur un même réseau, de `getSlot`, `getBlocksWithLimit` et `getBlock`. Le commitment est limité à `Confirmed`/`Finalized`. Par défaut, le Worker interroge toutes les secondes et borne la découverte à 128 blocs par cycle. `new_with_limits` permet de choisir une cadence entre 100 ms et 30 s et une limite entre 1 et 1024 blocs par cycle ; les limites de débit physiques restent celles de Transport.
|
||||||
|
|
||||||
|
Au démarrage du run, la première valeur `getSlot(commitment)` devient la borne inférieure stricte du poller. Il ne demande jamais de slot antérieur. `getBlocksWithLimit` détermine les slots réellement matérialisables, puis `getBlock observed` produit directement le Common RAW en Full/Base64 pour Legacy/V0/V1. Un slot listé dont `getBlock` retourne `null` reste la tête de reprise du prochain cycle et n'est jamais transformé en progression silencieuse.
|
||||||
|
|
||||||
|
## Préparer la source Yellowstone
|
||||||
|
|
||||||
|
La `YellowstoneSubscribeRequest` doit :
|
||||||
|
|
||||||
|
- être valide selon Transport ;
|
||||||
|
- contenir au moins une famille transaction-bearing admise par le Worker ;
|
||||||
|
- utiliser explicitement `Confirmed` ou `Finalized` ;
|
||||||
|
- rester compatible avec le réseau du `YellowstoneGrpcChannel`.
|
||||||
|
|
||||||
|
Le `HttpTransportPool` doit posséder au moins une route compatible avec le rôle d'hydration et `getTransaction` sur le même réseau. La construction de `RawTransactionIngestYellowstoneSource` vérifie ces invariants sans ouvrir la connexion réseau.
|
||||||
|
|
||||||
|
Le caller ne passe pas de signature, `program_id`, plage de slots ou limite historique au Worker. Ces paramètres appartiennent à un Job Backfill, pas au service continu.
|
||||||
|
|
||||||
|
## Comprendre la pipeline live
|
||||||
|
|
||||||
|
Les familles productives sont traitées ainsi :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Yellowstone Transaction -> signal -> HTTP getTransaction -> Common RAW -> admission
|
||||||
|
Yellowstone TransactionStatus -> signal -> HTTP getTransaction -> Common RAW -> admission
|
||||||
|
Yellowstone Block -> un signal par transaction -> HTTP getTransaction -> Common RAW -> admission
|
||||||
|
Standard WS logsSubscribe -> context.slot + signature -> HTTP getTransaction -> Common RAW -> admission
|
||||||
|
Standard WS blockSubscribe -> Full/Base64 Legacy|V0|V1 -> Common RAW direct par transaction -> admission
|
||||||
|
Helius transactionSubscribe -> Full envelope -> signature/slot/index -> HTTP getTransaction -> Common RAW -> admission
|
||||||
|
HTTP live block polling -> getSlot -> getBlocksWithLimit -> getBlock observed -> Common RAW direct Legacy|V0|V1 -> admission
|
||||||
|
Yellowstone BlockMeta -> continuity-only
|
||||||
|
Yellowstone Slot -> continuity-only
|
||||||
|
Yellowstone Account/Ping/Pong/Entry -> sans RAW Transaction dans cette verticale
|
||||||
|
```
|
||||||
|
|
||||||
|
Les signaux de même `(network, signature, commitment)` sont coalescés globalement avant l'hydration HTTP. Si plusieurs sources produisent ensuite la même transaction canonique, le Worker conserve une seule entité RAW et enregistre séparément les observations déterministes propres à chaque source. Le Worker ne possède pas une boucle de retry HTTP : reroutage/retry/backoff restent dans `ksp-onchain-transport-lib`.
|
||||||
|
|
||||||
|
Les sources qui nécessitent `getTransaction` partagent des quotas bornés et déterministes. Pour une composition valide :
|
||||||
|
|
||||||
|
```text
|
||||||
|
nombre de sources avec hydration <= admission_queue_capacity
|
||||||
|
nombre de sources avec hydration <= persistence_concurrency
|
||||||
|
somme de leurs pending <= admission_queue_capacity
|
||||||
|
somme de leurs tâches d'hydration actives <= persistence_concurrency
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces contraintes garantissent au moins une part à chaque source reference-bearing sans introduire de scheduler pondéré. Si les settings ne permettent pas cette répartition, le démarrage échoue avant spawn avec `runtime_invalid`; le caller doit augmenter la capacité concernée ou réduire le nombre de sources nécessitant une hydration.
|
||||||
|
|
||||||
|
Pour une même identité `(network, signature)`, une divergence canonique de slot, block time, format ou hash est un `content_conflict`. Le Worker ne choisit ni majorité ni provider préféré et n'écrase pas un contenu divergent ; le Store reste l'autorité durable finale du conflit.
|
||||||
|
|
||||||
|
## Observer le snapshot concret
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let current = source.current();
|
||||||
|
|
||||||
|
let sequence = current.worker_snapshot().sequence();
|
||||||
|
let state = current.worker_snapshot().state();
|
||||||
|
let queue_depth = current.admission_queue_depth();
|
||||||
|
let in_flight = current.in_flight_persistence();
|
||||||
|
let hydration_pending = current.hydration_pending();
|
||||||
|
let frontier = current.processing_frontier_slot();
|
||||||
|
let oldest_pending = current.oldest_pending_slot();
|
||||||
|
let source_total = current.source_total();
|
||||||
|
let source_active = current.source_active();
|
||||||
|
let source_reconnecting = current.source_reconnecting();
|
||||||
|
let source_failed = current.source_failed();
|
||||||
|
let source_state = current.source_state();
|
||||||
|
```
|
||||||
|
|
||||||
|
Pour attendre une valeur plus récente :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let observed = source.current().worker_snapshot().sequence();
|
||||||
|
let newer = source.wait_for_change(observed).await;
|
||||||
|
assert!(newer.worker_snapshot().sequence().is_after(observed));
|
||||||
|
```
|
||||||
|
|
||||||
|
Le flux est latest-value : les transitions intermédiaires peuvent être coalescées. Ne pas l'utiliser comme journal exhaustif.
|
||||||
|
|
||||||
|
## Utiliser la projection Worker API
|
||||||
|
|
||||||
|
La même source implémente `ksp_worker_api::WorkerSnapshotSource` :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let source = handle.worker_snapshot_source();
|
||||||
|
let current = ksp_worker_api::WorkerSnapshotSource::current(&source);
|
||||||
|
let observed = current.sequence();
|
||||||
|
let newer = ksp_worker_api::WorkerSnapshotSource::wait_for_change(&source, observed).await;
|
||||||
|
assert!(newer.sequence().is_after(observed));
|
||||||
|
```
|
||||||
|
|
||||||
|
La projection commune contient seulement les dimensions génériques Worker. Les compteurs et frontiers spécifiques restent sur `RawTransactionIngestSnapshot`.
|
||||||
|
|
||||||
|
## Lire les compteurs
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
|
||||||
|
let admitted = snapshot.admitted_total();
|
||||||
|
let canonicalized = snapshot.canonicalized_total();
|
||||||
|
let persisted = snapshot.persisted_total();
|
||||||
|
let inserted = snapshot.entity_inserted_total();
|
||||||
|
let existing = snapshot.entity_already_present_total();
|
||||||
|
let purged = snapshot.entity_skipped_purged_total();
|
||||||
|
let observations = snapshot.observation_inserted_total();
|
||||||
|
let conflicts = snapshot.content_conflict_total();
|
||||||
|
let store_failures = snapshot.store_failure_total();
|
||||||
|
let source_failures = snapshot.source_failure_total();
|
||||||
|
let backpressure = snapshot.backpressure_wait_total();
|
||||||
|
let reconnects = snapshot.source_reconnect_total();
|
||||||
|
let replay_attempts = snapshot.source_replay_attempt_total();
|
||||||
|
let proven_gaps = snapshot.source_continuity_gap_total();
|
||||||
|
```
|
||||||
|
|
||||||
|
`admission_queue_depth()`, `in_flight_persistence()`, `hydration_pending()`, `source_total()`, `source_active()`, `source_reconnecting()` et `source_failed()` sont des gauges latest-value. Les compteurs cumulés et les agrégats multi-source ne wrapent ni ne saturent silencieusement ; l'épuisement est terminal avec `worker_raw_transaction_ingest.counter_exhausted`.
|
||||||
|
|
||||||
|
## Interpréter la processing frontier
|
||||||
|
|
||||||
|
`processing_frontier_slot()` est la plus haute slot de travail source réellement observé qui n'est pas bloquée par un pending plus ancien connu. `oldest_pending_slot()` expose ce plus ancien pending lorsqu'il existe.
|
||||||
|
|
||||||
|
Cette frontier est strictement run-local :
|
||||||
|
|
||||||
|
```text
|
||||||
|
elle ne prouve pas que toutes les transactions blockchain d'une slot ont été observées
|
||||||
|
elle ne prouve pas la persistence durable des ingress déjà envoyés à l'admission
|
||||||
|
elle n'est pas persistée entre deux runs
|
||||||
|
elle n'est pas un checkpoint Backfill
|
||||||
|
```
|
||||||
|
|
||||||
|
Un `Missing` HTTP règle le signal du point de vue source-processing sans créer de RAW. Un ingress `Available` n'est réglé qu'après envoi réussi vers l'admission centrale. Une hydration abandonnée au shutdown est retirée des pending sans faire avancer artificiellement la frontier.
|
||||||
|
|
||||||
|
## Interpréter reconnect et replay
|
||||||
|
|
||||||
|
`source_state()` peut retourner `Active`, `Reconnecting`, `Closing`, `Closed` ou `Failed` comme état agrégé source-neutral. Les gauges `source_total()`, `source_active()`, `source_reconnecting()` et `source_failed()` permettent d'interpréter une composition multi-source sans exposer provider, endpoint, filtre ou `source_key`.
|
||||||
|
|
||||||
|
La health commune est conservative : pendant `Running`, au moins une source `Failed` donne `Unhealthy`; au moins une source `Reconnecting`, ou une composition dont toutes les sources attendues ne sont pas encore `Active`, donne `Degraded`; toutes les sources attendues `Active` permettent `Healthy`. Une transition Worker `Faulted` reste `Unhealthy`.
|
||||||
|
|
||||||
|
Les compteurs ont des sémantiques distinctes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
source_reconnect_total reconnects automatiques réussis observés
|
||||||
|
source_replay_attempt_total tentatives de reconnect portant une demande de replay
|
||||||
|
source_continuity_gap_total gaps de rétention prouvés par Transport
|
||||||
|
```
|
||||||
|
|
||||||
|
Une tentative de replay n'est pas une preuve de continuité. Le Worker ne choisit pas `from_slot` et ne traite pas directement `SubscribeReplayInfo` ; ces mécanismes appartiennent à Transport.
|
||||||
|
|
||||||
|
Si `source_continuity_gap_total` augmente, le Worker fault avec `worker_raw_transaction_ingest.source_failed`. Il ne déclenche pas automatiquement `ksp-job-backfill-lib`.
|
||||||
|
|
||||||
|
## Demander un stop et attendre le terminal
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let accepted = handle.request_stop();
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
|
||||||
|
match terminal {
|
||||||
|
std::result::Result::Ok(state) => {
|
||||||
|
assert!(state.is_terminal());
|
||||||
|
}
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
}
|
||||||
|
|
||||||
|
if accepted {
|
||||||
|
// La première demande de stop a été remise au runtime vivant.
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`request_stop()` est idempotent. Le terminal n'est publié qu'après le drain borné et la récupération des tâches possédées. Si la deadline de drain expire, toutes les tâches source/persistence encore possédées sont abortées puis jointes avant publication terminale ; une persistence libérée après ce terminal ne peut donc pas produire une complétion tardive. Une faute source déjà observée reste prioritaire face à un stop concurrent, sauf si le drain lui-même expire et devient le terminal `drain_timeout`.
|
||||||
|
|
||||||
|
## Interpréter les faults
|
||||||
|
|
||||||
|
```text
|
||||||
|
settings_invalid settings techniques hors contrat
|
||||||
|
runtime_invalid invariant runtime/lifecycle impossible
|
||||||
|
store_failed erreur Store non-conflict classifiée
|
||||||
|
content_conflict contenu canonique incompatible avec l'identité durable
|
||||||
|
counter_exhausted compteur/séquence monotone arrivé à sa borne
|
||||||
|
source_failed source/replay/hydration terminée par une erreur classifiée
|
||||||
|
drain_timeout drain de shutdown hors deadline
|
||||||
|
```
|
||||||
|
|
||||||
|
Un consumer doit traiter l'`ErrorCode` comme contrat stable et ne pas dépendre d'un texte backend/provider arbitraire.
|
||||||
|
|
||||||
|
## Composition supérieure
|
||||||
|
|
||||||
|
Le pattern attendu est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Config / application / service owner
|
||||||
|
-> résout endpoints, credentials et rôles
|
||||||
|
-> construit une source Transport Yellowstone, Standard Logs, Standard Block, Helius Transaction ou HTTP Block Polling
|
||||||
|
-> construit HttpTransportPool + rôle HTTP adapté à la source
|
||||||
|
-> construit le Store
|
||||||
|
-> construit RawTransactionIngestRuntimeResources
|
||||||
|
-> construit RawTransactionIngestSettings
|
||||||
|
-> démarre RawTransactionIngestWorker::start_with_runtime_resources
|
||||||
|
-> observe RawTransactionIngestSnapshotSource
|
||||||
|
-> demande stop lorsque nécessaire
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker ne reçoit pas de requête historique métier et ne dépend pas de Config. Une campagne `signature/program_id/plage/limite` appartient à `ksp-job-backfill-lib`.
|
||||||
110
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
Normal file
110
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
use sha2::Digest; // rust-rules: trait-import
|
||||||
|
|
||||||
|
const RAW_TRANSACTION_INGEST_OBSERVATION_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.observation.v1\0";
|
||||||
|
|
||||||
|
/// Crate-private source-neutral ingress admitted by the central bounded Worker queue.
|
||||||
|
pub(crate) struct RawTransactionIngress {
|
||||||
|
/// Complete Common RAW material supplied by one private source task.
|
||||||
|
pub(crate) material: ksp_raw_transaction_lib::RawTransactionMaterial,
|
||||||
|
/// Logical network expected to match both Worker settings and canonical material.
|
||||||
|
pub(crate) network: ksp_store_lib::RawNetworkId,
|
||||||
|
/// Safe source-independent provenance attached to the acquisition observation.
|
||||||
|
pub(crate) provenance: ksp_store_lib::RawAcquisitionProvenance,
|
||||||
|
/// Opaque deterministic source-owned key material used only for Worker observation-key derivation.
|
||||||
|
pub(crate) source_key: [u8; 32],
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Receiver side of the bounded central RAW transaction admission queue owned by the Worker supervisor.
|
||||||
|
pub(crate) struct RawTransactionAdmission {
|
||||||
|
backpressure_wait_observed: bool,
|
||||||
|
capacity: usize,
|
||||||
|
receiver: tokio::sync::mpsc::Receiver<crate::RawTransactionIngress>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionAdmission {
|
||||||
|
/// Creates one bounded admission queue and returns its crate-private source sender.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn new(capacity: usize) -> (crate::RawTransactionAdmission, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) {
|
||||||
|
let (sender, receiver) = tokio::sync::mpsc::channel(capacity);
|
||||||
|
return (Self { backpressure_wait_observed: false, capacity, receiver }, sender);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest receiver-side queue depth without consuming an ingress entry.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn queue_depth(&self) -> usize {
|
||||||
|
return self.receiver.len();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Closes new admissions while preserving already queued ingress for deterministic drain.
|
||||||
|
pub(crate) fn close(&mut self) {
|
||||||
|
self.receiver.close();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Receives and converts one queued ingress into a common RAW transaction acquisition.
|
||||||
|
pub(crate) async fn receive(
|
||||||
|
&mut self,
|
||||||
|
expected_network: &ksp_store_lib::RawNetworkId,
|
||||||
|
) -> ksp_core_lib::Result<std::option::Option<ksp_raw_transaction_lib::RawTransactionAcquisition>> {
|
||||||
|
self.backpressure_wait_observed = self.receiver.len() == self.capacity;
|
||||||
|
let ingress = match self.receiver.recv().await {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Ok(std::option::Option::None),
|
||||||
|
};
|
||||||
|
let acquisition = canonicalize_ingress(expected_network, ingress);
|
||||||
|
return match acquisition {
|
||||||
|
std::result::Result::Ok(value) => std::result::Result::Ok(std::option::Option::Some(value)),
|
||||||
|
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Takes and clears the source-neutral signal that the previous dequeue observed the bounded queue at full capacity.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn take_backpressure_wait_observed(&mut self) -> bool {
|
||||||
|
let observed = self.backpressure_wait_observed;
|
||||||
|
self.backpressure_wait_observed = false;
|
||||||
|
return observed;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn canonicalize_ingress(
|
||||||
|
expected_network: &ksp_store_lib::RawNetworkId,
|
||||||
|
ingress: crate::RawTransactionIngress,
|
||||||
|
) -> ksp_core_lib::Result<ksp_raw_transaction_lib::RawTransactionAcquisition> {
|
||||||
|
if &ingress.network != expected_network {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("admission.network_mismatch"));
|
||||||
|
}
|
||||||
|
let transaction = ksp_raw_transaction_lib::canonicalize_raw_transaction(ingress.material);
|
||||||
|
let transaction = match transaction {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("admission.canonicalization_invalid")),
|
||||||
|
};
|
||||||
|
if transaction.reference().network() != expected_network {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("admission.material_network_mismatch"));
|
||||||
|
}
|
||||||
|
let observation_key = observation_key(transaction.reference(), &ingress.source_key);
|
||||||
|
return std::result::Result::Ok(ksp_raw_transaction_lib::assemble_raw_transaction_acquisition(transaction, observation_key, ingress.provenance));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn hash_bytes(hasher: &mut sha2::Sha256, value: &[u8]) {
|
||||||
|
hasher.update((value.len() as u64).to_be_bytes());
|
||||||
|
hasher.update(value);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn observation_key(reference: &ksp_store_lib::RawTransactionReference, source_key: &[u8; 32]) -> ksp_store_lib::RawObservationKey {
|
||||||
|
let mut hasher = sha2::Sha256::new();
|
||||||
|
hasher.update(RAW_TRANSACTION_INGEST_OBSERVATION_DOMAIN);
|
||||||
|
hash_bytes(&mut hasher, reference.network().as_str().as_bytes());
|
||||||
|
hash_bytes(&mut hasher, reference.signature().as_bytes());
|
||||||
|
hash_bytes(&mut hasher, source_key);
|
||||||
|
let bytes: [u8; 32] = hasher.finalize().into();
|
||||||
|
return ksp_store_lib::RawObservationKey::new(bytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/admission.rs"]
|
||||||
|
mod tests;
|
||||||
54
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
Normal file
54
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
// version: 6
|
||||||
|
|
||||||
|
/// Error code used when durable Store content conflicts with one admitted canonical RAW transaction.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "content_conflict");
|
||||||
|
/// Error code used when one monotone RAW transaction ingest Worker counter or snapshot sequence cannot advance without wrapping.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "counter_exhausted");
|
||||||
|
/// Error code used when private Worker tasks cannot drain before the configured shutdown deadline.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "drain_timeout");
|
||||||
|
/// Error code used when the RAW transaction ingest Worker reaches an invalid runtime or lifecycle condition.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "runtime_invalid");
|
||||||
|
/// Error code used when RAW transaction ingest Worker settings violate one bounded runtime invariant.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "settings_invalid");
|
||||||
|
/// Error code used when one private source task terminates with a classified failure.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "source_failed");
|
||||||
|
/// Error code used when Store persistence fails for one admitted canonical RAW transaction.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "store_failed");
|
||||||
|
|
||||||
|
/// Creates one terminal counter-exhaustion error without exposing runtime material.
|
||||||
|
pub(crate) fn counter_exhausted_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED, "RAW transaction ingest Worker counter exhausted")
|
||||||
|
.with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one terminal content-conflict error without copying conflicting material into diagnostics.
|
||||||
|
pub(crate) fn content_conflict_error() -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT, "RAW transaction ingest Store content conflict");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one runtime-domain error carrying only one stable internal condition code.
|
||||||
|
pub(crate) fn runtime_error(condition: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID, "invalid RAW transaction ingest Worker runtime state")
|
||||||
|
.with_context("condition", condition);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one settings-domain error carrying only the stable invalid field name.
|
||||||
|
pub(crate) fn settings_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID, "invalid RAW transaction ingest Worker settings")
|
||||||
|
.with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one terminal Store error while retaining only the already-safe lower-layer ErrorCode.
|
||||||
|
pub(crate) fn store_error(store_code: ksp_core_lib::ErrorCode) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED, "RAW transaction ingest Store persistence failed")
|
||||||
|
.with_context("store_domain", store_code.domain())
|
||||||
|
.with_context("store_code", store_code.code());
|
||||||
|
}
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/identity.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
/// Stable Worker kind code used by the continuous RAW transaction ingest vertical.
|
||||||
|
pub const RAW_TRANSACTION_INGEST_WORKER_KIND_CODE: &str = "raw_transaction_ingest";
|
||||||
140
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
Normal file
140
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
Normal file
@@ -0,0 +1,140 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
// version: 26
|
||||||
|
|
||||||
|
#![warn(missing_docs)]
|
||||||
|
#![deny(unreachable_pub)]
|
||||||
|
#![forbid(unsafe_code)]
|
||||||
|
|
||||||
|
//! Source-neutral runtime foundation for continuous KSP RAW transaction ingestion.
|
||||||
|
//! Owned source, hydration and persistence tasks are bounded and joined before terminal publication.
|
||||||
|
//!
|
||||||
|
//! This tranche owns the concrete Worker family identity, validated technical settings
|
||||||
|
//! and the caller-runtime-owned lifecycle with private child-task supervision. This tranche also
|
||||||
|
//! owns bounded source-neutral admission, common RAW canonicalization/assembly and backend-neutral
|
||||||
|
//! Store persistence in normal mode plus concrete latest-value snapshots projected onto Worker API. The bounded 1..32 caller-composed aggregate starts
|
||||||
|
//! all validated Yellowstone, Standard Logs, Standard Block, Helius Transaction and HTTP live block polling sources under one private supervisor. Reference-bearing
|
||||||
|
//! sources share bounded cross-source hydration and fairness budgets; direct-qualified Standard Block and HTTP polling transactions enter the existing central
|
||||||
|
//! admission path directly. Public snapshots expose only source-neutral aggregate counts/state and conservative Worker health. Yellowstone
|
||||||
|
//! Transaction/TransactionStatus/Block and standard logs notifications become signature/slot references; BlockMeta/Slot remain continuity-only signals.
|
||||||
|
//! Hydration is coalesced by network/signature/commitment under bounded in-flight and pending budgets. A bounded run-local processing frontier projects
|
||||||
|
//! hydration pending, oldest pending slot and highest unblocked actually observed slot. The productive source also projects safe Transport reconnect/replay
|
||||||
|
//! state and faults conservatively when Transport proves a replay-retention continuity gap; history remediation remains outside this crate.
|
||||||
|
|
||||||
|
mod admission;
|
||||||
|
mod error;
|
||||||
|
mod identity;
|
||||||
|
mod persistence;
|
||||||
|
mod runtime;
|
||||||
|
mod runtime_resources;
|
||||||
|
mod settings;
|
||||||
|
mod snapshot;
|
||||||
|
|
||||||
|
/// Error code used when durable Store content conflicts with one admitted canonical RAW transaction.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT;
|
||||||
|
/// Error code used when one monotone Worker counter or snapshot sequence cannot advance without wrapping.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED;
|
||||||
|
/// Error code used when private Worker tasks cannot drain before the configured shutdown deadline.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT;
|
||||||
|
/// Error code used when the RAW transaction ingest Worker reaches an invalid runtime or lifecycle condition.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID;
|
||||||
|
/// Error code used when RAW transaction ingest Worker settings violate one bounded runtime invariant.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID;
|
||||||
|
/// Error code used when one private source task terminates with a classified failure.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED;
|
||||||
|
/// Error code used when Store persistence fails for one admitted canonical RAW transaction.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED;
|
||||||
|
/// Stable Worker kind code used by the continuous RAW transaction ingest vertical.
|
||||||
|
pub use self::identity::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE;
|
||||||
|
/// Cloneable external control handle for one continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime::RawTransactionIngestHandle;
|
||||||
|
/// Runtime-neutral boxed future resolving after one RAW transaction ingest Worker has fully reached a terminal lifecycle state.
|
||||||
|
pub use self::runtime::RawTransactionIngestTerminalFuture;
|
||||||
|
/// Entry point owning synchronous validation and task launch for one RAW transaction ingest Worker run.
|
||||||
|
pub use self::runtime::RawTransactionIngestWorker;
|
||||||
|
/// Default interval between HTTP live block polling cycles.
|
||||||
|
pub use self::runtime_resources::DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL;
|
||||||
|
/// Default maximum number of confirmed blocks discovered during one HTTP live block polling cycle.
|
||||||
|
pub use self::runtime_resources::DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE;
|
||||||
|
/// Maximum interval accepted between HTTP live block polling cycles.
|
||||||
|
pub use self::runtime_resources::MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL;
|
||||||
|
/// Maximum number of confirmed blocks accepted during one HTTP live block polling cycle.
|
||||||
|
pub use self::runtime_resources::MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE;
|
||||||
|
/// Maximum number of logical live sources accepted by one runtime-resource aggregate.
|
||||||
|
pub use self::runtime_resources::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES;
|
||||||
|
/// Minimum interval accepted between HTTP live block polling cycles.
|
||||||
|
pub use self::runtime_resources::MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL;
|
||||||
|
/// Minimum number of confirmed blocks accepted during one HTTP live block polling cycle.
|
||||||
|
pub use self::runtime_resources::MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE;
|
||||||
|
/// Validated Helius `transactionSubscribe` + HTTP hydration source contract owned by the continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime_resources::RawTransactionIngestHeliusTransactionSource;
|
||||||
|
/// Validated standard Solana HTTP live block polling source contract owned by the continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime_resources::RawTransactionIngestHttpBlockPollingSource;
|
||||||
|
/// Caller-composed bounded runtime resources for supported continuous RAW transaction live-source families.
|
||||||
|
pub use self::runtime_resources::RawTransactionIngestRuntimeResources;
|
||||||
|
/// Validated standard Solana `blockSubscribe` direct RAW source contract owned by the continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime_resources::RawTransactionIngestStandardBlockSource;
|
||||||
|
/// Validated standard Solana `logsSubscribe` + HTTP hydration source contract owned by the continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime_resources::RawTransactionIngestStandardLogsSource;
|
||||||
|
/// Validated Yellowstone + HTTP source contract owned by the continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime_resources::RawTransactionIngestYellowstoneSource;
|
||||||
|
/// Default bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY;
|
||||||
|
/// Default number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY;
|
||||||
|
/// Default cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT;
|
||||||
|
/// Maximum bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY;
|
||||||
|
/// Maximum number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY;
|
||||||
|
/// Maximum cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT;
|
||||||
|
/// Minimum bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY;
|
||||||
|
/// Minimum number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY;
|
||||||
|
/// Minimum cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT;
|
||||||
|
/// Validated source-neutral runtime settings for one continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::RawTransactionIngestSettings;
|
||||||
|
/// Complete safe latest-value snapshot of one continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::snapshot::RawTransactionIngestSnapshot;
|
||||||
|
/// Runtime-neutral boxed future resolving to one newer concrete RAW transaction ingest Worker snapshot.
|
||||||
|
pub use self::snapshot::RawTransactionIngestSnapshotFuture;
|
||||||
|
/// Cloneable latest-value source exposing concrete and common Worker snapshots from one shared watch state.
|
||||||
|
pub use self::snapshot::RawTransactionIngestSnapshotSource;
|
||||||
|
/// Source-neutral lifecycle state of the productive RAW transaction ingest source.
|
||||||
|
pub use self::snapshot::RawTransactionIngestSourceState;
|
||||||
|
|
||||||
|
/// Receiver-side owner of the private bounded RAW transaction admission queue.
|
||||||
|
pub(crate) use self::admission::RawTransactionAdmission;
|
||||||
|
/// Crate-private source-neutral ingress sent through the bounded central admission queue.
|
||||||
|
pub(crate) use self::admission::RawTransactionIngress;
|
||||||
|
/// Creates one terminal content-conflict error without copying conflicting material into diagnostics.
|
||||||
|
pub(crate) use self::error::content_conflict_error;
|
||||||
|
/// Creates one terminal counter-exhaustion error without exposing runtime material.
|
||||||
|
pub(crate) use self::error::counter_exhausted_error;
|
||||||
|
/// Creates one runtime-domain error without copying runtime/provider/Store values into diagnostics.
|
||||||
|
pub(crate) use self::error::runtime_error;
|
||||||
|
/// Creates one settings-domain error without copying caller-supplied values into diagnostics.
|
||||||
|
pub(crate) use self::error::settings_error;
|
||||||
|
/// Creates one terminal Store error while retaining only the already-safe lower-layer ErrorCode.
|
||||||
|
pub(crate) use self::error::store_error;
|
||||||
|
/// Canonical entity disposition produced by one successful Worker Store persistence attempt.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestEntityPersistence;
|
||||||
|
/// Observation disposition produced by one successful Worker Store persistence attempt.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestObservationPersistence;
|
||||||
|
/// Private bounded run-local cache serializing repeated canonical identities before Store writes.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestPersistenceConvergence;
|
||||||
|
/// Classified successful outcome of one atomic Worker Store persistence attempt.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestPersistenceOutcome;
|
||||||
|
/// Private backend-neutral Store persistence port used by the Worker and deterministic tests.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestPersistencePort;
|
||||||
|
/// Persists one already-canonical Worker acquisition through the private Store port in `Normal` mode.
|
||||||
|
pub(crate) use self::persistence::persist_raw_transaction_ingest_acquisition;
|
||||||
|
/// Persists one canonical acquisition through the bounded cross-source convergence cache.
|
||||||
|
pub(crate) use self::persistence::persist_raw_transaction_ingest_converged_acquisition;
|
||||||
|
/// Private latest-value processing-frontier projection emitted by the productive source task.
|
||||||
|
pub(crate) use self::snapshot::RawTransactionIngestProcessingFrontierProjection;
|
||||||
|
/// Private latest-value publisher and checked counter owner shared by the Worker supervisor.
|
||||||
|
pub(crate) use self::snapshot::RawTransactionIngestSnapshotPublisher;
|
||||||
290
crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
Normal file
290
crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
Normal file
@@ -0,0 +1,290 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
|
||||||
|
// version: 4
|
||||||
|
|
||||||
|
/// Canonical entity disposition produced by one Worker Store persistence attempt.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub(crate) enum RawTransactionIngestEntityPersistence {
|
||||||
|
/// The canonical RAW transaction was inserted for the first time.
|
||||||
|
Inserted,
|
||||||
|
/// Identical canonical RAW transaction content was already durable.
|
||||||
|
AlreadyPresent,
|
||||||
|
/// Normal persistence respected an existing purge tombstone.
|
||||||
|
SkippedPurged,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Observation disposition produced by one Worker Store persistence attempt.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub(crate) enum RawTransactionIngestObservationPersistence {
|
||||||
|
/// The deterministic acquisition observation was inserted for the first time.
|
||||||
|
Inserted,
|
||||||
|
/// The same deterministic acquisition observation was already durable.
|
||||||
|
AlreadyPresent,
|
||||||
|
/// No observation was recorded because the canonical entity was intentionally skipped.
|
||||||
|
NotRecorded,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Classified successful outcome of one atomic Worker Store persistence attempt.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub(crate) struct RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
/// Returns the canonical entity persistence disposition.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) const fn entity(self) -> crate::RawTransactionIngestEntityPersistence {
|
||||||
|
return self.entity;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the acquisition-observation persistence disposition.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) const fn observation(self) -> crate::RawTransactionIngestObservationPersistence {
|
||||||
|
return self.observation;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Private backend-neutral Store persistence port used by the Worker and deterministic tests.
|
||||||
|
pub(crate) trait RawTransactionIngestPersistencePort: std::marker::Send + std::marker::Sync {
|
||||||
|
/// Returns whether the persistence target belongs to the requested logical network.
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool;
|
||||||
|
|
||||||
|
/// Executes one atomic canonical RAW transaction plus observation Store write.
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
transaction: ksp_store_lib::RawTransaction,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>>;
|
||||||
|
|
||||||
|
/// Records one additional observation for an already durable canonical RAW transaction.
|
||||||
|
fn record_observation<'a>(
|
||||||
|
&'a self,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawObservationWriteOutcome>>;
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistencePort for ksp_store_lib::Store {
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool {
|
||||||
|
return self.runtime_snapshot().network() == network;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
transaction: ksp_store_lib::RawTransaction,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>> {
|
||||||
|
return ksp_store_lib::RawTransactionWrite::persist_raw_transaction_acquisition(self, transaction, observation, mode);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn record_observation<'a>(
|
||||||
|
&'a self,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawObservationWriteOutcome>> {
|
||||||
|
return ksp_store_lib::RawTransactionObservationWrite::record_raw_transaction_observation(self, observation);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Eq, Ord, PartialEq, PartialOrd)]
|
||||||
|
struct RawTransactionIngestPersistenceKey {
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
signature: ksp_store_lib::RawTransactionSignature,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
struct RawTransactionIngestCanonicalState {
|
||||||
|
block_time: std::option::Option<ksp_store_lib::RawTimestamp>,
|
||||||
|
content_hash: ksp_store_lib::RawContentHash,
|
||||||
|
format_id: ksp_store_lib::RawFormatId,
|
||||||
|
format_version: u32,
|
||||||
|
slot: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Private bounded run-local cache serializing repeated canonical identities before Store writes.
|
||||||
|
pub(crate) struct RawTransactionIngestPersistenceConvergence {
|
||||||
|
max_entries: usize,
|
||||||
|
entries: std::sync::Mutex<
|
||||||
|
std::collections::BTreeMap<
|
||||||
|
RawTransactionIngestPersistenceKey,
|
||||||
|
std::sync::Arc<tokio::sync::Mutex<std::option::Option<RawTransactionIngestCanonicalState>>>,
|
||||||
|
>,
|
||||||
|
>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistenceConvergence {
|
||||||
|
/// Creates one bounded run-local convergence cache using the caller-provided effective runtime bound.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn new(max_entries: usize) -> Self {
|
||||||
|
return Self { max_entries, entries: std::sync::Mutex::new(std::collections::BTreeMap::new()) };
|
||||||
|
}
|
||||||
|
|
||||||
|
fn entry(
|
||||||
|
&self,
|
||||||
|
key: RawTransactionIngestPersistenceKey,
|
||||||
|
) -> std::option::Option<std::sync::Arc<tokio::sync::Mutex<std::option::Option<RawTransactionIngestCanonicalState>>>> {
|
||||||
|
let mut entries = match self.entries.lock() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(poisoned) => poisoned.into_inner(),
|
||||||
|
};
|
||||||
|
if let std::option::Option::Some(entry) = entries.get(&key) {
|
||||||
|
return std::option::Option::Some(std::sync::Arc::clone(entry));
|
||||||
|
}
|
||||||
|
if entries.len() >= self.max_entries {
|
||||||
|
let removable = entries.iter().find_map(|(candidate, entry)| {
|
||||||
|
if std::sync::Arc::strong_count(entry) == 1 {
|
||||||
|
return std::option::Option::Some(candidate.clone());
|
||||||
|
}
|
||||||
|
return std::option::Option::None;
|
||||||
|
});
|
||||||
|
if let std::option::Option::Some(removable) = removable {
|
||||||
|
entries.remove(&removable);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if entries.len() >= self.max_entries {
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
let entry = std::sync::Arc::new(tokio::sync::Mutex::new(std::option::Option::None));
|
||||||
|
entries.insert(key, std::sync::Arc::clone(&entry));
|
||||||
|
return std::option::Option::Some(entry);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Persists one already-canonical Worker acquisition through the private Store port in `Normal` mode.
|
||||||
|
pub(crate) async fn persist_raw_transaction_ingest_acquisition<P>(
|
||||||
|
port: &P,
|
||||||
|
acquisition: ksp_raw_transaction_lib::RawTransactionAcquisition,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>
|
||||||
|
where
|
||||||
|
P: crate::RawTransactionIngestPersistencePort + ?Sized,
|
||||||
|
{
|
||||||
|
let reference = acquisition.transaction().reference().clone();
|
||||||
|
if !port.network_matches(reference.network()) {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("persistence.store_network_mismatch"));
|
||||||
|
}
|
||||||
|
if acquisition.observation().transaction() != &reference {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("persistence.acquisition_reference_mismatch"));
|
||||||
|
}
|
||||||
|
let (transaction, observation) = acquisition.into_parts();
|
||||||
|
let result = port.persist_acquisition(transaction, observation, ksp_store_lib::RawTransactionAcquisitionMode::Normal).await;
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(outcome) => map_store_outcome(outcome),
|
||||||
|
std::result::Result::Err(error) => map_store_error(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Persists one canonical acquisition through the bounded cross-source convergence cache.
|
||||||
|
pub(crate) async fn persist_raw_transaction_ingest_converged_acquisition<P>(
|
||||||
|
port: &P,
|
||||||
|
acquisition: ksp_raw_transaction_lib::RawTransactionAcquisition,
|
||||||
|
convergence: &crate::RawTransactionIngestPersistenceConvergence,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>
|
||||||
|
where
|
||||||
|
P: crate::RawTransactionIngestPersistencePort + ?Sized,
|
||||||
|
{
|
||||||
|
let reference = acquisition.transaction().reference().clone();
|
||||||
|
if !port.network_matches(reference.network()) {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("persistence.store_network_mismatch"));
|
||||||
|
}
|
||||||
|
if acquisition.observation().transaction() != &reference {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("persistence.acquisition_reference_mismatch"));
|
||||||
|
}
|
||||||
|
let canonical_state = canonical_state(acquisition.transaction());
|
||||||
|
let key = RawTransactionIngestPersistenceKey { network: reference.network().clone(), signature: reference.signature() };
|
||||||
|
let entry = convergence.entry(key);
|
||||||
|
let entry = match entry {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return crate::persist_raw_transaction_ingest_acquisition(port, acquisition).await,
|
||||||
|
};
|
||||||
|
let mut known_state = entry.lock().await;
|
||||||
|
if let std::option::Option::Some(known_state_value) = known_state.as_ref() {
|
||||||
|
if known_state_value != &canonical_state {
|
||||||
|
return std::result::Result::Err(crate::content_conflict_error());
|
||||||
|
}
|
||||||
|
let (_transaction, observation) = acquisition.into_parts();
|
||||||
|
let result = port.record_observation(observation).await;
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(outcome) => map_additional_observation_outcome(outcome),
|
||||||
|
std::result::Result::Err(error) => map_store_error(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
let outcome = crate::persist_raw_transaction_ingest_acquisition(port, acquisition).await;
|
||||||
|
let outcome = match outcome {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if outcome.entity() != crate::RawTransactionIngestEntityPersistence::SkippedPurged {
|
||||||
|
*known_state = std::option::Option::Some(canonical_state);
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(outcome);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn canonical_state(transaction: &ksp_store_lib::RawTransaction) -> RawTransactionIngestCanonicalState {
|
||||||
|
return RawTransactionIngestCanonicalState {
|
||||||
|
block_time: transaction.block_time(),
|
||||||
|
content_hash: transaction.payload().content_hash(),
|
||||||
|
format_id: transaction.payload().format_id().clone(),
|
||||||
|
format_version: transaction.payload().format_version(),
|
||||||
|
slot: transaction.slot(),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn map_store_error(error: ksp_core_lib::Error) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome> {
|
||||||
|
if error.code() == ksp_store_lib::ERROR_CODE_RAW_CONFLICT {
|
||||||
|
return std::result::Result::Err(crate::content_conflict_error());
|
||||||
|
}
|
||||||
|
return std::result::Result::Err(crate::store_error(error.code()));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn map_store_outcome(outcome: ksp_store_lib::RawAcquisitionWriteOutcome) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome> {
|
||||||
|
let entity = outcome.entity();
|
||||||
|
let observation = outcome.observation();
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::Inserted && observation == ksp_store_lib::RawObservationWriteOutcome::Inserted {
|
||||||
|
return std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::Inserted,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent && observation == ksp_store_lib::RawObservationWriteOutcome::Inserted {
|
||||||
|
return std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent && observation == ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent {
|
||||||
|
return std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::AlreadyPresent,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::SkippedPurged && observation == ksp_store_lib::RawObservationWriteOutcome::NotRecorded {
|
||||||
|
return std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::SkippedPurged,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::NotRecorded,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return std::result::Result::Err(crate::runtime_error("persistence.store_outcome_invalid"));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn map_additional_observation_outcome(
|
||||||
|
outcome: ksp_store_lib::RawObservationWriteOutcome,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome> {
|
||||||
|
return match outcome {
|
||||||
|
ksp_store_lib::RawObservationWriteOutcome::Inserted => std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
}),
|
||||||
|
ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent => std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::AlreadyPresent,
|
||||||
|
}),
|
||||||
|
ksp_store_lib::RawObservationWriteOutcome::NotRecorded => {
|
||||||
|
std::result::Result::Err(crate::runtime_error("persistence.additional_observation_not_recorded"))
|
||||||
|
},
|
||||||
|
_ => std::result::Result::Err(crate::runtime_error("persistence.additional_observation_outcome_invalid")),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/persistence.rs"]
|
||||||
|
mod tests;
|
||||||
662
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
Normal file
662
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
Normal file
@@ -0,0 +1,662 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
// version: 17
|
||||||
|
|
||||||
|
type PersistencePort = std::sync::Arc<dyn crate::RawTransactionIngestPersistencePort + 'static>;
|
||||||
|
type PersistenceTasks = tokio::task::JoinSet<ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>>;
|
||||||
|
type ProcessingFrontierReceiver = tokio::sync::watch::Receiver<crate::RawTransactionIngestProcessingFrontierProjection>;
|
||||||
|
type SourceTasks = tokio::task::JoinSet<ksp_core_lib::Result<()>>;
|
||||||
|
|
||||||
|
/// Runtime-neutral boxed future resolving after one RAW transaction ingest Worker has fully reached a terminal lifecycle state.
|
||||||
|
pub type RawTransactionIngestTerminalFuture<'a> =
|
||||||
|
std::pin::Pin<std::boxed::Box<dyn std::future::Future<Output = ksp_core_lib::Result<ksp_worker_api::WorkerState>> + std::marker::Send + 'a>>;
|
||||||
|
|
||||||
|
/// Cloneable external control handle for one continuous RAW transaction ingest Worker.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct RawTransactionIngestHandle {
|
||||||
|
snapshots: crate::RawTransactionIngestSnapshotSource,
|
||||||
|
stop_sender: tokio::sync::watch::Sender<bool>,
|
||||||
|
stop_token: ksp_worker_api::WorkerStopToken,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestHandle {
|
||||||
|
/// Requests cooperative stop and returns `true` only for the first request accepted by the live runtime.
|
||||||
|
#[must_use]
|
||||||
|
pub fn request_stop(&self) -> bool {
|
||||||
|
if !self.stop_token.request_stop() {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return self.stop_sender.send(true).is_ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns an independent latest-value source for concrete RAW transaction ingest Worker snapshots.
|
||||||
|
#[must_use]
|
||||||
|
pub fn snapshot_source(&self) -> crate::RawTransactionIngestSnapshotSource {
|
||||||
|
return self.snapshots.clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns an independent source implementing the common [`ksp_worker_api::WorkerSnapshotSource`] projection.
|
||||||
|
#[must_use]
|
||||||
|
pub fn worker_snapshot_source(&self) -> crate::RawTransactionIngestSnapshotSource {
|
||||||
|
return self.snapshots.clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Waits until the private runtime task has published one terminal lifecycle state after draining owned work.
|
||||||
|
#[must_use]
|
||||||
|
pub fn wait_terminal(&self) -> crate::RawTransactionIngestTerminalFuture<'_> {
|
||||||
|
let source = self.snapshots.clone();
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
loop {
|
||||||
|
let current = source.current();
|
||||||
|
let state = current.worker_snapshot().state();
|
||||||
|
if state.is_terminal() {
|
||||||
|
return std::result::Result::Ok(state);
|
||||||
|
}
|
||||||
|
let observed = current.worker_snapshot().sequence();
|
||||||
|
let changed = source.wait_for_change(observed).await;
|
||||||
|
let state = changed.worker_snapshot().state();
|
||||||
|
if state.is_terminal() {
|
||||||
|
return std::result::Result::Ok(state);
|
||||||
|
}
|
||||||
|
if changed.worker_snapshot().sequence() == observed && source.is_closed() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("terminal.closed_before_terminal"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionIngestHandle {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
let snapshot = self.snapshots.current();
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionIngestHandle")
|
||||||
|
.field("stop_requested", &self.stop_token.is_stop_requested())
|
||||||
|
.field("sequence", &snapshot.worker_snapshot().sequence())
|
||||||
|
.field("state", &snapshot.worker_snapshot().state())
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Entry point owning synchronous validation and task launch for one RAW transaction ingest Worker run.
|
||||||
|
pub struct RawTransactionIngestWorker;
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestWorker {
|
||||||
|
/// Starts one Worker on the caller-owned current Tokio runtime while retaining the caller-owned Store facade through an `Arc`.
|
||||||
|
pub fn start(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
store: std::sync::Arc<ksp_store_lib::Store>,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle> {
|
||||||
|
let runtime = match current_runtime_handle() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let store_snapshot = store.runtime_snapshot();
|
||||||
|
if let std::result::Result::Err(error) = validate_store_network(&settings, store_snapshot.network()) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
return start_foundation(settings, runtime, std::option::Option::Some(store));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Starts one Worker with caller-composed runtime resources.
|
||||||
|
///
|
||||||
|
/// Every validated source in the bounded runtime-resource aggregate is started concurrently under one private source supervisor. Any configured source
|
||||||
|
/// failure remains terminal for the Worker because this release does not infer equivalent coverage or failover between source families.
|
||||||
|
pub fn start_with_runtime_resources(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
store: std::sync::Arc<ksp_store_lib::Store>,
|
||||||
|
runtime_resources: crate::RawTransactionIngestRuntimeResources,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle> {
|
||||||
|
let runtime = match current_runtime_handle() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let store_snapshot = store.runtime_snapshot();
|
||||||
|
if let std::result::Result::Err(error) = validate_store_network(&settings, store_snapshot.network()) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = runtime_resources.validate_network(settings.network()) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let source_total = runtime_resources.source_count();
|
||||||
|
let source_settings = settings.clone();
|
||||||
|
let (processing_frontier_sender, processing_frontier_receiver) =
|
||||||
|
tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty());
|
||||||
|
let port: PersistencePort = store;
|
||||||
|
return start_foundation_with_port_source_spawner_and_frontier(
|
||||||
|
settings,
|
||||||
|
runtime,
|
||||||
|
std::option::Option::Some(port),
|
||||||
|
std::option::Option::Some(processing_frontier_receiver),
|
||||||
|
source_total,
|
||||||
|
move |children, stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
return runtime_resources.run_live_sources(source_settings, stop_receiver, admission_sender, processing_frontier_sender).await;
|
||||||
|
});
|
||||||
|
},
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn begin_stopping(
|
||||||
|
lifecycle: &mut ksp_worker_api::WorkerLifecycle,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
if lifecycle.mark_stopping().is_err() {
|
||||||
|
return std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = snapshots.publish_state(lifecycle.state(), admission_queue_depth, in_flight_persistence) {
|
||||||
|
return std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn current_runtime_handle() -> ksp_core_lib::Result<tokio::runtime::Handle> {
|
||||||
|
return match tokio::runtime::Handle::try_current() {
|
||||||
|
std::result::Result::Ok(value) => std::result::Result::Ok(value),
|
||||||
|
std::result::Result::Err(_) => std::result::Result::Err(crate::runtime_error("start.runtime_unavailable")),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn drain_children(
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
children: &mut SourceTasks,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let mut fault = std::option::Option::None;
|
||||||
|
while let std::option::Option::Some(joined) = children.join_next().await {
|
||||||
|
let current = source_completion(joined, state, 0, 0, snapshots);
|
||||||
|
fault = merge_fault(fault, current);
|
||||||
|
}
|
||||||
|
return fault;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn drain_persistence(
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
admission: &crate::RawTransactionAdmission,
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let mut fault = std::option::Option::None;
|
||||||
|
while let std::option::Option::Some(joined) = persistence.join_next().await {
|
||||||
|
let current = persistence_completion(joined, lifecycle.state(), admission.queue_depth(), persistence.len(), snapshots);
|
||||||
|
fault = merge_fault(fault, current);
|
||||||
|
}
|
||||||
|
return fault;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn drain_admission_and_persistence(
|
||||||
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
admission: &mut crate::RawTransactionAdmission,
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
persistence_convergence: &std::sync::Arc<crate::RawTransactionIngestPersistenceConvergence>,
|
||||||
|
port: &std::option::Option<PersistencePort>,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let mut fault = std::option::Option::None;
|
||||||
|
admission.close();
|
||||||
|
loop {
|
||||||
|
while persistence.len() >= settings.persistence_concurrency() {
|
||||||
|
let joined = persistence.join_next().await;
|
||||||
|
let current = match joined {
|
||||||
|
std::option::Option::Some(value) => persistence_completion(value, lifecycle.state(), admission.queue_depth(), persistence.len(), snapshots),
|
||||||
|
std::option::Option::None => std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID),
|
||||||
|
};
|
||||||
|
fault = merge_fault(fault, current);
|
||||||
|
}
|
||||||
|
let received = admission.receive(settings.network()).await;
|
||||||
|
let backpressure_wait_observed = admission.take_backpressure_wait_observed();
|
||||||
|
match received {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(acquisition)) => {
|
||||||
|
let spawned = spawn_persistence(persistence, persistence_convergence, port, acquisition);
|
||||||
|
let published = snapshots.record_admission_success(lifecycle.state(), admission.queue_depth(), persistence.len(), backpressure_wait_observed);
|
||||||
|
if !spawned && fault.is_none() {
|
||||||
|
fault = std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = published
|
||||||
|
&& fault.is_none()
|
||||||
|
{
|
||||||
|
fault = std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Ok(std::option::Option::None) => break,
|
||||||
|
std::result::Result::Err(_) => {
|
||||||
|
let published = snapshots.record_admission_failure(lifecycle.state(), admission.queue_depth(), persistence.len(), backpressure_wait_observed);
|
||||||
|
match published {
|
||||||
|
std::result::Result::Ok(()) => {
|
||||||
|
if fault.is_none() {
|
||||||
|
fault = std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
if fault.is_none() {
|
||||||
|
fault = std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let persistence_fault = drain_persistence(lifecycle, admission, persistence, snapshots).await;
|
||||||
|
fault = merge_fault(fault, persistence_fault);
|
||||||
|
return fault;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn drain_owned_work(
|
||||||
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
children: &mut SourceTasks,
|
||||||
|
admission: &mut crate::RawTransactionAdmission,
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
persistence_convergence: &std::sync::Arc<crate::RawTransactionIngestPersistenceConvergence>,
|
||||||
|
port: &std::option::Option<PersistencePort>,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let drain = async {
|
||||||
|
let mut fault = drain_admission_and_persistence(settings, lifecycle, admission, persistence, persistence_convergence, port, snapshots).await;
|
||||||
|
let child_fault = drain_children(lifecycle.state(), children, snapshots).await;
|
||||||
|
fault = merge_fault(fault, child_fault);
|
||||||
|
return fault;
|
||||||
|
};
|
||||||
|
let timed = tokio::time::timeout(settings.shutdown_drain_timeout(), drain).await;
|
||||||
|
return match timed {
|
||||||
|
std::result::Result::Ok(fault) => fault,
|
||||||
|
std::result::Result::Err(_) => {
|
||||||
|
admission.close();
|
||||||
|
persistence.abort_all();
|
||||||
|
children.abort_all();
|
||||||
|
while persistence.join_next().await.is_some() {}
|
||||||
|
while children.join_next().await.is_some() {}
|
||||||
|
std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT)
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn finish_faulted(
|
||||||
|
lifecycle: &mut ksp_worker_api::WorkerLifecycle,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
code: ksp_core_lib::ErrorCode,
|
||||||
|
) {
|
||||||
|
if lifecycle.fault(code).is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if snapshots.publish_state(lifecycle.state(), 0, 0).is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn finish_stopped(lifecycle: &mut ksp_worker_api::WorkerLifecycle, snapshots: &mut crate::RawTransactionIngestSnapshotPublisher) {
|
||||||
|
if lifecycle.mark_stopped().is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if snapshots.publish_state(lifecycle.state(), 0, 0).is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn merge_fault(
|
||||||
|
first: std::option::Option<ksp_core_lib::ErrorCode>,
|
||||||
|
next: std::option::Option<ksp_core_lib::ErrorCode>,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
if first.is_some() {
|
||||||
|
return first;
|
||||||
|
}
|
||||||
|
return next;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persistence_completion(
|
||||||
|
joined: std::result::Result<ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>, tokio::task::JoinError>,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
return match joined {
|
||||||
|
std::result::Result::Ok(std::result::Result::Ok(outcome)) => {
|
||||||
|
match snapshots.record_persistence_success(state, admission_queue_depth, in_flight_persistence, outcome) {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::None,
|
||||||
|
std::result::Result::Err(error) => std::option::Option::Some(error.code()),
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Ok(std::result::Result::Err(error)) => {
|
||||||
|
let code = error.code();
|
||||||
|
let published = snapshots.record_persistence_fault(state, admission_queue_depth, in_flight_persistence, code);
|
||||||
|
match published {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::Some(code),
|
||||||
|
std::result::Result::Err(snapshot_error) => std::option::Option::Some(snapshot_error.code()),
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Err(_) => std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn source_completion(
|
||||||
|
joined: std::result::Result<ksp_core_lib::Result<()>, tokio::task::JoinError>,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let source_code = match joined {
|
||||||
|
std::result::Result::Ok(std::result::Result::Ok(())) => return std::option::Option::None,
|
||||||
|
std::result::Result::Ok(std::result::Result::Err(error)) if error.code() == crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED => {
|
||||||
|
crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED
|
||||||
|
},
|
||||||
|
std::result::Result::Ok(std::result::Result::Err(_)) | std::result::Result::Err(_) => crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED,
|
||||||
|
};
|
||||||
|
let published = snapshots.record_source_failure(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
return match published {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::Some(source_code),
|
||||||
|
std::result::Result::Err(error) => std::option::Option::Some(error.code()),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn run_supervisor<Spawner>(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
mut lifecycle: ksp_worker_api::WorkerLifecycle,
|
||||||
|
port: std::option::Option<PersistencePort>,
|
||||||
|
mut stop_receiver: tokio::sync::watch::Receiver<bool>,
|
||||||
|
mut snapshots: crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
mut processing_frontier_receiver: std::option::Option<ProcessingFrontierReceiver>,
|
||||||
|
source_spawner: Spawner,
|
||||||
|
) where
|
||||||
|
Spawner:
|
||||||
|
FnOnce(&mut SourceTasks, tokio::sync::watch::Receiver<bool>, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) + std::marker::Send + 'static,
|
||||||
|
{
|
||||||
|
if *stop_receiver.borrow() {
|
||||||
|
let stopping_fault = begin_stopping(&mut lifecycle, &mut snapshots, 0, 0);
|
||||||
|
match stopping_fault {
|
||||||
|
std::option::Option::Some(code) => finish_faulted(&mut lifecycle, &mut snapshots, code),
|
||||||
|
std::option::Option::None => finish_stopped(&mut lifecycle, &mut snapshots),
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if lifecycle.mark_running().is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = snapshots.publish_state(lifecycle.state(), 0, 0) {
|
||||||
|
finish_faulted(&mut lifecycle, &mut snapshots, error.code());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let mut children = SourceTasks::new();
|
||||||
|
let mut persistence = PersistenceTasks::new();
|
||||||
|
let persistence_convergence = std::sync::Arc::new(crate::RawTransactionIngestPersistenceConvergence::new(
|
||||||
|
settings.admission_queue_capacity().max(settings.persistence_concurrency()),
|
||||||
|
));
|
||||||
|
let (source_stop_sender, source_stop_receiver) = tokio::sync::watch::channel(false);
|
||||||
|
let (mut admission, admission_sender) = crate::RawTransactionAdmission::new(settings.admission_queue_capacity());
|
||||||
|
source_spawner(&mut children, source_stop_receiver, admission_sender);
|
||||||
|
let mut fault = supervise_until_stop(
|
||||||
|
&settings,
|
||||||
|
&lifecycle,
|
||||||
|
&source_stop_sender,
|
||||||
|
&mut stop_receiver,
|
||||||
|
&mut children,
|
||||||
|
&mut admission,
|
||||||
|
&mut persistence,
|
||||||
|
&persistence_convergence,
|
||||||
|
&port,
|
||||||
|
&mut snapshots,
|
||||||
|
&mut processing_frontier_receiver,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
let stopping_fault = begin_stopping(&mut lifecycle, &mut snapshots, admission.queue_depth(), persistence.len());
|
||||||
|
fault = merge_fault(fault, stopping_fault);
|
||||||
|
let drain_fault =
|
||||||
|
drain_owned_work(&settings, &lifecycle, &mut children, &mut admission, &mut persistence, &persistence_convergence, &port, &mut snapshots).await;
|
||||||
|
match drain_fault {
|
||||||
|
std::option::Option::Some(code) if code == crate::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT => {
|
||||||
|
fault = std::option::Option::Some(code);
|
||||||
|
},
|
||||||
|
std::option::Option::Some(code) => {
|
||||||
|
fault = merge_fault(fault, std::option::Option::Some(code));
|
||||||
|
},
|
||||||
|
std::option::Option::None => {},
|
||||||
|
}
|
||||||
|
match fault {
|
||||||
|
std::option::Option::Some(code) => finish_faulted(&mut lifecycle, &mut snapshots, code),
|
||||||
|
std::option::Option::None => finish_stopped(&mut lifecycle, &mut snapshots),
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn spawn_persistence(
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
persistence_convergence: &std::sync::Arc<crate::RawTransactionIngestPersistenceConvergence>,
|
||||||
|
port: &std::option::Option<PersistencePort>,
|
||||||
|
acquisition: ksp_raw_transaction_lib::RawTransactionAcquisition,
|
||||||
|
) -> bool {
|
||||||
|
let port = match port {
|
||||||
|
std::option::Option::Some(value) => std::sync::Arc::clone(value),
|
||||||
|
std::option::Option::None => return false,
|
||||||
|
};
|
||||||
|
let persistence_convergence = std::sync::Arc::clone(persistence_convergence);
|
||||||
|
let _abort_handle = persistence.spawn(async move {
|
||||||
|
return crate::persist_raw_transaction_ingest_converged_acquisition(port.as_ref(), acquisition, persistence_convergence.as_ref()).await;
|
||||||
|
});
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn start_foundation(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
runtime: tokio::runtime::Handle,
|
||||||
|
store_guard: std::option::Option<std::sync::Arc<ksp_store_lib::Store>>,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle> {
|
||||||
|
return start_foundation_with_source_spawner(settings, runtime, store_guard, |_children, _stop_receiver, _admission_sender| {});
|
||||||
|
}
|
||||||
|
|
||||||
|
fn start_foundation_with_port_and_source_spawner<Spawner>(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
runtime: tokio::runtime::Handle,
|
||||||
|
port: std::option::Option<PersistencePort>,
|
||||||
|
source_spawner: Spawner,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle>
|
||||||
|
where
|
||||||
|
Spawner:
|
||||||
|
FnOnce(&mut SourceTasks, tokio::sync::watch::Receiver<bool>, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) + std::marker::Send + 'static,
|
||||||
|
{
|
||||||
|
return start_foundation_with_port_source_spawner_and_frontier(settings, runtime, port, std::option::Option::None, 0, source_spawner);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn start_foundation_with_port_source_spawner_and_frontier<Spawner>(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
runtime: tokio::runtime::Handle,
|
||||||
|
port: std::option::Option<PersistencePort>,
|
||||||
|
processing_frontier_receiver: std::option::Option<ProcessingFrontierReceiver>,
|
||||||
|
source_total: usize,
|
||||||
|
source_spawner: Spawner,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle>
|
||||||
|
where
|
||||||
|
Spawner:
|
||||||
|
FnOnce(&mut SourceTasks, tokio::sync::watch::Receiver<bool>, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) + std::marker::Send + 'static,
|
||||||
|
{
|
||||||
|
let kind = match ksp_worker_api::WorkerKindCode::new(crate::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("start.worker_kind_invalid")),
|
||||||
|
};
|
||||||
|
let mut lifecycle = ksp_worker_api::WorkerLifecycle::new(settings.worker_id().clone(), kind);
|
||||||
|
if lifecycle.start().is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("start.lifecycle_invalid"));
|
||||||
|
}
|
||||||
|
let stop_token = ksp_worker_api::WorkerStopToken::new();
|
||||||
|
let (stop_sender, stop_receiver) = tokio::sync::watch::channel(false);
|
||||||
|
let (snapshots, snapshot_source) = crate::RawTransactionIngestSnapshotPublisher::new(&settings, &lifecycle, source_total);
|
||||||
|
let handle = crate::RawTransactionIngestHandle { snapshots: snapshot_source, stop_sender, stop_token };
|
||||||
|
std::mem::drop(runtime.spawn(run_supervisor(settings, lifecycle, port, stop_receiver, snapshots, processing_frontier_receiver, source_spawner)));
|
||||||
|
return std::result::Result::Ok(handle);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn start_foundation_with_source_spawner<Spawner>(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
runtime: tokio::runtime::Handle,
|
||||||
|
store_guard: std::option::Option<std::sync::Arc<ksp_store_lib::Store>>,
|
||||||
|
source_spawner: Spawner,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle>
|
||||||
|
where
|
||||||
|
Spawner:
|
||||||
|
FnOnce(&mut SourceTasks, tokio::sync::watch::Receiver<bool>, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) + std::marker::Send + 'static,
|
||||||
|
{
|
||||||
|
let port = match store_guard {
|
||||||
|
std::option::Option::Some(store) => {
|
||||||
|
let port: PersistencePort = store;
|
||||||
|
std::option::Option::Some(port)
|
||||||
|
},
|
||||||
|
std::option::Option::None => std::option::Option::None,
|
||||||
|
};
|
||||||
|
return start_foundation_with_port_and_source_spawner(settings, runtime, port, source_spawner);
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn supervise_until_stop(
|
||||||
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
source_stop_sender: &tokio::sync::watch::Sender<bool>,
|
||||||
|
stop_receiver: &mut tokio::sync::watch::Receiver<bool>,
|
||||||
|
children: &mut SourceTasks,
|
||||||
|
admission: &mut crate::RawTransactionAdmission,
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
persistence_convergence: &std::sync::Arc<crate::RawTransactionIngestPersistenceConvergence>,
|
||||||
|
port: &std::option::Option<PersistencePort>,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
processing_frontier_receiver: &mut std::option::Option<ProcessingFrontierReceiver>,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let mut admission_open = true;
|
||||||
|
loop {
|
||||||
|
if *stop_receiver.borrow() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if children.is_empty() && !admission_open && persistence.is_empty() {
|
||||||
|
let changed = stop_receiver.changed().await;
|
||||||
|
if changed.is_err() || *stop_receiver.borrow() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
tokio::select! {
|
||||||
|
biased;
|
||||||
|
changed = stop_receiver.changed() => {
|
||||||
|
if changed.is_err() || *stop_receiver.borrow() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
processing_frontier = wait_processing_frontier(processing_frontier_receiver) => {
|
||||||
|
match processing_frontier {
|
||||||
|
std::option::Option::Some(projection) => {
|
||||||
|
let published = snapshots.record_processing_frontier(
|
||||||
|
lifecycle.state(),
|
||||||
|
admission.queue_depth(),
|
||||||
|
persistence.len(),
|
||||||
|
projection,
|
||||||
|
);
|
||||||
|
if let std::result::Result::Err(error) = published {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::option::Option::None => {
|
||||||
|
*processing_frontier_receiver = std::option::Option::None;
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
joined = children.join_next(), if !children.is_empty() => {
|
||||||
|
let source_fault = match joined {
|
||||||
|
std::option::Option::Some(value) => source_completion(value, lifecycle.state(), admission.queue_depth(), persistence.len(), snapshots),
|
||||||
|
std::option::Option::None => std::option::Option::None,
|
||||||
|
};
|
||||||
|
if source_fault.is_some() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return source_fault;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
joined = persistence.join_next(), if !persistence.is_empty() => {
|
||||||
|
match joined {
|
||||||
|
std::option::Option::Some(value) => {
|
||||||
|
let fault = persistence_completion(value, lifecycle.state(), admission.queue_depth(), persistence.len(), snapshots);
|
||||||
|
if fault.is_some() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return fault;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::option::Option::None => {},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
received = admission.receive(settings.network()), if admission_open && persistence.len() < settings.persistence_concurrency() => {
|
||||||
|
match received {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(acquisition)) => {
|
||||||
|
let backpressure_wait_observed = admission.take_backpressure_wait_observed();
|
||||||
|
let spawned = spawn_persistence(persistence, persistence_convergence, port, acquisition);
|
||||||
|
let published = snapshots.record_admission_success(
|
||||||
|
lifecycle.state(),
|
||||||
|
admission.queue_depth(),
|
||||||
|
persistence.len(),
|
||||||
|
backpressure_wait_observed,
|
||||||
|
);
|
||||||
|
if !spawned {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = published {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Ok(std::option::Option::None) => {
|
||||||
|
let _backpressure_wait_observed = admission.take_backpressure_wait_observed();
|
||||||
|
admission_open = false;
|
||||||
|
},
|
||||||
|
std::result::Result::Err(_) => {
|
||||||
|
let backpressure_wait_observed = admission.take_backpressure_wait_observed();
|
||||||
|
let published = snapshots.record_admission_failure(
|
||||||
|
lifecycle.state(),
|
||||||
|
admission.queue_depth(),
|
||||||
|
persistence.len(),
|
||||||
|
backpressure_wait_observed,
|
||||||
|
);
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return match published {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID),
|
||||||
|
std::result::Result::Err(error) => std::option::Option::Some(error.code()),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_processing_frontier(
|
||||||
|
receiver: &mut std::option::Option<ProcessingFrontierReceiver>,
|
||||||
|
) -> std::option::Option<crate::RawTransactionIngestProcessingFrontierProjection> {
|
||||||
|
let receiver = match receiver.as_mut() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => {
|
||||||
|
return std::future::pending::<std::option::Option<crate::RawTransactionIngestProcessingFrontierProjection>>().await;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
if receiver.changed().await.is_err() {
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
return std::option::Option::Some(*receiver.borrow_and_update());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_store_network(settings: &crate::RawTransactionIngestSettings, store_network: &ksp_store_lib::RawNetworkId) -> ksp_core_lib::Result<()> {
|
||||||
|
if settings.network() != store_network {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("start.store_network_mismatch"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/runtime.rs"]
|
||||||
|
mod tests;
|
||||||
File diff suppressed because it is too large
Load Diff
131
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
Normal file
131
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
Normal file
@@ -0,0 +1,131 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
|
||||||
|
// version: 4
|
||||||
|
|
||||||
|
/// Default bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub const DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY: usize = 256;
|
||||||
|
/// Default number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub const DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY: usize = 8;
|
||||||
|
/// Default cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub const DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
|
||||||
|
/// Maximum bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub const MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY: usize = 65_536;
|
||||||
|
/// Maximum number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub const MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY: usize = 64;
|
||||||
|
/// Maximum cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub const MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
|
||||||
|
/// Minimum bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub const MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY: usize = 1;
|
||||||
|
/// Minimum number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub const MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY: usize = 1;
|
||||||
|
/// Minimum cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub const MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(100);
|
||||||
|
|
||||||
|
/// Validated source-neutral runtime settings for one continuous RAW transaction ingest Worker.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct RawTransactionIngestSettings {
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
worker_id: ksp_worker_api::WorkerId,
|
||||||
|
admission_queue_capacity: usize,
|
||||||
|
persistence_concurrency: usize,
|
||||||
|
shutdown_drain_timeout: std::time::Duration,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestSettings {
|
||||||
|
/// Creates settings from validated network/Worker identities and explicit bounded runtime limits.
|
||||||
|
pub fn new(
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
worker_id: ksp_worker_api::WorkerId,
|
||||||
|
admission_queue_capacity: usize,
|
||||||
|
persistence_concurrency: usize,
|
||||||
|
shutdown_drain_timeout: std::time::Duration,
|
||||||
|
) -> ksp_core_lib::Result<Self> {
|
||||||
|
if let std::result::Result::Err(error) = validate_inclusive_usize(
|
||||||
|
admission_queue_capacity,
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
"admission_queue_capacity",
|
||||||
|
) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = validate_inclusive_usize(
|
||||||
|
persistence_concurrency,
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
"persistence_concurrency",
|
||||||
|
) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
if !(crate::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT..=crate::MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT)
|
||||||
|
.contains(&shutdown_drain_timeout)
|
||||||
|
{
|
||||||
|
return std::result::Result::Err(crate::settings_error("shutdown_drain_timeout"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(Self { network, worker_id, admission_queue_capacity, persistence_concurrency, shutdown_drain_timeout });
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates settings using the stable V1 runtime defaults for one validated network and Worker identity.
|
||||||
|
#[must_use]
|
||||||
|
pub fn with_defaults(network: ksp_store_lib::RawNetworkId, worker_id: ksp_worker_api::WorkerId) -> Self {
|
||||||
|
return Self {
|
||||||
|
network,
|
||||||
|
worker_id,
|
||||||
|
admission_queue_capacity: crate::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
persistence_concurrency: crate::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
shutdown_drain_timeout: crate::DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the Store-compatible logical network binding.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn network(&self) -> &ksp_store_lib::RawNetworkId {
|
||||||
|
return &self.network;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the validated logical identity of this Worker instance.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn worker_id(&self) -> &ksp_worker_api::WorkerId {
|
||||||
|
return &self.worker_id;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the bounded central admission queue capacity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn admission_queue_capacity(&self) -> usize {
|
||||||
|
return self.admission_queue_capacity;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the bounded maximum number of concurrent Store persistence operations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn persistence_concurrency(&self) -> usize {
|
||||||
|
return self.persistence_concurrency;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the bounded cooperative shutdown drain deadline.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn shutdown_drain_timeout(&self) -> std::time::Duration {
|
||||||
|
return self.shutdown_drain_timeout;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionIngestSettings {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionIngestSettings")
|
||||||
|
.field("network", &self.network)
|
||||||
|
.field("worker_id", &self.worker_id)
|
||||||
|
.field("admission_queue_capacity", &self.admission_queue_capacity)
|
||||||
|
.field("persistence_concurrency", &self.persistence_concurrency)
|
||||||
|
.field("shutdown_drain_timeout", &self.shutdown_drain_timeout)
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_inclusive_usize(value: usize, minimum: usize, maximum: usize, field: &'static str) -> ksp_core_lib::Result<()> {
|
||||||
|
if !(minimum..=maximum).contains(&value) {
|
||||||
|
return std::result::Result::Err(crate::settings_error(field));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/settings.rs"]
|
||||||
|
mod tests;
|
||||||
820
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
Normal file
820
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
Normal file
@@ -0,0 +1,820 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
|
||||||
|
// version: 5
|
||||||
|
|
||||||
|
/// Runtime-neutral boxed future resolving to one newer concrete RAW transaction ingest Worker snapshot.
|
||||||
|
pub type RawTransactionIngestSnapshotFuture<'a> =
|
||||||
|
std::pin::Pin<std::boxed::Box<dyn std::future::Future<Output = crate::RawTransactionIngestSnapshot> + std::marker::Send + 'a>>;
|
||||||
|
|
||||||
|
/// Source-neutral lifecycle state of the productive RAW transaction ingest source.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
|
||||||
|
pub enum RawTransactionIngestSourceState {
|
||||||
|
/// The productive source is active.
|
||||||
|
Active,
|
||||||
|
/// Transport is performing one bounded reconnect/replay attempt.
|
||||||
|
Reconnecting,
|
||||||
|
/// Cooperative source shutdown has started.
|
||||||
|
Closing,
|
||||||
|
/// The productive source closed cleanly.
|
||||||
|
Closed,
|
||||||
|
/// The productive source reached a terminal failure.
|
||||||
|
Failed,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Private latest-value source-processing projection emitted by the productive source task.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub(crate) struct RawTransactionIngestProcessingFrontierProjection {
|
||||||
|
hydration_pending: usize,
|
||||||
|
processing_frontier_slot: std::option::Option<u64>,
|
||||||
|
oldest_pending_slot: std::option::Option<u64>,
|
||||||
|
source_state: std::option::Option<crate::RawTransactionIngestSourceState>,
|
||||||
|
source_total: usize,
|
||||||
|
source_active: usize,
|
||||||
|
source_reconnecting: usize,
|
||||||
|
source_failed: usize,
|
||||||
|
source_reconnect_total: u64,
|
||||||
|
source_replay_attempt_total: u64,
|
||||||
|
source_continuity_gap_total: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestProcessingFrontierProjection {
|
||||||
|
/// Returns the empty run-local processing projection used before the source observes work.
|
||||||
|
pub(crate) const fn empty() -> Self {
|
||||||
|
return Self {
|
||||||
|
hydration_pending: 0,
|
||||||
|
processing_frontier_slot: std::option::Option::None,
|
||||||
|
oldest_pending_slot: std::option::Option::None,
|
||||||
|
source_state: std::option::Option::None,
|
||||||
|
source_total: 0,
|
||||||
|
source_active: 0,
|
||||||
|
source_reconnecting: 0,
|
||||||
|
source_failed: 0,
|
||||||
|
source_reconnect_total: 0,
|
||||||
|
source_replay_attempt_total: 0,
|
||||||
|
source_continuity_gap_total: 0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one run-local processing projection from bounded source-owned state.
|
||||||
|
pub(crate) const fn new(
|
||||||
|
hydration_pending: usize,
|
||||||
|
processing_frontier_slot: std::option::Option<u64>,
|
||||||
|
oldest_pending_slot: std::option::Option<u64>,
|
||||||
|
) -> Self {
|
||||||
|
return Self {
|
||||||
|
hydration_pending,
|
||||||
|
processing_frontier_slot,
|
||||||
|
oldest_pending_slot,
|
||||||
|
source_state: std::option::Option::None,
|
||||||
|
source_total: 0,
|
||||||
|
source_active: 0,
|
||||||
|
source_reconnecting: 0,
|
||||||
|
source_failed: 0,
|
||||||
|
source_reconnect_total: 0,
|
||||||
|
source_replay_attempt_total: 0,
|
||||||
|
source_continuity_gap_total: 0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of source signals still pending hydration/admission processing.
|
||||||
|
pub(crate) const fn hydration_pending(&self) -> usize {
|
||||||
|
return self.hydration_pending;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the highest actually observed slot not blocked by older pending source work.
|
||||||
|
pub(crate) const fn processing_frontier_slot(&self) -> std::option::Option<u64> {
|
||||||
|
return self.processing_frontier_slot;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the oldest actually observed slot that still owns pending source work.
|
||||||
|
pub(crate) const fn oldest_pending_slot(&self) -> std::option::Option<u64> {
|
||||||
|
return self.oldest_pending_slot;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns a copy carrying the latest source reconnect/replay projection.
|
||||||
|
pub(crate) const fn with_source_continuity(
|
||||||
|
mut self,
|
||||||
|
source_state: std::option::Option<crate::RawTransactionIngestSourceState>,
|
||||||
|
source_reconnect_total: u64,
|
||||||
|
source_replay_attempt_total: u64,
|
||||||
|
source_continuity_gap_total: u64,
|
||||||
|
) -> Self {
|
||||||
|
self.source_state = source_state;
|
||||||
|
self.source_reconnect_total = source_reconnect_total;
|
||||||
|
self.source_replay_attempt_total = source_replay_attempt_total;
|
||||||
|
self.source_continuity_gap_total = source_continuity_gap_total;
|
||||||
|
return self;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns a copy carrying source-neutral multi-source lifecycle counts.
|
||||||
|
pub(crate) const fn with_source_counts(mut self, source_total: usize, source_active: usize, source_reconnecting: usize, source_failed: usize) -> Self {
|
||||||
|
self.source_total = source_total;
|
||||||
|
self.source_active = source_active;
|
||||||
|
self.source_reconnecting = source_reconnecting;
|
||||||
|
self.source_failed = source_failed;
|
||||||
|
return self;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the configured logical source count carried by this private projection.
|
||||||
|
pub(crate) const fn source_total(&self) -> usize {
|
||||||
|
return self.source_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of sources currently projected Active.
|
||||||
|
pub(crate) const fn source_active(&self) -> usize {
|
||||||
|
return self.source_active;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of sources currently projected Reconnecting.
|
||||||
|
pub(crate) const fn source_reconnecting(&self) -> usize {
|
||||||
|
return self.source_reconnecting;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of sources currently projected Failed.
|
||||||
|
pub(crate) const fn source_failed(&self) -> usize {
|
||||||
|
return self.source_failed;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest source-neutral lifecycle state carried by this private projection.
|
||||||
|
pub(crate) const fn source_state(&self) -> std::option::Option<crate::RawTransactionIngestSourceState> {
|
||||||
|
return self.source_state;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns successful reconnects carried by this private projection.
|
||||||
|
pub(crate) const fn source_reconnect_total(&self) -> u64 {
|
||||||
|
return self.source_reconnect_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns replay-bearing reconnect attempts carried by this private projection.
|
||||||
|
pub(crate) const fn source_replay_attempt_total(&self) -> u64 {
|
||||||
|
return self.source_replay_attempt_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns proven replay-retention gaps carried by this private projection.
|
||||||
|
pub(crate) const fn source_continuity_gap_total(&self) -> u64 {
|
||||||
|
return self.source_continuity_gap_total;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Complete safe latest-value snapshot of one continuous RAW transaction ingest Worker.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct RawTransactionIngestSnapshot {
|
||||||
|
worker: ksp_worker_api::WorkerSnapshot,
|
||||||
|
admission_queue_capacity: usize,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
persistence_concurrency: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
admitted_total: u64,
|
||||||
|
canonicalized_total: u64,
|
||||||
|
persisted_total: u64,
|
||||||
|
entity_inserted_total: u64,
|
||||||
|
entity_already_present_total: u64,
|
||||||
|
entity_skipped_purged_total: u64,
|
||||||
|
observation_inserted_total: u64,
|
||||||
|
observation_already_present_total: u64,
|
||||||
|
content_conflict_total: u64,
|
||||||
|
store_failure_total: u64,
|
||||||
|
source_failure_total: u64,
|
||||||
|
backpressure_wait_total: u64,
|
||||||
|
hydration_pending: usize,
|
||||||
|
processing_frontier_slot: std::option::Option<u64>,
|
||||||
|
oldest_pending_slot: std::option::Option<u64>,
|
||||||
|
source_state: std::option::Option<crate::RawTransactionIngestSourceState>,
|
||||||
|
source_total: usize,
|
||||||
|
source_active: usize,
|
||||||
|
source_reconnecting: usize,
|
||||||
|
source_failed: usize,
|
||||||
|
source_reconnect_total: u64,
|
||||||
|
source_replay_attempt_total: u64,
|
||||||
|
source_continuity_gap_total: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestSnapshot {
|
||||||
|
/// Returns the common Worker projection carried by this concrete snapshot.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn worker_snapshot(&self) -> &ksp_worker_api::WorkerSnapshot {
|
||||||
|
return &self.worker;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the configured bounded admission queue capacity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn admission_queue_capacity(&self) -> usize {
|
||||||
|
return self.admission_queue_capacity;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest observed number of queued ingress entries awaiting supervisor admission.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn admission_queue_depth(&self) -> usize {
|
||||||
|
return self.admission_queue_depth;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the configured maximum number of concurrent Store persistence operations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn persistence_concurrency(&self) -> usize {
|
||||||
|
return self.persistence_concurrency;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest observed number of in-flight Store persistence operations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn in_flight_persistence(&self) -> usize {
|
||||||
|
return self.in_flight_persistence;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of ingress entries removed from the bounded admission queue.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn admitted_total(&self) -> u64 {
|
||||||
|
return self.admitted_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of admitted ingress entries successfully canonicalized through the Common RAW contract.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn canonicalized_total(&self) -> u64 {
|
||||||
|
return self.canonicalized_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of successful atomic Store persistence outcomes.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn persisted_total(&self) -> u64 {
|
||||||
|
return self.persisted_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of newly inserted canonical RAW transaction entities.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn entity_inserted_total(&self) -> u64 {
|
||||||
|
return self.entity_inserted_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of identical canonical RAW transaction entities already durable.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn entity_already_present_total(&self) -> u64 {
|
||||||
|
return self.entity_already_present_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of durable purge tombstones respected by normal persistence.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn entity_skipped_purged_total(&self) -> u64 {
|
||||||
|
return self.entity_skipped_purged_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of newly inserted deterministic acquisition observations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn observation_inserted_total(&self) -> u64 {
|
||||||
|
return self.observation_inserted_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of deterministic acquisition observations already durable.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn observation_already_present_total(&self) -> u64 {
|
||||||
|
return self.observation_already_present_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of durable Store content conflicts observed by this run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn content_conflict_total(&self) -> u64 {
|
||||||
|
return self.content_conflict_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of non-conflict Store persistence failures observed by this run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn store_failure_total(&self) -> u64 {
|
||||||
|
return self.store_failure_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of source-task failures observed by this run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_failure_total(&self) -> u64 {
|
||||||
|
return self.source_failure_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of supervisor dequeues that observed the bounded admission queue at full capacity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn backpressure_wait_total(&self) -> u64 {
|
||||||
|
return self.backpressure_wait_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest number of source signals pending hydration/admission processing.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn hydration_pending(&self) -> usize {
|
||||||
|
return self.hydration_pending;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the run-local processing frontier over actually observed source work.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn processing_frontier_slot(&self) -> std::option::Option<u64> {
|
||||||
|
return self.processing_frontier_slot;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the oldest actually observed slot that still owns pending source work.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn oldest_pending_slot(&self) -> std::option::Option<u64> {
|
||||||
|
return self.oldest_pending_slot;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the configured number of logical live sources for this Worker run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_total(&self) -> usize {
|
||||||
|
return self.source_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest number of sources projected Active.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_active(&self) -> usize {
|
||||||
|
return self.source_active;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest number of sources projected Reconnecting.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_reconnecting(&self) -> usize {
|
||||||
|
return self.source_reconnecting;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest number of sources projected Failed.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_failed(&self) -> usize {
|
||||||
|
return self.source_failed;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest source-neutral lifecycle state when the productive source has started.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_state(&self) -> std::option::Option<crate::RawTransactionIngestSourceState> {
|
||||||
|
return self.source_state;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns successful automatic Yellowstone reconnects observed by this Worker run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_reconnect_total(&self) -> u64 {
|
||||||
|
return self.source_reconnect_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns Yellowstone replay-bearing reconnect attempts observed by this Worker run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_replay_attempt_total(&self) -> u64 {
|
||||||
|
return self.source_replay_attempt_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns replay retention gaps conservatively proven by Transport during this Worker run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_continuity_gap_total(&self) -> u64 {
|
||||||
|
return self.source_continuity_gap_total;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionIngestSnapshot {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionIngestSnapshot")
|
||||||
|
.field("worker", &self.worker)
|
||||||
|
.field("admission_queue_capacity", &self.admission_queue_capacity)
|
||||||
|
.field("admission_queue_depth", &self.admission_queue_depth)
|
||||||
|
.field("persistence_concurrency", &self.persistence_concurrency)
|
||||||
|
.field("in_flight_persistence", &self.in_flight_persistence)
|
||||||
|
.field("admitted_total", &self.admitted_total)
|
||||||
|
.field("canonicalized_total", &self.canonicalized_total)
|
||||||
|
.field("persisted_total", &self.persisted_total)
|
||||||
|
.field("entity_inserted_total", &self.entity_inserted_total)
|
||||||
|
.field("entity_already_present_total", &self.entity_already_present_total)
|
||||||
|
.field("entity_skipped_purged_total", &self.entity_skipped_purged_total)
|
||||||
|
.field("observation_inserted_total", &self.observation_inserted_total)
|
||||||
|
.field("observation_already_present_total", &self.observation_already_present_total)
|
||||||
|
.field("content_conflict_total", &self.content_conflict_total)
|
||||||
|
.field("store_failure_total", &self.store_failure_total)
|
||||||
|
.field("source_failure_total", &self.source_failure_total)
|
||||||
|
.field("backpressure_wait_total", &self.backpressure_wait_total)
|
||||||
|
.field("hydration_pending", &self.hydration_pending)
|
||||||
|
.field("processing_frontier_slot", &self.processing_frontier_slot)
|
||||||
|
.field("oldest_pending_slot", &self.oldest_pending_slot)
|
||||||
|
.field("source_state", &self.source_state)
|
||||||
|
.field("source_total", &self.source_total)
|
||||||
|
.field("source_active", &self.source_active)
|
||||||
|
.field("source_reconnecting", &self.source_reconnecting)
|
||||||
|
.field("source_failed", &self.source_failed)
|
||||||
|
.field("source_reconnect_total", &self.source_reconnect_total)
|
||||||
|
.field("source_replay_attempt_total", &self.source_replay_attempt_total)
|
||||||
|
.field("source_continuity_gap_total", &self.source_continuity_gap_total)
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cloneable latest-value source exposing concrete and common Worker snapshots from one shared watch state.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct RawTransactionIngestSnapshotSource {
|
||||||
|
receiver: tokio::sync::watch::Receiver<crate::RawTransactionIngestSnapshot>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestSnapshotSource {
|
||||||
|
/// Returns the complete current concrete snapshot without replaying intermediate updates.
|
||||||
|
#[must_use]
|
||||||
|
pub fn current(&self) -> crate::RawTransactionIngestSnapshot {
|
||||||
|
return self.receiver.borrow().clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reports whether the private publisher has closed after the runtime task returned.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn is_closed(&self) -> bool {
|
||||||
|
return self.receiver.has_changed().is_err();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Waits for one concrete snapshot newer than `observed`, coalescing intermediate updates to the latest value.
|
||||||
|
#[must_use]
|
||||||
|
pub fn wait_for_change(&self, observed: ksp_worker_api::WorkerSnapshotSequence) -> crate::RawTransactionIngestSnapshotFuture<'_> {
|
||||||
|
let mut receiver = self.receiver.clone();
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
loop {
|
||||||
|
let current = receiver.borrow_and_update().clone();
|
||||||
|
if current.worker_snapshot().sequence().is_after(observed) {
|
||||||
|
return current;
|
||||||
|
}
|
||||||
|
let changed = receiver.changed().await;
|
||||||
|
if changed.is_err() {
|
||||||
|
return receiver.borrow_and_update().clone();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ksp_worker_api::WorkerSnapshotSource for crate::RawTransactionIngestSnapshotSource {
|
||||||
|
fn current(&self) -> ksp_worker_api::WorkerSnapshot {
|
||||||
|
return self.receiver.borrow().worker_snapshot().clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn wait_for_change(&self, observed: ksp_worker_api::WorkerSnapshotSequence) -> ksp_worker_api::WorkerSnapshotFuture<'_> {
|
||||||
|
let mut receiver = self.receiver.clone();
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
loop {
|
||||||
|
let current = receiver.borrow_and_update().worker_snapshot().clone();
|
||||||
|
if current.sequence().is_after(observed) {
|
||||||
|
return current;
|
||||||
|
}
|
||||||
|
let changed = receiver.changed().await;
|
||||||
|
if changed.is_err() {
|
||||||
|
return receiver.borrow_and_update().worker_snapshot().clone();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionIngestSnapshotSource {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
let current = self.receiver.borrow();
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionIngestSnapshotSource")
|
||||||
|
.field("sequence", ¤t.worker_snapshot().sequence())
|
||||||
|
.field("state", ¤t.worker_snapshot().state())
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Private latest-value publisher and checked counter owner shared by the Worker supervisor.
|
||||||
|
pub(crate) struct RawTransactionIngestSnapshotPublisher {
|
||||||
|
sender: tokio::sync::watch::Sender<crate::RawTransactionIngestSnapshot>,
|
||||||
|
snapshot: crate::RawTransactionIngestSnapshot,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestSnapshotPublisher {
|
||||||
|
/// Creates the initial `Starting` snapshot stream for one validated Worker run.
|
||||||
|
pub(crate) fn new(
|
||||||
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
source_total: usize,
|
||||||
|
) -> (crate::RawTransactionIngestSnapshotPublisher, crate::RawTransactionIngestSnapshotSource) {
|
||||||
|
let worker = ksp_worker_api::WorkerSnapshot::new(
|
||||||
|
lifecycle.id().clone(),
|
||||||
|
lifecycle.kind().clone(),
|
||||||
|
ksp_worker_api::WorkerSnapshotSequence::initial(),
|
||||||
|
lifecycle.state(),
|
||||||
|
ksp_worker_api::WorkerHealth::Unknown,
|
||||||
|
ksp_worker_api::WorkerActivity::Unknown,
|
||||||
|
);
|
||||||
|
let snapshot = crate::RawTransactionIngestSnapshot {
|
||||||
|
worker,
|
||||||
|
admission_queue_capacity: settings.admission_queue_capacity(),
|
||||||
|
admission_queue_depth: 0,
|
||||||
|
persistence_concurrency: settings.persistence_concurrency(),
|
||||||
|
in_flight_persistence: 0,
|
||||||
|
admitted_total: 0,
|
||||||
|
canonicalized_total: 0,
|
||||||
|
persisted_total: 0,
|
||||||
|
entity_inserted_total: 0,
|
||||||
|
entity_already_present_total: 0,
|
||||||
|
entity_skipped_purged_total: 0,
|
||||||
|
observation_inserted_total: 0,
|
||||||
|
observation_already_present_total: 0,
|
||||||
|
content_conflict_total: 0,
|
||||||
|
store_failure_total: 0,
|
||||||
|
source_failure_total: 0,
|
||||||
|
backpressure_wait_total: 0,
|
||||||
|
hydration_pending: 0,
|
||||||
|
processing_frontier_slot: std::option::Option::None,
|
||||||
|
oldest_pending_slot: std::option::Option::None,
|
||||||
|
source_state: std::option::Option::None,
|
||||||
|
source_total,
|
||||||
|
source_active: 0,
|
||||||
|
source_reconnecting: 0,
|
||||||
|
source_failed: 0,
|
||||||
|
source_reconnect_total: 0,
|
||||||
|
source_replay_attempt_total: 0,
|
||||||
|
source_continuity_gap_total: 0,
|
||||||
|
};
|
||||||
|
let (sender, receiver) = tokio::sync::watch::channel(snapshot.clone());
|
||||||
|
return (Self { sender, snapshot }, crate::RawTransactionIngestSnapshotSource { receiver });
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Forces one terminal latest value without advancing the sequence, used only when sequence publication itself is exhausted or invalid.
|
||||||
|
pub(crate) fn force_terminal(&mut self, state: ksp_worker_api::WorkerState) {
|
||||||
|
let worker = ksp_worker_api::WorkerSnapshot::new(
|
||||||
|
self.snapshot.worker.id().clone(),
|
||||||
|
self.snapshot.worker.kind().clone(),
|
||||||
|
self.snapshot.worker.sequence(),
|
||||||
|
state,
|
||||||
|
health_for_state(
|
||||||
|
state,
|
||||||
|
self.snapshot.worker.health(),
|
||||||
|
self.snapshot.source_total,
|
||||||
|
self.snapshot.source_active,
|
||||||
|
self.snapshot.source_reconnecting,
|
||||||
|
self.snapshot.source_failed,
|
||||||
|
),
|
||||||
|
ksp_worker_api::WorkerActivity::Idle,
|
||||||
|
);
|
||||||
|
self.snapshot.worker = worker;
|
||||||
|
self.snapshot.admission_queue_depth = 0;
|
||||||
|
self.snapshot.in_flight_persistence = 0;
|
||||||
|
self.sender.send_replace(self.snapshot.clone());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publishes one lifecycle/depth refresh with a strictly newer common sequence.
|
||||||
|
pub(crate) fn publish_state(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one dequeued ingress that failed canonicalization and publishes the resulting latest value.
|
||||||
|
pub(crate) fn record_admission_failure(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
backpressure_wait_observed: bool,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
let admitted_total = match checked_counter(self.snapshot.admitted_total, "admitted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let backpressure_wait_total =
|
||||||
|
match checked_optional_counter(self.snapshot.backpressure_wait_total, backpressure_wait_observed, "backpressure_wait_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.admitted_total = admitted_total;
|
||||||
|
self.snapshot.backpressure_wait_total = backpressure_wait_total;
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one successfully canonicalized admitted ingress and publishes the resulting latest value.
|
||||||
|
pub(crate) fn record_admission_success(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
backpressure_wait_observed: bool,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
let admitted_total = match checked_counter(self.snapshot.admitted_total, "admitted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let backpressure_wait_total =
|
||||||
|
match checked_optional_counter(self.snapshot.backpressure_wait_total, backpressure_wait_observed, "backpressure_wait_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let canonicalized_total = match checked_counter(self.snapshot.canonicalized_total, "canonicalized_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.admitted_total = admitted_total;
|
||||||
|
self.snapshot.backpressure_wait_total = backpressure_wait_total;
|
||||||
|
self.snapshot.canonicalized_total = canonicalized_total;
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publishes one latest run-local processing-frontier projection emitted by the source task.
|
||||||
|
pub(crate) fn record_processing_frontier(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
projection: crate::RawTransactionIngestProcessingFrontierProjection,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
self.snapshot.hydration_pending = projection.hydration_pending();
|
||||||
|
self.snapshot.processing_frontier_slot = projection.processing_frontier_slot();
|
||||||
|
self.snapshot.oldest_pending_slot = projection.oldest_pending_slot();
|
||||||
|
self.snapshot.source_state = projection.source_state();
|
||||||
|
if projection.source_total() > 0 {
|
||||||
|
self.snapshot.source_total = projection.source_total();
|
||||||
|
self.snapshot.source_active = projection.source_active();
|
||||||
|
self.snapshot.source_reconnecting = projection.source_reconnecting();
|
||||||
|
self.snapshot.source_failed = projection.source_failed();
|
||||||
|
}
|
||||||
|
self.snapshot.source_reconnect_total = projection.source_reconnect_total();
|
||||||
|
self.snapshot.source_replay_attempt_total = projection.source_replay_attempt_total();
|
||||||
|
self.snapshot.source_continuity_gap_total = projection.source_continuity_gap_total();
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one failed private source task without retaining provider-specific error material.
|
||||||
|
pub(crate) fn record_source_failure(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
let source_failure_total = match checked_counter(self.snapshot.source_failure_total, "source_failure_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.source_failure_total = source_failure_total;
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one successful Store persistence outcome and publishes the resulting latest value.
|
||||||
|
pub(crate) fn record_persistence_success(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
outcome: crate::RawTransactionIngestPersistenceOutcome,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
let persisted_total = match checked_counter(self.snapshot.persisted_total, "persisted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let mut entity_inserted_total = self.snapshot.entity_inserted_total;
|
||||||
|
let mut entity_already_present_total = self.snapshot.entity_already_present_total;
|
||||||
|
let mut entity_skipped_purged_total = self.snapshot.entity_skipped_purged_total;
|
||||||
|
let mut observation_inserted_total = self.snapshot.observation_inserted_total;
|
||||||
|
let mut observation_already_present_total = self.snapshot.observation_already_present_total;
|
||||||
|
match outcome.entity() {
|
||||||
|
crate::RawTransactionIngestEntityPersistence::Inserted => {
|
||||||
|
entity_inserted_total = match checked_counter(entity_inserted_total, "entity_inserted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::RawTransactionIngestEntityPersistence::AlreadyPresent => {
|
||||||
|
entity_already_present_total = match checked_counter(entity_already_present_total, "entity_already_present_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::RawTransactionIngestEntityPersistence::SkippedPurged => {
|
||||||
|
entity_skipped_purged_total = match checked_counter(entity_skipped_purged_total, "entity_skipped_purged_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
}
|
||||||
|
match outcome.observation() {
|
||||||
|
crate::RawTransactionIngestObservationPersistence::Inserted => {
|
||||||
|
observation_inserted_total = match checked_counter(observation_inserted_total, "observation_inserted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::RawTransactionIngestObservationPersistence::AlreadyPresent => {
|
||||||
|
observation_already_present_total = match checked_counter(observation_already_present_total, "observation_already_present_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::RawTransactionIngestObservationPersistence::NotRecorded => {},
|
||||||
|
}
|
||||||
|
self.snapshot.persisted_total = persisted_total;
|
||||||
|
self.snapshot.entity_inserted_total = entity_inserted_total;
|
||||||
|
self.snapshot.entity_already_present_total = entity_already_present_total;
|
||||||
|
self.snapshot.entity_skipped_purged_total = entity_skipped_purged_total;
|
||||||
|
self.snapshot.observation_inserted_total = observation_inserted_total;
|
||||||
|
self.snapshot.observation_already_present_total = observation_already_present_total;
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one classified persistence fault counter and publishes the resulting latest value.
|
||||||
|
pub(crate) fn record_persistence_fault(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
code: ksp_core_lib::ErrorCode,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
if code == crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT {
|
||||||
|
let next = match checked_counter(self.snapshot.content_conflict_total, "content_conflict_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.content_conflict_total = next;
|
||||||
|
} else if code == crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED {
|
||||||
|
let next = match checked_counter(self.snapshot.store_failure_total, "store_failure_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.store_failure_total = next;
|
||||||
|
}
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn publish(&mut self, state: ksp_worker_api::WorkerState, admission_queue_depth: usize, in_flight_persistence: usize) -> ksp_core_lib::Result<()> {
|
||||||
|
let sequence = match self.snapshot.worker.sequence().next() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::counter_exhausted_error("snapshot_sequence")),
|
||||||
|
};
|
||||||
|
let worker = ksp_worker_api::WorkerSnapshot::new(
|
||||||
|
self.snapshot.worker.id().clone(),
|
||||||
|
self.snapshot.worker.kind().clone(),
|
||||||
|
sequence,
|
||||||
|
state,
|
||||||
|
health_for_state(
|
||||||
|
state,
|
||||||
|
self.snapshot.worker.health(),
|
||||||
|
self.snapshot.source_total,
|
||||||
|
self.snapshot.source_active,
|
||||||
|
self.snapshot.source_reconnecting,
|
||||||
|
self.snapshot.source_failed,
|
||||||
|
),
|
||||||
|
activity_for_state(state, admission_queue_depth, in_flight_persistence, self.snapshot.hydration_pending, self.snapshot.source_state),
|
||||||
|
);
|
||||||
|
self.snapshot.worker = worker;
|
||||||
|
self.snapshot.admission_queue_depth = admission_queue_depth;
|
||||||
|
self.snapshot.in_flight_persistence = in_flight_persistence;
|
||||||
|
self.sender.send_replace(self.snapshot.clone());
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn activity_for_state(
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
hydration_pending: usize,
|
||||||
|
source_state: std::option::Option<crate::RawTransactionIngestSourceState>,
|
||||||
|
) -> ksp_worker_api::WorkerActivity {
|
||||||
|
if admission_queue_depth > 0
|
||||||
|
|| in_flight_persistence > 0
|
||||||
|
|| hydration_pending > 0
|
||||||
|
|| matches!(
|
||||||
|
source_state,
|
||||||
|
std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting)
|
||||||
|
| std::option::Option::Some(crate::RawTransactionIngestSourceState::Closing)
|
||||||
|
)
|
||||||
|
{
|
||||||
|
return ksp_worker_api::WorkerActivity::Active;
|
||||||
|
}
|
||||||
|
return match state {
|
||||||
|
ksp_worker_api::WorkerState::Running
|
||||||
|
| ksp_worker_api::WorkerState::Stopping
|
||||||
|
| ksp_worker_api::WorkerState::Stopped
|
||||||
|
| ksp_worker_api::WorkerState::Faulted(_) => ksp_worker_api::WorkerActivity::Idle,
|
||||||
|
_ => ksp_worker_api::WorkerActivity::Unknown,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn checked_counter(current: u64, field: &'static str) -> ksp_core_lib::Result<u64> {
|
||||||
|
return match current.checked_add(1) {
|
||||||
|
std::option::Option::Some(value) => std::result::Result::Ok(value),
|
||||||
|
std::option::Option::None => std::result::Result::Err(crate::counter_exhausted_error(field)),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn checked_optional_counter(current: u64, increment: bool, field: &'static str) -> ksp_core_lib::Result<u64> {
|
||||||
|
if !increment {
|
||||||
|
return std::result::Result::Ok(current);
|
||||||
|
}
|
||||||
|
return checked_counter(current, field);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn health_for_state(
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
previous: ksp_worker_api::WorkerHealth,
|
||||||
|
source_total: usize,
|
||||||
|
source_active: usize,
|
||||||
|
source_reconnecting: usize,
|
||||||
|
source_failed: usize,
|
||||||
|
) -> ksp_worker_api::WorkerHealth {
|
||||||
|
return match state {
|
||||||
|
ksp_worker_api::WorkerState::Running if source_failed > 0 => ksp_worker_api::WorkerHealth::Unhealthy,
|
||||||
|
ksp_worker_api::WorkerState::Running if source_reconnecting > 0 => ksp_worker_api::WorkerHealth::Degraded,
|
||||||
|
ksp_worker_api::WorkerState::Running if source_total > 0 && source_active < source_total => ksp_worker_api::WorkerHealth::Degraded,
|
||||||
|
ksp_worker_api::WorkerState::Running => ksp_worker_api::WorkerHealth::Healthy,
|
||||||
|
ksp_worker_api::WorkerState::Stopping | ksp_worker_api::WorkerState::Stopped => previous,
|
||||||
|
ksp_worker_api::WorkerState::Faulted(_) => ksp_worker_api::WorkerHealth::Unhealthy,
|
||||||
|
_ => ksp_worker_api::WorkerHealth::Unknown,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/snapshot.rs"]
|
||||||
|
mod tests;
|
||||||
@@ -0,0 +1,365 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/cross_layer_completeness.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
//! Cross-layer completeness and security canaries through `0.3.13-pre.012`.
|
||||||
|
|
||||||
|
fn dependency_names(manifest: &str) -> std::vec::Vec<&str> {
|
||||||
|
let mut section = "";
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for line in manifest.lines() {
|
||||||
|
let trimmed = line.trim();
|
||||||
|
if trimmed.starts_with('[') && trimmed.ends_with(']') {
|
||||||
|
section = trimmed;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if section != "[dependencies]" || trimmed.is_empty() || trimmed.starts_with('#') {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match trimmed.split_once('=') {
|
||||||
|
std::option::Option::Some((name, _)) => name.trim().trim_end_matches(".workspace"),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
names.sort_unstable();
|
||||||
|
return names;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_010_dependency_firewall_is_exact_across_transport_worker_common_raw_store() {
|
||||||
|
let transport = dependency_names(include_str!("../../ksp-onchain-transport-lib/Cargo.toml"));
|
||||||
|
let worker = dependency_names(include_str!("../Cargo.toml"));
|
||||||
|
let common_raw = dependency_names(include_str!("../../ksp-raw-transaction-lib/Cargo.toml"));
|
||||||
|
let store_api = dependency_names(include_str!("../../ksp-store-api/Cargo.toml"));
|
||||||
|
let store = dependency_names(include_str!("../../ksp-store-lib/Cargo.toml"));
|
||||||
|
assert_eq!(
|
||||||
|
transport,
|
||||||
|
std::vec![
|
||||||
|
"futures-util",
|
||||||
|
"http",
|
||||||
|
"ksp-core-lib",
|
||||||
|
"ksp-logging-lib",
|
||||||
|
"reqwest",
|
||||||
|
"serde",
|
||||||
|
"serde_json",
|
||||||
|
"tokio",
|
||||||
|
"tokio-tungstenite",
|
||||||
|
"tonic",
|
||||||
|
"tonic-prost",
|
||||||
|
"yellowstone-grpc-proto",
|
||||||
|
]
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
worker,
|
||||||
|
std::vec![
|
||||||
|
"ksp-core-lib",
|
||||||
|
"ksp-logging-lib",
|
||||||
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-raw-transaction-lib",
|
||||||
|
"ksp-store-lib",
|
||||||
|
"ksp-worker-api",
|
||||||
|
"sha2",
|
||||||
|
"tokio",
|
||||||
|
]
|
||||||
|
);
|
||||||
|
assert_eq!(common_raw, std::vec!["base64", "ksp-core-lib", "ksp-store-api", "serde_json", "sha2"]);
|
||||||
|
assert_eq!(store_api, std::vec!["ksp-core-lib"]);
|
||||||
|
assert_eq!(store, std::vec!["ksp-logging-lib", "ksp-store-api", "ksp-store-postgres-lib"]);
|
||||||
|
for forbidden in ["ksp-config-lib", "ksp-job-api", "ksp-job-backfill-lib", "ksp-store-lib", "ksp-worker-api", "ksp-worker-raw-transaction-ingest-lib"] {
|
||||||
|
assert!(!transport.contains(&forbidden), "Transport crossed upward dependency boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"ksp-config-lib",
|
||||||
|
"ksp-job-api",
|
||||||
|
"ksp-job-backfill-lib",
|
||||||
|
"ksp-store-api",
|
||||||
|
"ksp-store-postgres-lib",
|
||||||
|
"reqwest",
|
||||||
|
"tonic",
|
||||||
|
"yellowstone-grpc-proto",
|
||||||
|
] {
|
||||||
|
assert!(!worker.contains(&forbidden), "Worker crossed direct lower/backend dependency boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"ksp-config-lib",
|
||||||
|
"ksp-job-api",
|
||||||
|
"ksp-job-backfill-lib",
|
||||||
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-store-lib",
|
||||||
|
"ksp-store-postgres-lib",
|
||||||
|
"ksp-worker-api",
|
||||||
|
] {
|
||||||
|
assert!(!common_raw.contains(&forbidden), "Common RAW crossed dependency boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
for forbidden in ["ksp-config-lib", "ksp-onchain-transport-lib", "ksp-raw-transaction-lib", "ksp-store-lib", "ksp-store-postgres-lib", "ksp-worker-api"] {
|
||||||
|
assert!(!store_api.contains(&forbidden), "Store API crossed dependency boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
for forbidden in ["ksp-config-lib", "ksp-job-api", "ksp-job-backfill-lib", "ksp-onchain-transport-lib", "ksp-raw-transaction-lib", "ksp-worker-api"] {
|
||||||
|
assert!(!store.contains(&forbidden), "Store facade crossed dependency boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_010_legacy_v0_fixture_matrix_is_exact_across_transport_worker_and_common_raw() {
|
||||||
|
let worker = include_str!("../unit_tests/runtime_resources.rs");
|
||||||
|
let common_raw_wire = include_str!("../../ksp-raw-transaction-lib/unit_tests/wire.rs");
|
||||||
|
let transport = include_str!("../../ksp-onchain-transport-lib/unit_tests/rpc_transactions.rs");
|
||||||
|
for required in [
|
||||||
|
"pre_004_observed_get_transaction_closes_signal_to_common_raw_ingress_with_composite_provenance",
|
||||||
|
"pre_004_finalized_hydration_preserves_request_and_provenance_commitment",
|
||||||
|
"pre_004_v0_null_and_omitted_wire_fields_preserve_common_raw_semantics",
|
||||||
|
r#"\"version\":\"legacy\""#,
|
||||||
|
r#"\"version\":0"#,
|
||||||
|
] {
|
||||||
|
assert!(worker.contains(required), "Worker Legacy/V0 fixture missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"LEGACY_GOLDEN_BASE64",
|
||||||
|
"V0_GOLDEN_BASE64",
|
||||||
|
"pre_006_legacy_v0_and_v1_wire_goldens_are_exact",
|
||||||
|
"RawSolanaMessageVersion::Legacy",
|
||||||
|
"RawSolanaMessageVersion::V0",
|
||||||
|
] {
|
||||||
|
assert!(common_raw_wire.contains(required), "Common RAW Legacy/V0 golden missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"confirmed_transaction_fixture_preserves_meta_version_and_transaction_index_states",
|
||||||
|
"typed_get_transaction_modern_covers_all_agave_v4_2_1_encoding_labels_and_response_shapes",
|
||||||
|
"SolanaTransactionVersion::Legacy",
|
||||||
|
] {
|
||||||
|
assert!(transport.contains(required), "Transport getTransaction fixture missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_010_redaction_scanner_matrix_is_present_across_all_layers() {
|
||||||
|
let transport_subscribe = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_subscribe.rs");
|
||||||
|
let transport_unary = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_unary.rs");
|
||||||
|
let worker = include_str!("hardening.rs");
|
||||||
|
let common_raw = include_str!("../../ksp-raw-transaction-lib/unit_tests/canonical.rs");
|
||||||
|
let store_api = include_str!("../../ksp-store-api/tests/security_hardening.rs");
|
||||||
|
let store = include_str!("../../ksp-store-lib/tests/hardening_completeness.rs");
|
||||||
|
assert!(transport_subscribe.contains("yellowstone_subscribe_debug_omits_filter_names_and_future_payloads"));
|
||||||
|
assert!(transport_unary.contains("yellowstone_remote_status_does_not_copy_message_details_or_metadata"));
|
||||||
|
for required in [
|
||||||
|
"pre_010_debug_and_settings_errors_redact_worker_identity_and_invalid_values",
|
||||||
|
"v0_3_12_pre_003_private_signal_debug_and_shape_do_not_expose_signature_filters_or_payload",
|
||||||
|
"v0_3_12_pre_004_hydration_provenance_and_remote_material_are_bounded_and_redacted",
|
||||||
|
"v0_3_12_pre_008_reconnect_projection_is_source_neutral_bounded_and_contains_no_replay_material",
|
||||||
|
"v0_3_12_pre_009_hydration_retry_ownership_and_no_orphan_cleanup_are_explicit",
|
||||||
|
] {
|
||||||
|
assert!(worker.contains(required), "Worker security scanner missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(common_raw.contains("pre_002_material_and_wire_debug_do_not_render_transaction_or_meta_material"));
|
||||||
|
assert!(store_api.contains("pre_007_raw_debug_surfaces_do_not_render_payload_hash_signature_or_account_bytes"));
|
||||||
|
assert!(store.contains("pre_009_secret_canary_never_crosses_settings_or_pre_io_error_debug"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_010_public_roots_keep_implementation_modules_private() {
|
||||||
|
let roots = [
|
||||||
|
("transport", include_str!("../../ksp-onchain-transport-lib/src/lib.rs")),
|
||||||
|
("worker", include_str!("../src/lib.rs")),
|
||||||
|
("common_raw", include_str!("../../ksp-raw-transaction-lib/src/lib.rs")),
|
||||||
|
("store_api", include_str!("../../ksp-store-api/src/lib.rs")),
|
||||||
|
("store", include_str!("../../ksp-store-lib/src/lib.rs")),
|
||||||
|
];
|
||||||
|
for (layer, root) in roots {
|
||||||
|
assert!(!root.contains("pub mod "), "public implementation module leaked from {layer}");
|
||||||
|
}
|
||||||
|
let worker = include_str!("../src/lib.rs");
|
||||||
|
for line in worker.lines() {
|
||||||
|
let trimmed = line.trim();
|
||||||
|
if !trimmed.starts_with("pub use ") {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"RawTransactionAdmission",
|
||||||
|
"RawTransactionIngress",
|
||||||
|
"RawTransactionIngestPersistencePort",
|
||||||
|
"JoinHandle",
|
||||||
|
"JoinSet",
|
||||||
|
"HttpTransportPool",
|
||||||
|
"SolanaYellowstoneGrpcSubscribeSession",
|
||||||
|
] {
|
||||||
|
assert!(!trimmed.contains(forbidden), "Worker implementation type leaked publicly: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_all_five_live_source_families_share_one_worker_common_raw_store_pipeline() {
|
||||||
|
let transport_root = include_str!("../../ksp-onchain-transport-lib/src/lib.rs");
|
||||||
|
let transport_http_transactions = include_str!("../../ksp-onchain-transport-lib/src/rpc_transactions.rs");
|
||||||
|
let transport_http_blocks = include_str!("../../ksp-onchain-transport-lib/src/rpc_blocks.rs");
|
||||||
|
let transport_ws_protocol = include_str!("../../ksp-onchain-transport-lib/src/ws_protocol_session.rs");
|
||||||
|
let transport_grpc_stream = include_str!("../../ksp-onchain-transport-lib/src/grpc_stream.rs");
|
||||||
|
let worker_resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let worker_persistence = include_str!("../src/persistence.rs");
|
||||||
|
let common_raw_root = include_str!("../../ksp-raw-transaction-lib/src/lib.rs");
|
||||||
|
let store_root = include_str!("../../ksp-store-lib/src/lib.rs");
|
||||||
|
let store = include_str!("../../ksp-store-lib/src/store.rs");
|
||||||
|
for required in [
|
||||||
|
"HeliusTransaction(crate::RawTransactionIngestHeliusTransactionSource)",
|
||||||
|
"HttpBlockPolling(crate::RawTransactionIngestHttpBlockPollingSource)",
|
||||||
|
"StandardBlock(crate::RawTransactionIngestStandardBlockSource)",
|
||||||
|
"StandardLogs(crate::RawTransactionIngestStandardLogsSource)",
|
||||||
|
"Yellowstone(crate::RawTransactionIngestYellowstoneSource)",
|
||||||
|
"RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"run_live_sources",
|
||||||
|
"RawTransactionIngress",
|
||||||
|
] {
|
||||||
|
assert!(worker_resources.contains(required), "pre.012 Worker source/pipeline contract missing: {required}");
|
||||||
|
}
|
||||||
|
for required in ["persist_raw_transaction_ingest_converged_acquisition", "RawTransactionIngestPersistenceConvergence", "record_observation"] {
|
||||||
|
assert!(worker_persistence.contains(required), "pre.012 Worker persistence contract missing: {required}");
|
||||||
|
}
|
||||||
|
for required in ["SolanaStandardWsSession", "HeliusLaserStreamWsSession"] {
|
||||||
|
assert!(transport_ws_protocol.contains(required), "pre.012 WS Transport facade missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(transport_grpc_stream.contains("pub struct SolanaYellowstoneGrpcSubscribeSession"));
|
||||||
|
assert!(transport_http_transactions.contains("pub async fn get_transaction_observed("));
|
||||||
|
assert!(transport_http_blocks.contains("pub async fn get_block_observed("));
|
||||||
|
assert!(transport_http_blocks.contains("pub async fn get_blocks_with_limit("));
|
||||||
|
for required in ["RawTransactionAcquisition", "RawSolanaMessageVersion", "RawSolanaTransactionWire"] {
|
||||||
|
assert!(common_raw_root.contains(required), "pre.012 Common RAW contract missing: {required}");
|
||||||
|
}
|
||||||
|
for required in ["RawTransactionWrite", "RawTransactionObservationWrite", "RawPayload"] {
|
||||||
|
assert!(store_root.contains(required), "pre.012 Store facade contract missing: {required}");
|
||||||
|
}
|
||||||
|
for required in ["persist_raw_transaction_acquisition", "record_raw_transaction_observation"] {
|
||||||
|
assert!(store.contains(required), "pre.012 Store dispatch contract missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(!transport_root.contains("ksp_raw_transaction_lib"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_legacy_v0_v1_is_proven_from_transport_through_worker_common_raw_to_store() {
|
||||||
|
let transport_grpc = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_subscribe.rs");
|
||||||
|
let transport_blocks = include_str!("../../ksp-onchain-transport-lib/unit_tests/rpc_blocks.rs");
|
||||||
|
let worker = include_str!("../unit_tests/runtime_resources.rs");
|
||||||
|
let worker_resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let worker_persistence = include_str!("../src/persistence.rs");
|
||||||
|
let common_raw_wire = include_str!("../../ksp-raw-transaction-lib/unit_tests/wire.rs");
|
||||||
|
let common_raw_canonical = include_str!("../../ksp-raw-transaction-lib/unit_tests/canonical.rs");
|
||||||
|
let store_api = include_str!("../../ksp-store-api/tests/public_api.rs");
|
||||||
|
let store = include_str!("../../ksp-store-lib/src/store.rs");
|
||||||
|
assert!(transport_grpc.contains("yellowstone_transaction_update_decodes_current_storage_wire_including_v1_config_and_meta"));
|
||||||
|
assert!(transport_blocks.contains("typed_get_block_modern_full_config_preserves_rich_wire_and_simd_fields"));
|
||||||
|
for required in [
|
||||||
|
"v0_3_13_pre_004_standard_block_qualifies_legacy_v0_v1_and_rejects_ambiguous_or_future_versions",
|
||||||
|
"v0_3_13_pre_004_legacy_v0_v1_block_materials_are_direct_common_raw_with_exact_version_and_index",
|
||||||
|
"v0_3_13_pre_006_http_block_polling_qualifies_legacy_v0_v1_and_rejects_ambiguous_or_future_versions",
|
||||||
|
"RawSolanaMessageVersion::Legacy",
|
||||||
|
"RawSolanaMessageVersion::V0",
|
||||||
|
"RawSolanaMessageVersion::V1",
|
||||||
|
] {
|
||||||
|
assert!(worker.contains(required), "pre.012 Worker Legacy/V0/V1 proof missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(worker_resources.contains("SolanaGetTransactionConfig::new"));
|
||||||
|
assert!(worker_resources.contains("std::option::Option::Some(1),"));
|
||||||
|
for required in [
|
||||||
|
"pre_006_legacy_v0_and_v1_wire_goldens_are_exact",
|
||||||
|
"pre_006_v1_structural_guards_reject_alt_duplicates_and_missing_config",
|
||||||
|
"RawSolanaMessageVersion::V1",
|
||||||
|
] {
|
||||||
|
assert!(common_raw_wire.contains(required), "pre.012 Common RAW V1 proof missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(common_raw_canonical.contains("pre_002_canonical_raw_v1_golden_bytes_and_hash_are_exact"));
|
||||||
|
for required in ["block_time", "content_hash", "format_id", "format_version", "slot"] {
|
||||||
|
assert!(worker_persistence.contains(required), "pre.012 canonical Store identity field missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(store_api.contains("public_pre_003_raw_transaction_and_observation_are_constructible_from_crate_root"));
|
||||||
|
assert!(store_api.contains("RawPayload::try_new(format, 1"));
|
||||||
|
assert!(store.contains("persist_raw_transaction_acquisition"));
|
||||||
|
assert!(store.contains("record_raw_transaction_observation"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_yellowstone_non_regression_covers_identity_v1_reconnect_gap_and_worker_hydration() {
|
||||||
|
let transport_subscribe = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_subscribe.rs");
|
||||||
|
let transport_stream = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_stream.rs");
|
||||||
|
let transport_public = include_str!("../../ksp-onchain-transport-lib/tests/public_api.rs");
|
||||||
|
let worker = include_str!("../unit_tests/runtime_resources.rs");
|
||||||
|
let worker_hardening = include_str!("hardening.rs");
|
||||||
|
for required in [
|
||||||
|
"yellowstone_subscribe_request_identity_is_order_stable_exact_and_debug_redacted",
|
||||||
|
"yellowstone_transaction_update_decodes_current_storage_wire_including_v1_config_and_meta",
|
||||||
|
"yellowstone_transaction_status_update_preserves_error_and_rejects_malformed_signature",
|
||||||
|
"yellowstone_block_update_reuses_transaction_account_entry_dtos_and_preserves_server_counts",
|
||||||
|
] {
|
||||||
|
assert!(transport_subscribe.contains(required), "pre.012 Yellowstone Transport subscribe proof missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"yellowstone_reconnect_replays_from_last_observed_slot_and_counts_duplicate_identity",
|
||||||
|
"yellowstone_replay_info_proves_and_clamps_retention_gap_without_lossless_claim",
|
||||||
|
"yellowstone_shutdown_interrupts_reconnect_backoff_and_mutation_is_rejected_during_reconnect",
|
||||||
|
] {
|
||||||
|
assert!(transport_stream.contains(required), "pre.012 Yellowstone Transport lifecycle proof missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(transport_public.contains("public_v0_3_13_pre_002_yellowstone_subscribe_identity_is_opaque_and_available_from_crate_root"));
|
||||||
|
for required in [
|
||||||
|
"pre_002_source_accepts_matching_confirmed_transaction_and_get_transaction_route_without_io",
|
||||||
|
"v0_3_13_pre_002_live_source_key_is_stable_and_request_sensitive",
|
||||||
|
"pre_003_transaction_signal_fixture_preserves_exact_source_neutral_identity",
|
||||||
|
"pre_005_block_projects_transaction_signals_in_exact_source_order",
|
||||||
|
"pre_005_block_meta_and_slot_project_continuity_only_without_remote_dead_error",
|
||||||
|
"pre_004_observed_get_transaction_closes_signal_to_common_raw_ingress_with_composite_provenance",
|
||||||
|
"pre_008_reconnect_replay_and_proven_gap_are_distinct_monotone_and_frontier_preserving",
|
||||||
|
"v0_3_13_pre_008_global_hydration_registry_coalesces_cross_source_key_to_one_leader",
|
||||||
|
] {
|
||||||
|
assert!(worker.contains(required), "pre.012 Yellowstone Worker non-regression proof missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"v0_3_12_pre_003_private_signal_debug_and_shape_do_not_expose_signature_filters_or_payload",
|
||||||
|
"v0_3_12_pre_008_reconnect_projection_is_source_neutral_bounded_and_contains_no_replay_material",
|
||||||
|
] {
|
||||||
|
assert!(worker_hardening.contains(required), "pre.012 Yellowstone Worker hardening proof missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_security_redaction_matrix_covers_all_live_sources_and_lower_layers() {
|
||||||
|
let transport_subscribe = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_subscribe.rs");
|
||||||
|
let transport_unary = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_unary.rs");
|
||||||
|
let transport_ws = include_str!("../../ksp-onchain-transport-lib/unit_tests/ws_helius_transactions.rs");
|
||||||
|
let worker = include_str!("hardening.rs");
|
||||||
|
let common_raw = include_str!("../../ksp-raw-transaction-lib/unit_tests/canonical.rs");
|
||||||
|
let store_api = include_str!("../../ksp-store-api/tests/security_hardening.rs");
|
||||||
|
let store = include_str!("../../ksp-store-lib/tests/hardening_completeness.rs");
|
||||||
|
let postgres = include_str!("../../ksp-store-postgres-lib/tests/hardening_completeness.rs");
|
||||||
|
for required in [
|
||||||
|
"yellowstone_subscribe_debug_omits_filter_names_and_future_payloads",
|
||||||
|
"yellowstone_transaction_filters_encode_complete_current_wire_and_redact_selectors",
|
||||||
|
] {
|
||||||
|
assert!(transport_subscribe.contains(required), "pre.012 Transport gRPC redaction proof missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(transport_unary.contains("yellowstone_remote_status_does_not_copy_message_details_or_metadata"));
|
||||||
|
assert!(transport_ws.contains("helius_transaction_notification_debug_omits_raw_provider_payloads"));
|
||||||
|
for required in [
|
||||||
|
"v0_3_12_pre_003_private_signal_debug_and_shape_do_not_expose_signature_filters_or_payload",
|
||||||
|
"v0_3_13_pre_003_standard_logs_redaction_and_reference_only_contract_are_explicit",
|
||||||
|
"v0_3_13_pre_004_standard_block_redaction_version_and_null_guards_are_explicit",
|
||||||
|
"v0_3_13_pre_005_helius_transaction_redaction_full_reference_and_tier_neutrality_are_explicit",
|
||||||
|
"v0_3_13_pre_006_http_block_polling_is_bounded_run_local_stop_preemptible_and_redacted",
|
||||||
|
"v0_3_13_pre_007_multi_source_supervisor_is_fail_closed_joined_and_does_not_publish_source_identity",
|
||||||
|
"v0_3_13_pre_008_cross_source_convergence_is_bounded_conflict_checked_and_private",
|
||||||
|
"v0_3_13_pre_009_duplicate_storm_disagreement_and_starvation_guards_are_explicit",
|
||||||
|
"v0_3_13_pre_010_multi_source_health_is_conservative_counted_and_redacted",
|
||||||
|
"v0_3_13_pre_011_shutdown_races_are_bounded_joined_atomic_and_counter_safe",
|
||||||
|
] {
|
||||||
|
assert!(worker.contains(required), "pre.012 Worker security proof missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(common_raw.contains("pre_002_material_and_wire_debug_do_not_render_transaction_or_meta_material"));
|
||||||
|
assert!(store_api.contains("pre_007_raw_debug_surfaces_do_not_render_payload_hash_signature_or_account_bytes"));
|
||||||
|
assert!(store.contains("pre_009_secret_canary_never_crosses_settings_or_pre_io_error_debug"));
|
||||||
|
assert!(postgres.contains("pre_009_backend_error_bridge_cannot_retain_external_error_or_secret_text"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,734 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
// version: 29
|
||||||
|
|
||||||
|
//! Dependency firewall canaries for the RAW transaction ingest Worker foundation.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_manifest_dependency_surface_is_exact() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let dependencies = dependency_section(manifest);
|
||||||
|
let names = manifest_dependency_names(dependencies);
|
||||||
|
assert_eq!(
|
||||||
|
names,
|
||||||
|
vec![
|
||||||
|
"ksp-core-lib",
|
||||||
|
"ksp-logging-lib",
|
||||||
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-raw-transaction-lib",
|
||||||
|
"ksp-store-lib",
|
||||||
|
"ksp-worker-api",
|
||||||
|
"sha2",
|
||||||
|
"tokio",
|
||||||
|
],
|
||||||
|
);
|
||||||
|
for required in [
|
||||||
|
"ksp-core-lib = { path = \"../ksp-core-lib\" }",
|
||||||
|
"ksp-logging-lib = { path = \"../ksp-logging-lib\" }",
|
||||||
|
"ksp-onchain-transport-lib = { path = \"../ksp-onchain-transport-lib\" }",
|
||||||
|
"ksp-raw-transaction-lib = { path = \"../ksp-raw-transaction-lib\" }",
|
||||||
|
"ksp-store-lib = { path = \"../ksp-store-lib\", default-features = false }",
|
||||||
|
"ksp-worker-api = { path = \"../ksp-worker-api\" }",
|
||||||
|
"sha2 = { workspace = true }",
|
||||||
|
"tokio = { workspace = true, features = [\"macros\", \"rt\", \"sync\", \"time\"] }",
|
||||||
|
] {
|
||||||
|
assert!(dependencies.contains(required), "required Worker dependency missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_002_manifest_opens_only_the_onchain_transport_live_source_edge() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let dependencies = dependency_section(manifest);
|
||||||
|
for forbidden in [
|
||||||
|
"futures-util",
|
||||||
|
"ksp-config-lib",
|
||||||
|
"ksp-interface-lib",
|
||||||
|
"ksp-job-api",
|
||||||
|
"ksp-job-backfill-lib",
|
||||||
|
"ksp-program-api",
|
||||||
|
"ksp-store-api",
|
||||||
|
"ksp-store-postgres-lib",
|
||||||
|
"ksp-wallet-lib",
|
||||||
|
"reqwest",
|
||||||
|
"solana-",
|
||||||
|
"tokio-tungstenite",
|
||||||
|
"tonic",
|
||||||
|
"yellowstone-grpc-proto",
|
||||||
|
] {
|
||||||
|
assert!(!dependencies.contains(forbidden), "forbidden Worker dependency present: {forbidden}");
|
||||||
|
}
|
||||||
|
assert!(!manifest.contains("[dev-dependencies]"), "foundation requires no test-only dependency");
|
||||||
|
assert!(!manifest.contains("[build-dependencies]"), "foundation requires no build dependency");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_003_transaction_and_status_adapters_are_private_and_transport_facade_only() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestYellowstoneSignalView for ksp_onchain_transport_lib::YellowstoneTransactionUpdate",
|
||||||
|
"RawTransactionIngestYellowstoneSignalView for ksp_onchain_transport_lib::YellowstoneTransactionStatusUpdate",
|
||||||
|
"YellowstoneTransactionUpdate::transaction(self).signature()",
|
||||||
|
"YellowstoneTransactionUpdate::transaction(self).index()",
|
||||||
|
"YellowstoneTransactionStatusUpdate::signature(self)",
|
||||||
|
"YellowstoneTransactionStatusUpdate::index(self)",
|
||||||
|
"RawTransactionSignature::new",
|
||||||
|
"matched_filter_fingerprint",
|
||||||
|
"yellowstone_provider_unrepresentable",
|
||||||
|
"yellowstone_endpoint_unrepresentable",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.003 signal-adapter contract missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"pub struct RawTransactionIngestSourceSignal",
|
||||||
|
"pub(crate) struct RawTransactionIngestSourceSignal",
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestSourceSignal",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
] {
|
||||||
|
assert!(!resources.contains(forbidden) && !root.contains(forbidden), "pre.003 crossed a private/offline boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
let status_impl =
|
||||||
|
match resources.split_once("impl RawTransactionIngestYellowstoneSignalView for ksp_onchain_transport_lib::YellowstoneTransactionStatusUpdate") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl std::convert::From<") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => tail,
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert!(!status_impl.contains("::error(self)"), "TransactionStatus remote error material must not enter the private signal");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_004_hydration_contract_uses_only_transport_facade_common_raw_and_existing_ingress() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"get_transaction_observed",
|
||||||
|
"SolanaGetTransactionConfig::new",
|
||||||
|
"SolanaTransactionEncoding::Base64",
|
||||||
|
"format_raw_transaction_signature",
|
||||||
|
"extract_raw_transaction_signature_from_binary_base64",
|
||||||
|
"RawTransactionMaterial::binary_base64",
|
||||||
|
"RawTransactionIngress",
|
||||||
|
"yellowstone_http",
|
||||||
|
"transaction_get_transaction",
|
||||||
|
"status_get_transaction",
|
||||||
|
"RawAcquisitionOrigin::Live",
|
||||||
|
"composite_provenance_codes",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.004 hydration contract missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(resources.contains("fn finalize_hydration"));
|
||||||
|
assert!(resources.contains("fn fetch_hydration"));
|
||||||
|
for forbidden in ["ksp_config_lib::", "ksp_job_backfill_lib::", "ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::"] {
|
||||||
|
assert!(!resources.contains(forbidden) && !root.contains(forbidden), "pre.004 crossed a forbidden boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_005_block_and_continuity_adapters_remain_private_and_transport_facade_only() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestYellowstoneBlockView",
|
||||||
|
"YellowstoneBlockUpdate",
|
||||||
|
"project_yellowstone_block_signals",
|
||||||
|
"RawTransactionIngestYellowstoneContinuityView",
|
||||||
|
"YellowstoneBlockMetaUpdate",
|
||||||
|
"YellowstoneSlotUpdate",
|
||||||
|
"RawTransactionIngestContinuitySignal",
|
||||||
|
"block_get_transaction",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.005 private adapter contract missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["ksp_config_lib::", "ksp_job_backfill_lib::", "ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::"] {
|
||||||
|
assert!(!resources.contains(forbidden) && !root.contains(forbidden), "pre.005 crossed a forbidden boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_006_productive_source_uses_transport_session_bounded_coalescence_and_existing_admission() {
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in ["children.spawn", ".run_live_sources(source_settings, stop_receiver, admission_sender, processing_frontier_sender)"] {
|
||||||
|
assert!(runtime.contains(required), "required pre.006 supervisor wiring missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"open_standard_subscribe",
|
||||||
|
"next_update",
|
||||||
|
"RawTransactionIngestHydrationCoordinator",
|
||||||
|
"std::collections::BTreeMap",
|
||||||
|
"tokio::task::JoinSet",
|
||||||
|
"RawTransactionIngestHydrationCoordinator::with_global_registry(",
|
||||||
|
"hydration_pending_limit,",
|
||||||
|
"hydration_in_flight_limit,",
|
||||||
|
"get_transaction_observed",
|
||||||
|
"admission_sender.send(ingress)",
|
||||||
|
"tasks.abort_all()",
|
||||||
|
"session.close().await",
|
||||||
|
"commitment: hydration.commitment.as_str()",
|
||||||
|
"network: signal.network.clone()",
|
||||||
|
"signature: signal.signature",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.006 productive-source contract missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"unbounded_channel",
|
||||||
|
] {
|
||||||
|
assert!(!runtime.contains(forbidden) && !resources.contains(forbidden), "pre.006 crossed a forbidden productive-source boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_006_http_block_polling_reuses_transport_facades_without_becoming_backfill() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestHttpBlockPollingSource",
|
||||||
|
"get_slot(&self.polling_role",
|
||||||
|
"get_blocks_with_limit(",
|
||||||
|
"get_block_observed(&self.polling_role",
|
||||||
|
"let mut next_scan_slot = start_slot",
|
||||||
|
"next_scan_slot = slot",
|
||||||
|
"tokio::time::sleep(self.poll_interval)",
|
||||||
|
"RawAcquisitionOrigin::Live",
|
||||||
|
"block_polling_get_block",
|
||||||
|
"SolanaTransactionEncoding::Base64",
|
||||||
|
"SolanaTransactionDetails::Full",
|
||||||
|
"std::option::Option::Some(1)",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.006 HTTP polling contract missing: {required}");
|
||||||
|
}
|
||||||
|
let source_impl = match resources.split_once("impl crate::RawTransactionIngestHttpBlockPollingSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl std::fmt::Debug for crate::RawTransactionIngestHttpBlockPollingSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert_eq!(source_impl.matches("get_block_observed(").count(), 1);
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"unbounded_channel",
|
||||||
|
] {
|
||||||
|
assert!(!source_impl.contains(forbidden) && !root.contains(forbidden), "pre.006 HTTP polling crossed a forbidden boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_003_standard_logs_source_reuses_transport_facades_and_common_hydration_only() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestStandardLogsSource",
|
||||||
|
"SolanaStandardWsSession::connect",
|
||||||
|
".logs_subscribe(",
|
||||||
|
"SolanaLogsSubscribeFilter",
|
||||||
|
"SolanaCommitmentConfig::new",
|
||||||
|
"project_standard_logs_signal",
|
||||||
|
"RawTransactionIngestSourceFamily::Logs",
|
||||||
|
"logs_get_transaction",
|
||||||
|
"solana_ws_http",
|
||||||
|
"get_transaction_observed",
|
||||||
|
"RawTransactionIngestHydrationCoordinator::with_global_registry(",
|
||||||
|
"hydration_pending_limit,",
|
||||||
|
"hydration_in_flight_limit,",
|
||||||
|
"std::option::Option::Some(1)",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.003 standard logs contract missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(runtime.contains("run_live_sources"));
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"unbounded_channel",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!runtime.contains(forbidden) && !resources.contains(forbidden) && !root.contains(forbidden),
|
||||||
|
"pre.003 crossed a forbidden boundary: {forbidden}",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_004_standard_block_source_is_direct_raw_and_transport_facade_only() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestStandardBlockSource",
|
||||||
|
"SolanaStandardWsSession::connect",
|
||||||
|
".block_subscribe(",
|
||||||
|
"SolanaBlockSubscribeConfig::new",
|
||||||
|
"SolanaTransactionEncoding::Base64",
|
||||||
|
"SolanaTransactionDetails::Full",
|
||||||
|
"std::option::Option::Some(1)",
|
||||||
|
"std::option::Option::Some(false)",
|
||||||
|
"build_standard_block_material",
|
||||||
|
"RawTransactionMaterial::binary_base64_with_embedded_signature",
|
||||||
|
"RawTransactionVersion::Number(1)",
|
||||||
|
"block_subscribe",
|
||||||
|
"source.standard_block_missing",
|
||||||
|
"source.standard_block_remote_error",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.004 standard block contract missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"unbounded_channel",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!runtime_contains(forbidden) && !resources.contains(forbidden) && !root.contains(forbidden),
|
||||||
|
"pre.004 crossed a forbidden standard-block boundary: {forbidden}",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_005_helius_transaction_source_reuses_transport_facade_and_common_hydration_only() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestHeliusTransactionSource",
|
||||||
|
"HeliusLaserStreamWsSession::connect",
|
||||||
|
".transaction_subscribe(",
|
||||||
|
"HeliusTransactionSubscribeOptions::new",
|
||||||
|
"HeliusTransactionSubscribeEncoding::Base64",
|
||||||
|
"SolanaTransactionDetails::Full",
|
||||||
|
"std::option::Option::Some(1)",
|
||||||
|
"project_helius_transaction_signal",
|
||||||
|
"helius_ws_http",
|
||||||
|
"get_transaction_observed",
|
||||||
|
"RawTransactionIngestHydrationCoordinator::with_global_registry(",
|
||||||
|
"hydration_pending_limit,",
|
||||||
|
"hydration_in_flight_limit,",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.005 Helius transaction contract missing: {required}");
|
||||||
|
}
|
||||||
|
assert_eq!(resources.matches("get_transaction_observed(").count(), 1);
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"helius_sdk",
|
||||||
|
"unbounded_channel",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!runtime.contains(forbidden) && !resources.contains(forbidden) && !root.contains(forbidden),
|
||||||
|
"pre.005 crossed a forbidden Helius transaction boundary: {forbidden}",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn runtime_contains(value: &str) -> bool {
|
||||||
|
return include_str!("../src/runtime.rs").contains(value);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_006_source_surface_hardens_shutdown_and_faults_without_backend_edges() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let admission = include_str!("../src/admission.rs");
|
||||||
|
let persistence = include_str!("../src/persistence.rs");
|
||||||
|
let snapshot = include_str!("../src/snapshot.rs");
|
||||||
|
let runtime_resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"tokio::sync::mpsc::channel",
|
||||||
|
"canonicalize_raw_transaction",
|
||||||
|
"assemble_raw_transaction_acquisition",
|
||||||
|
"RawTransactionIngestPersistencePort",
|
||||||
|
"RawTransactionAcquisitionMode::Normal",
|
||||||
|
"RawTransactionIngestSnapshot",
|
||||||
|
"RawTransactionIngestSnapshotSource",
|
||||||
|
"WorkerSnapshotSource",
|
||||||
|
"tokio::sync::watch::channel",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED",
|
||||||
|
"record_source_failure",
|
||||||
|
"tokio::time::timeout",
|
||||||
|
"abort_all",
|
||||||
|
"backpressure_wait_total",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
root.contains(required)
|
||||||
|
|| runtime.contains(required)
|
||||||
|
|| admission.contains(required)
|
||||||
|
|| persistence.contains(required)
|
||||||
|
|| snapshot.contains(required)
|
||||||
|
|| runtime_resources.contains(required),
|
||||||
|
"required pre.009 hardening contract missing: {required}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"pub use self::admission::RawTransactionAdmission",
|
||||||
|
"pub use self::admission::RawTransactionIngress",
|
||||||
|
"pub use self::persistence::RawTransactionIngestPersistencePort",
|
||||||
|
"unbounded_channel",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ForceRehydrate",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!root.contains(forbidden)
|
||||||
|
&& !runtime.contains(forbidden)
|
||||||
|
&& !admission.contains(forbidden)
|
||||||
|
&& !persistence.contains(forbidden)
|
||||||
|
&& !snapshot.contains(forbidden)
|
||||||
|
&& !runtime_resources.contains(forbidden),
|
||||||
|
"pre.009 crossed a forbidden runtime boundary: {forbidden}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
runtime.matches("tokio::sync::watch::channel(false)").count(),
|
||||||
|
2,
|
||||||
|
"runtime retains exactly the external-control and private-source stop watches"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
runtime.matches("tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty())").count(),
|
||||||
|
1,
|
||||||
|
"runtime retains exactly one run-local processing-frontier latest-value watch",
|
||||||
|
);
|
||||||
|
assert_eq!(runtime.matches("tokio::sync::watch::channel").count(), 3, "runtime watch inventory is two stop watches plus one processing-frontier watch");
|
||||||
|
assert_eq!(snapshot.matches("tokio::sync::watch::channel").count(), 1, "exactly one shared concrete snapshot watch is allowed");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn dependency_section(manifest: &str) -> &str {
|
||||||
|
let after_header = match manifest.split_once("[dependencies]") {
|
||||||
|
std::option::Option::Some((_, section)) => section,
|
||||||
|
std::option::Option::None => {
|
||||||
|
return "";
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let end = match after_header.find("\n[") {
|
||||||
|
std::option::Option::Some(index) => index,
|
||||||
|
std::option::Option::None => after_header.len(),
|
||||||
|
};
|
||||||
|
return &after_header[..end];
|
||||||
|
}
|
||||||
|
|
||||||
|
fn manifest_dependency_names(section: &str) -> std::vec::Vec<&str> {
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for line in section.lines() {
|
||||||
|
let content = match line.split('#').next() {
|
||||||
|
std::option::Option::Some(value) => value.trim(),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
if content.is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match content.split('=').next() {
|
||||||
|
std::option::Option::Some(value) => value.trim().trim_end_matches(".workspace"),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
if !name.is_empty() {
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
names.sort_unstable();
|
||||||
|
return names;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_007_multi_source_supervisor_and_inventory_remain_private_bounded_and_source_neutral() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestSourceInventory",
|
||||||
|
"RawTransactionIngestSourceInventoryPublisher",
|
||||||
|
"for (entry_index, source) in self.sources.into_iter().enumerate()",
|
||||||
|
"tokio::task::JoinSet::new()",
|
||||||
|
"supervise_live_source_tasks",
|
||||||
|
"drain_live_source_tasks",
|
||||||
|
"source_stop_sender.send_replace(true)",
|
||||||
|
"processing_frontier_sender.clone()",
|
||||||
|
"source.configured_source_closed",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.007 multi-source supervisor contract missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(runtime.contains("run_live_sources"));
|
||||||
|
assert!(!runtime.contains("validate_single_source_activation"));
|
||||||
|
assert!(!resources.contains("multi_source_activation_pending"));
|
||||||
|
for forbidden in [
|
||||||
|
"pub struct RawTransactionIngestSourceInventory",
|
||||||
|
"pub struct RawTransactionIngestSourceInventoryPublisher",
|
||||||
|
"unbounded_channel",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!root.contains(forbidden) && !runtime.contains(forbidden) && !resources.contains(forbidden),
|
||||||
|
"pre.007 crossed a private/bounded boundary: {forbidden}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_007_processing_frontier_remains_run_local_and_backend_neutral_after_continuity_extension() {
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let snapshot = include_str!("../src/snapshot.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestProcessingFrontierProjection",
|
||||||
|
"processing_frontier_sender",
|
||||||
|
"wait_processing_frontier",
|
||||||
|
"record_processing_frontier",
|
||||||
|
"hydration_pending",
|
||||||
|
"processing_frontier_slot",
|
||||||
|
"oldest_pending_slot",
|
||||||
|
"RawTransactionIngestProcessingFrontier",
|
||||||
|
"settle_pending",
|
||||||
|
"observe_settled",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
runtime.contains(required) || resources.contains(required) || snapshot.contains(required),
|
||||||
|
"required pre.007 frontier contract missing: {required}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let frontier = match resources.split_once("struct RawTransactionIngestProcessingFrontier {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("struct RawTransactionIngestProcessingFrontierReporter") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["SubscribeReplayInfo", "from_slot", "repair", "ksp_job_backfill_lib::", "ksp_store_postgres_lib::"] {
|
||||||
|
assert!(!frontier.contains(forbidden), "pre.007 processing frontier absorbed replay/backend responsibility: {forbidden}");
|
||||||
|
}
|
||||||
|
for forbidden in ["ksp_job_backfill_lib::", "ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::"] {
|
||||||
|
assert!(
|
||||||
|
!runtime.contains(forbidden) && !resources.contains(forbidden) && !snapshot.contains(forbidden),
|
||||||
|
"Worker crossed backend boundary: {forbidden}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_008_worker_observes_transport_reconnect_replay_and_faults_only_on_proven_retention_gap() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let transport = include_str!("../../ksp-onchain-transport-lib/src/grpc_stream.rs");
|
||||||
|
for required in [
|
||||||
|
"snapshot_source()",
|
||||||
|
"wait_yellowstone_session_snapshot",
|
||||||
|
"source_reconnect_total",
|
||||||
|
"source_replay_attempt_total",
|
||||||
|
"source_continuity_gap_total",
|
||||||
|
"source.continuity_gap_proven",
|
||||||
|
"RawTransactionIngestSourceState::Reconnecting",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.008 Worker continuity token missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["subscribe_replay_info(", "set_from_slot(", "YellowstoneReplayInfo", "last_requested_from_slot()", "ksp_job_backfill_lib::"] {
|
||||||
|
assert!(!resources.contains(forbidden), "Worker took ownership of replay/repair responsibility: {forbidden}");
|
||||||
|
}
|
||||||
|
for required in ["SubscribeReplayInfo", "set_from_slot(effective_from_slot)", "first_available > requested", "replay_attempt_count"] {
|
||||||
|
assert!(transport.contains(required), "Transport replay ownership token missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_008_cross_source_convergence_reuses_store_and_transport_facades_only() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let persistence = include_str!("../src/persistence.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestPersistenceConvergence",
|
||||||
|
"persist_raw_transaction_ingest_converged_acquisition",
|
||||||
|
"RawTransactionIngestPersistencePort",
|
||||||
|
"record_raw_transaction_observation",
|
||||||
|
"RawTransactionObservationWrite",
|
||||||
|
"payload().content_hash()",
|
||||||
|
"content_conflict_error()",
|
||||||
|
] {
|
||||||
|
assert!(persistence.contains(required) || root.contains(required), "required pre.008 persistence convergence contract missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"fetch_hydration_shared",
|
||||||
|
"subscribe_or_lead",
|
||||||
|
"get_transaction_observed",
|
||||||
|
"settings.admission_queue_capacity()",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.008 hydration convergence contract missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(runtime.contains("settings.admission_queue_capacity().max(settings.persistence_concurrency())"));
|
||||||
|
for forbidden in ["ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::", "ksp_job_backfill_lib::", "unbounded_channel"] {
|
||||||
|
assert!(
|
||||||
|
!persistence.contains(forbidden) && !runtime.contains(forbidden) && !resources.contains(forbidden),
|
||||||
|
"pre.008 convergence crossed a dependency/boundedness boundary: {forbidden}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_009_global_bounds_and_fairness_stay_inside_worker_facades() {
|
||||||
|
let admission = include_str!("../src/admission.rs");
|
||||||
|
let persistence = include_str!("../src/persistence.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"validate_hydration_fairness_capacity",
|
||||||
|
"let hydration_pending_budget = settings.admission_queue_capacity();",
|
||||||
|
"let hydration_in_flight_budget = settings.persistence_concurrency();",
|
||||||
|
"hydration_pending_limit",
|
||||||
|
"hydration_in_flight_limit",
|
||||||
|
"tokio::sync::Semaphore",
|
||||||
|
"acquire_owned",
|
||||||
|
"RawTransactionIngestHydrationLeaderGuard",
|
||||||
|
"RawTransactionIngestCanonicalState",
|
||||||
|
"content_conflict_error()",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
admission.contains(required) || persistence.contains(required) || resources.contains(required),
|
||||||
|
"required pre.009 global-bound contract missing: {required}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"unbounded_channel",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!admission.contains(forbidden) && !persistence.contains(forbidden) && !resources.contains(forbidden),
|
||||||
|
"pre.009 crossed a Worker facade boundary: {forbidden}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_010_multi_source_snapshot_health_remains_source_neutral_and_backend_free() {
|
||||||
|
let snapshot = include_str!("../src/snapshot.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in ["source_total", "source_active", "source_reconnecting", "source_failed", "WorkerHealth::Degraded", "with_source_counts"] {
|
||||||
|
assert!(snapshot.contains(required) || resources.contains(required), "required pre.010 source-neutral snapshot contract missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::", "SourceSnapshotByKey", "provider_health"] {
|
||||||
|
assert!(!snapshot.contains(forbidden), "pre.010 snapshot crossed source-neutral/backend boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_011_shutdown_race_hardening_stays_inside_worker_and_existing_facades() {
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"tokio::time::timeout(settings.shutdown_drain_timeout(), drain).await",
|
||||||
|
"persistence.abort_all()",
|
||||||
|
"children.abort_all()",
|
||||||
|
"while persistence.join_next().await.is_some() {}",
|
||||||
|
"while children.join_next().await.is_some() {}",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED",
|
||||||
|
] {
|
||||||
|
assert!(runtime.contains(required), "required pre.011 runtime shutdown guard missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestHydrationLeaderGuard",
|
||||||
|
"publish_and_remove",
|
||||||
|
"pending.remove(key)",
|
||||||
|
"source.inventory_key_mismatch",
|
||||||
|
"checked_add(projection.hydration_pending())",
|
||||||
|
"checked_add(projection.source_reconnect_total())",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.011 source-race guard missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"unbounded_channel",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
] {
|
||||||
|
assert!(!runtime.contains(forbidden) && !resources.contains(forbidden), "pre.011 crossed a Worker/facade boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_cross_layer_closure_does_not_expand_worker_dependency_graph() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let dependencies = dependency_section(manifest);
|
||||||
|
let names = manifest_dependency_names(dependencies);
|
||||||
|
assert_eq!(
|
||||||
|
names,
|
||||||
|
vec![
|
||||||
|
"ksp-core-lib",
|
||||||
|
"ksp-logging-lib",
|
||||||
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-raw-transaction-lib",
|
||||||
|
"ksp-store-lib",
|
||||||
|
"ksp-worker-api",
|
||||||
|
"sha2",
|
||||||
|
"tokio",
|
||||||
|
],
|
||||||
|
);
|
||||||
|
let cross_layer = include_str!("cross_layer_completeness.rs");
|
||||||
|
for required in [
|
||||||
|
"v0_3_13_pre_012_all_five_live_source_families_share_one_worker_common_raw_store_pipeline",
|
||||||
|
"v0_3_13_pre_012_legacy_v0_v1_is_proven_from_transport_through_worker_common_raw_to_store",
|
||||||
|
"v0_3_13_pre_012_yellowstone_non_regression_covers_identity_v1_reconnect_gap_and_worker_hydration",
|
||||||
|
"v0_3_13_pre_012_security_redaction_matrix_covers_all_live_sources_and_lower_layers",
|
||||||
|
] {
|
||||||
|
assert!(cross_layer.contains(required), "pre.012 cross-layer closure proof missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
959
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
Normal file
959
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
Normal file
@@ -0,0 +1,959 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
// version: 24
|
||||||
|
|
||||||
|
//! External public, security, redaction and release-boundary hardening canaries through `pre.012`.
|
||||||
|
|
||||||
|
fn network(value: &'static str) -> std::option::Option<ksp_store_lib::RawNetworkId> {
|
||||||
|
let result = ksp_store_lib::RawNetworkId::new(value);
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
assert_eq!(error.code(), ksp_store_lib::ERROR_CODE_RAW_MODEL_INVALID);
|
||||||
|
std::option::Option::None
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn worker_id(value: &'static str) -> std::option::Option<ksp_worker_api::WorkerId> {
|
||||||
|
let result = ksp_worker_api::WorkerId::new(value);
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
assert_eq!(error.code(), ksp_worker_api::ERROR_CODE_WORKER_ID_INVALID);
|
||||||
|
std::option::Option::None
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_external_error_codes_are_stable_unique_and_domain_scoped() {
|
||||||
|
let codes = [
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT, "content_conflict"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED, "counter_exhausted"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT, "drain_timeout"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID, "runtime_invalid"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID, "settings_invalid"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED, "source_failed"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED, "store_failed"),
|
||||||
|
];
|
||||||
|
for (index, (code, expected)) in codes.iter().enumerate() {
|
||||||
|
assert_eq!(code.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(code.code(), *expected);
|
||||||
|
for (other_index, (other, _)) in codes.iter().enumerate() {
|
||||||
|
if index != other_index {
|
||||||
|
assert_ne!(code, other);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_debug_and_settings_errors_redact_worker_identity_and_invalid_values() {
|
||||||
|
let raw_worker_id = "raw-ingest-secret-marker-001";
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let parsed_worker_id = match worker_id(raw_worker_id) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert_eq!(std::format!("{parsed_worker_id:?}"), "WorkerId(..)");
|
||||||
|
let settings = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::new(
|
||||||
|
network.clone(),
|
||||||
|
parsed_worker_id,
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
std::time::Duration::from_millis(100),
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let settings_debug = std::format!("{settings:?}");
|
||||||
|
assert!(!settings_debug.contains(raw_worker_id));
|
||||||
|
assert!(settings_debug.contains("WorkerId(..)"));
|
||||||
|
let invalid_worker_id = match worker_id("raw-ingest-secret-marker-002") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let invalid =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::new(network, invalid_worker_id, 65_537, 1, std::time::Duration::from_millis(100));
|
||||||
|
let error = match invalid {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(error) => error,
|
||||||
|
};
|
||||||
|
let error_debug = std::format!("{error:?}");
|
||||||
|
assert_eq!(error.code(), ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID);
|
||||||
|
assert!(!error_debug.contains("raw-ingest-secret-marker-002"));
|
||||||
|
assert!(!error_debug.contains("65537"));
|
||||||
|
assert!(error.context().iter().any(|context| return context.key() == "field" && context.value() == "admission_queue_capacity"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_002_manifest_dependency_surface_opens_only_transport_and_remains_backend_neutral() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let mut section = "";
|
||||||
|
let mut normal = std::collections::BTreeSet::new();
|
||||||
|
let mut dev = std::collections::BTreeSet::new();
|
||||||
|
let mut build = std::collections::BTreeSet::new();
|
||||||
|
for line in manifest.lines() {
|
||||||
|
let trimmed = line.trim();
|
||||||
|
if trimmed.starts_with('[') && trimmed.ends_with(']') {
|
||||||
|
section = trimmed;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if trimmed.is_empty() || trimmed.starts_with('#') {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match trimmed.split_once('=') {
|
||||||
|
std::option::Option::Some((name, _)) => name.trim(),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
if section == "[dependencies]" {
|
||||||
|
normal.insert(name);
|
||||||
|
} else if section == "[dev-dependencies]" {
|
||||||
|
dev.insert(name);
|
||||||
|
} else if section == "[build-dependencies]" {
|
||||||
|
build.insert(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
normal,
|
||||||
|
std::collections::BTreeSet::from([
|
||||||
|
"ksp-core-lib",
|
||||||
|
"ksp-logging-lib",
|
||||||
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-raw-transaction-lib",
|
||||||
|
"ksp-store-lib",
|
||||||
|
"ksp-worker-api",
|
||||||
|
"sha2",
|
||||||
|
"tokio",
|
||||||
|
])
|
||||||
|
);
|
||||||
|
assert!(dev.is_empty());
|
||||||
|
assert!(build.is_empty());
|
||||||
|
assert!(manifest.contains("ksp-store-lib = { path = \"../ksp-store-lib\", default-features = false }"));
|
||||||
|
assert!(manifest.contains("tokio = { workspace = true, features = [\"macros\", \"rt\", \"sync\", \"time\"] }"));
|
||||||
|
for forbidden in [
|
||||||
|
"ksp-config-lib",
|
||||||
|
"ksp-job-api",
|
||||||
|
"ksp-job-backfill-lib",
|
||||||
|
"ksp-store-api",
|
||||||
|
"ksp-store-postgres-lib",
|
||||||
|
"reqwest",
|
||||||
|
"tokio-tungstenite",
|
||||||
|
"tonic",
|
||||||
|
"yellowstone-grpc-proto",
|
||||||
|
] {
|
||||||
|
assert!(!manifest.contains(forbidden), "forbidden Worker manifest dependency present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_source_visibility_contract_uses_crate_root_for_shared_items() {
|
||||||
|
let source_contracts: [(&str, &[&str]); 5] = [
|
||||||
|
(include_str!("../src/settings.rs"), &["RawTransactionIngestSettings"]),
|
||||||
|
(include_str!("../src/runtime.rs"), &["RawTransactionIngestHandle", "RawTransactionIngestWorker"]),
|
||||||
|
(include_str!("../src/snapshot.rs"), &["RawTransactionIngestSnapshot", "RawTransactionIngestSnapshotSource"]),
|
||||||
|
(include_str!("../src/persistence.rs"), &["RawTransactionIngestPersistenceOutcome", "RawTransactionIngestPersistencePort"]),
|
||||||
|
(
|
||||||
|
include_str!("../src/runtime_resources.rs"),
|
||||||
|
&[
|
||||||
|
"RawTransactionIngestStandardBlockSource",
|
||||||
|
"RawTransactionIngestStandardLogsSource",
|
||||||
|
"RawTransactionIngestYellowstoneSource",
|
||||||
|
"RawTransactionIngestRuntimeResources",
|
||||||
|
],
|
||||||
|
),
|
||||||
|
];
|
||||||
|
for (source, symbols) in source_contracts {
|
||||||
|
for symbol in symbols {
|
||||||
|
let required = std::format!("impl crate::{symbol}");
|
||||||
|
assert!(source.contains(required.as_str()), "shared item must use crate-root impl path: {symbol}");
|
||||||
|
let forbidden = std::format!("impl {symbol}");
|
||||||
|
assert!(!source.contains(forbidden.as_str()), "shared item must not use bare impl path: {symbol}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (module, source) in [
|
||||||
|
("admission", include_str!("../src/admission.rs")),
|
||||||
|
("error", include_str!("../src/error.rs")),
|
||||||
|
("identity", include_str!("../src/identity.rs")),
|
||||||
|
("persistence", include_str!("../src/persistence.rs")),
|
||||||
|
("runtime", include_str!("../src/runtime.rs")),
|
||||||
|
("runtime_resources", include_str!("../src/runtime_resources.rs")),
|
||||||
|
("settings", include_str!("../src/settings.rs")),
|
||||||
|
("snapshot", include_str!("../src/snapshot.rs")),
|
||||||
|
] {
|
||||||
|
let forbidden = std::format!("crate::{module}::");
|
||||||
|
assert!(!source.contains(forbidden.as_str()), "internal module path bypasses crate-root façade: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_production_surface_has_no_historical_backfill_or_retriever_contract() {
|
||||||
|
let sources = [
|
||||||
|
include_str!("../src/admission.rs"),
|
||||||
|
include_str!("../src/error.rs"),
|
||||||
|
include_str!("../src/identity.rs"),
|
||||||
|
include_str!("../src/lib.rs"),
|
||||||
|
include_str!("../src/persistence.rs"),
|
||||||
|
include_str!("../src/runtime.rs"),
|
||||||
|
include_str!("../src/runtime_resources.rs"),
|
||||||
|
include_str!("../src/settings.rs"),
|
||||||
|
include_str!("../src/snapshot.rs"),
|
||||||
|
];
|
||||||
|
for source in sources {
|
||||||
|
for forbidden in [
|
||||||
|
"ksp-worker-raw-retriever",
|
||||||
|
"raw_transaction_retriever",
|
||||||
|
"RawTransactionRetriever",
|
||||||
|
"Backfill",
|
||||||
|
"backfill",
|
||||||
|
"Checkpoint",
|
||||||
|
"checkpoint",
|
||||||
|
"historical",
|
||||||
|
] {
|
||||||
|
assert!(!source.contains(forbidden), "historical/retriever surface leaked into Worker production source: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_002_production_sources_keep_transport_confined_to_runtime_resources() {
|
||||||
|
for source in [
|
||||||
|
include_str!("../src/admission.rs"),
|
||||||
|
include_str!("../src/error.rs"),
|
||||||
|
include_str!("../src/identity.rs"),
|
||||||
|
include_str!("../src/lib.rs"),
|
||||||
|
include_str!("../src/persistence.rs"),
|
||||||
|
include_str!("../src/runtime.rs"),
|
||||||
|
include_str!("../src/runtime_resources.rs"),
|
||||||
|
include_str!("../src/settings.rs"),
|
||||||
|
include_str!("../src/snapshot.rs"),
|
||||||
|
] {
|
||||||
|
let lower = source.to_ascii_lowercase();
|
||||||
|
for forbidden in ["api_key", "api-key", "authorization", "bearer ", "password", "credential", "secret"] {
|
||||||
|
assert!(!lower.contains(forbidden), "secret-like material leaked into Worker production source: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let transport_source = include_str!("../src/runtime_resources.rs");
|
||||||
|
assert!(transport_source.contains("ksp_onchain_transport_lib::"));
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ksp_offchain_transport_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"postgresql://",
|
||||||
|
"postgres://",
|
||||||
|
] {
|
||||||
|
assert!(!transport_source.contains(forbidden), "forbidden implementation detail leaked into runtime resources: {forbidden}");
|
||||||
|
}
|
||||||
|
for source in [
|
||||||
|
include_str!("../src/admission.rs"),
|
||||||
|
include_str!("../src/error.rs"),
|
||||||
|
include_str!("../src/identity.rs"),
|
||||||
|
include_str!("../src/lib.rs"),
|
||||||
|
include_str!("../src/persistence.rs"),
|
||||||
|
include_str!("../src/runtime.rs"),
|
||||||
|
include_str!("../src/settings.rs"),
|
||||||
|
include_str!("../src/snapshot.rs"),
|
||||||
|
] {
|
||||||
|
assert!(!source.contains("ksp_onchain_transport_lib::"), "Transport dependency escaped runtime_resources.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ksp_offchain_transport_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"postgresql://",
|
||||||
|
"postgres://",
|
||||||
|
] {
|
||||||
|
assert!(!source.contains(forbidden), "forbidden implementation detail leaked into Worker production source: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_003_private_signal_debug_and_shape_do_not_expose_signature_filters_or_payload() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
assert!(resources.contains("struct RawTransactionIngestSourceSignal"));
|
||||||
|
assert!(!root.contains("RawTransactionIngestSourceSignal"));
|
||||||
|
assert!(!resources.contains(".field(\"matched_filter_fingerprint\", &self.matched_filter_fingerprint)"));
|
||||||
|
assert!(!resources.contains(".field(\"signature\", &self.signature)"));
|
||||||
|
let signal_struct = match resources.split_once("struct RawTransactionIngestSourceSignal {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl std::fmt::Debug for RawTransactionIngestSourceSignal") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["transaction:", "meta:", "error:", "payload:", "body:", "is_vote:"] {
|
||||||
|
assert!(!signal_struct.contains(forbidden), "payload/provider-specific field leaked into private source signal: {forbidden}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"created_at:",
|
||||||
|
"family:",
|
||||||
|
"matched_filter_count:",
|
||||||
|
"matched_filter_fingerprint:",
|
||||||
|
"matched_filter_id:",
|
||||||
|
"network:",
|
||||||
|
"route:",
|
||||||
|
"signature:",
|
||||||
|
"slot:",
|
||||||
|
"transaction_index:",
|
||||||
|
] {
|
||||||
|
assert!(signal_struct.contains(required), "required private signal field missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_004_hydration_provenance_and_remote_material_are_bounded_and_redacted() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"route_prefix",
|
||||||
|
"{}.{}:http.{}",
|
||||||
|
"composite_provider_unrepresentable",
|
||||||
|
"composite_endpoint_unrepresentable",
|
||||||
|
"RawAcquisitionOrigin::Live",
|
||||||
|
"with_capture_session_id",
|
||||||
|
"with_commitment",
|
||||||
|
"with_endpoint_id",
|
||||||
|
"with_filter_id",
|
||||||
|
"try_with_observed_at",
|
||||||
|
"hydration.signature_mismatch",
|
||||||
|
"hydration.slot_mismatch",
|
||||||
|
"hydration.transaction_index_mismatch",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.004 bounded provenance/mismatch guard missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["source_payload_hash", "source_payload_size_bytes", "HTTP-SECRET-CANARY", "GRPC-SECRET-CANARY", "TransactionStatus.error", ".error()"] {
|
||||||
|
assert!(!resources.contains(forbidden), "pre.004 retained forbidden remote/source material: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_005_block_and_continuity_shapes_are_bounded_redacted_and_raw_separated() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"impl RawTransactionIngestYellowstoneBlockView for ksp_onchain_transport_lib::YellowstoneBlockUpdate",
|
||||||
|
"impl RawTransactionIngestYellowstoneContinuityView for ksp_onchain_transport_lib::YellowstoneBlockMetaUpdate",
|
||||||
|
"impl RawTransactionIngestYellowstoneContinuityView for ksp_onchain_transport_lib::YellowstoneSlotUpdate",
|
||||||
|
"RawTransactionIngestSourceFamily::Block",
|
||||||
|
"project_yellowstone_block_signals",
|
||||||
|
"project_yellowstone_continuity_signal",
|
||||||
|
"RawTransactionIngestContinuityStatus::Dead",
|
||||||
|
"block_get_transaction",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.005 bounded adapter guard missing: {required}");
|
||||||
|
}
|
||||||
|
let continuity_struct = match resources.split_once("struct RawTransactionIngestContinuitySignal {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl std::fmt::Debug for RawTransactionIngestContinuitySignal") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for required in [
|
||||||
|
"created_at:",
|
||||||
|
"family:",
|
||||||
|
"matched_filter_count:",
|
||||||
|
"matched_filter_fingerprint:",
|
||||||
|
"matched_filter_id:",
|
||||||
|
"network:",
|
||||||
|
"parent_slot:",
|
||||||
|
"route:",
|
||||||
|
"slot:",
|
||||||
|
"status:",
|
||||||
|
] {
|
||||||
|
assert!(continuity_struct.contains(required), "required continuity-only field missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["signature:", "transaction:", "meta:", "payload:", "body:", "error:", "dead_error:"] {
|
||||||
|
assert!(!continuity_struct.contains(forbidden), "RAW/remote material leaked into continuity-only signal: {forbidden}");
|
||||||
|
}
|
||||||
|
assert!(!resources.contains("YellowstoneSlotUpdate::dead_error"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_006_runtime_resource_contract_opens_one_supervised_transport_source() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
for required in ["open_standard_subscribe", "next_update", "get_transaction_observed", "RawTransactionIngestHydrationCoordinator", "session.close().await"]
|
||||||
|
{
|
||||||
|
assert!(resources.contains(required), "productive runtime-resource source behavior missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(runtime.contains("start_with_runtime_resources"));
|
||||||
|
assert!(runtime.contains("run_live_sources(source_settings, stop_receiver, admission_sender, processing_frontier_sender)"));
|
||||||
|
for forbidden in ["ksp_config_lib::", "ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::"] {
|
||||||
|
assert!(!resources.contains(forbidden) && !runtime.contains(forbidden), "pre.006 runtime source crossed a forbidden boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_006_source_coalescence_is_bounded_stop_preemptible_and_redacted() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"max_in_flight: usize",
|
||||||
|
"max_pending_signals: usize",
|
||||||
|
"RawTransactionIngestHydrationCoordinator::with_global_registry(",
|
||||||
|
"hydration_pending_limit,",
|
||||||
|
"hydration_in_flight_limit,",
|
||||||
|
"pending_signal_count",
|
||||||
|
"stop_receiver.changed()",
|
||||||
|
"tasks.abort_all()",
|
||||||
|
"source.hydration_pending_saturated",
|
||||||
|
"source.hydration_task_join_failed",
|
||||||
|
"hydration_transport_error(error.code())",
|
||||||
|
"source_transport_error(error.code())",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.006 bounded/stop-safe source guard missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["remote_message", "response_body", "dead_error", "TransactionStatus.error", ".error()", "Authorization", "Bearer "] {
|
||||||
|
assert!(!resources.contains(forbidden), "remote/secret material leaked into productive source: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_lower_layers_have_no_dependency_return_to_concrete_worker() {
|
||||||
|
for manifest in [
|
||||||
|
include_str!("../../ksp-core-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-logging-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-raw-transaction-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-store-api/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-store-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-store-postgres-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-worker-api/Cargo.toml"),
|
||||||
|
] {
|
||||||
|
assert!(!manifest.contains("ksp-worker-raw-transaction-ingest-lib"));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_002_public_root_exposes_contract_types_without_transport_implementation_paths() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"pub mod ",
|
||||||
|
"tokio::",
|
||||||
|
"JoinSet",
|
||||||
|
"watch::Receiver",
|
||||||
|
"watch::Sender",
|
||||||
|
"mpsc::Sender",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ksp_onchain_transport_lib::",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tonic::",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "implementation/backend/live-source detail leaked into public root: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_007_processing_frontier_is_bounded_processing_only_and_redacted() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let snapshot = include_str!("../src/snapshot.rs");
|
||||||
|
for required in [
|
||||||
|
"pending_total",
|
||||||
|
"std::collections::BTreeMap<u64, RawTransactionIngestProcessingSlotState>",
|
||||||
|
"settled_interval_highs",
|
||||||
|
"oldest_pending_slot",
|
||||||
|
"processing_frontier_slot",
|
||||||
|
"hydration_pending",
|
||||||
|
"processing_frontier.observe_pending",
|
||||||
|
"processing_frontier.settle_pending",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required) || snapshot.contains(required), "required pre.007 bounded frontier guard missing: {required}");
|
||||||
|
}
|
||||||
|
let frontier = match resources.split_once("struct RawTransactionIngestProcessingFrontier {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("struct RawTransactionIngestProcessingFrontierReporter") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in [
|
||||||
|
"signature: ksp_store_lib::RawTransactionSignature",
|
||||||
|
"filter_id",
|
||||||
|
"provider",
|
||||||
|
"endpoint_id",
|
||||||
|
"transaction:",
|
||||||
|
"meta:",
|
||||||
|
"ReplayInfo",
|
||||||
|
"from_slot",
|
||||||
|
"continuity_gap",
|
||||||
|
"repair",
|
||||||
|
] {
|
||||||
|
assert!(!frontier.contains(forbidden), "pre.007 frontier absorbed forbidden material: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_008_reconnect_projection_is_source_neutral_bounded_and_contains_no_replay_material() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let snapshot = include_str!("../src/snapshot.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestSourceState",
|
||||||
|
"source_reconnect_total",
|
||||||
|
"source_replay_attempt_total",
|
||||||
|
"source_continuity_gap_total",
|
||||||
|
"source.continuity_counter_regression",
|
||||||
|
"source.continuity_gap_proven",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required) || snapshot.contains(required), "required pre.008 bounded continuity guard missing: {required}");
|
||||||
|
}
|
||||||
|
let projection = match snapshot.split_once("struct RawTransactionIngestProcessingFrontierProjection {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl crate::RawTransactionIngestProcessingFrontierProjection") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["signature", "provider", "endpoint", "filter", "transaction", "meta", "from_slot", "first_available", "ReplayInfo"] {
|
||||||
|
assert!(!projection.contains(forbidden), "pre.008 source projection leaked replay/provider/RAW material: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_003_standard_logs_redaction_and_reference_only_contract_are_explicit() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestStandardLogsSource",
|
||||||
|
".field(\"filter_kind\"",
|
||||||
|
".field(\"filter_fingerprint_bytes\"",
|
||||||
|
".field(\"source_key_bytes\"",
|
||||||
|
"project_standard_logs_signal",
|
||||||
|
"matched_filter_count: 1",
|
||||||
|
"transaction_index: std::option::Option::None",
|
||||||
|
"created_at: std::option::Option::None",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.003 redaction/reference guard missing: {required}");
|
||||||
|
}
|
||||||
|
let source_struct = match resources.split_once("pub struct RawTransactionIngestStandardLogsSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl crate::RawTransactionIngestStandardLogsSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert!(source_struct.contains("filter: ksp_onchain_transport_lib::SolanaLogsSubscribeFilter"));
|
||||||
|
for forbidden in ["logs:", "err:", "payload:", "body:", "url:"] {
|
||||||
|
assert!(!source_struct.contains(forbidden), "remote/sensitive material stored in standard logs source: {forbidden}");
|
||||||
|
}
|
||||||
|
let projection = match resources.split_once("fn project_standard_logs_signal") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("fn route_yellowstone_update") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in [".logs()", ".err()"] {
|
||||||
|
assert!(!projection.contains(forbidden), "logs/error material copied into standard logs projection: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_004_standard_block_redaction_version_and_null_guards_are_explicit() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestStandardBlockSource",
|
||||||
|
".field(\"filter_kind\"",
|
||||||
|
".field(\"filter_fingerprint_bytes\"",
|
||||||
|
".field(\"source_key_bytes\"",
|
||||||
|
"source.standard_block_context_slot_mismatch",
|
||||||
|
"source.standard_block_remote_error",
|
||||||
|
"source.standard_block_missing",
|
||||||
|
"source.standard_block_transactions_missing",
|
||||||
|
"source.standard_block_transaction_version_unsupported",
|
||||||
|
"source.standard_block_transaction_version_unqualified",
|
||||||
|
"SolanaTransactionVersion::Number(0)",
|
||||||
|
"SolanaTransactionVersion::Number(1)",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.004 standard block hardening guard missing: {required}");
|
||||||
|
}
|
||||||
|
let source_struct = match resources.split_once("pub struct RawTransactionIngestStandardBlockSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl crate::RawTransactionIngestStandardBlockSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["err:", "block:", "transaction_data:", "url:"] {
|
||||||
|
assert!(!source_struct.contains(forbidden), "remote/raw material stored in standard block source: {forbidden}");
|
||||||
|
}
|
||||||
|
let source_impl = match resources.split_once("impl crate::RawTransactionIngestStandardBlockSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl std::fmt::Debug for crate::RawTransactionIngestStandardBlockSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert!(!source_impl.contains("observe_pending("), "direct Standard Block source must not inflate hydration_pending");
|
||||||
|
assert!(source_impl.contains("processing_frontier.observe_settled(slot)"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_005_helius_transaction_redaction_full_reference_and_tier_neutrality_are_explicit() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestHeliusTransactionSource",
|
||||||
|
".field(\"filter\"",
|
||||||
|
".field(\"filter_fingerprint_bytes\"",
|
||||||
|
".field(\"source_key_bytes\"",
|
||||||
|
"HeliusTransactionNotification::Full(value)",
|
||||||
|
"source.helius_transaction_notification_unqualified",
|
||||||
|
"project_helius_transaction_signal",
|
||||||
|
"transaction_index: std::option::Option::Some(response.transaction_index())",
|
||||||
|
"RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_HTTP_PROTOCOL",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.005 Helius hardening guard missing: {required}");
|
||||||
|
}
|
||||||
|
let source_struct = match resources.split_once("pub struct RawTransactionIngestHeliusTransactionSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl crate::RawTransactionIngestHeliusTransactionSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["url:", "api_key", "credential", "transaction:", "payload:", "tier:"] {
|
||||||
|
assert!(!source_struct.contains(forbidden), "sensitive/provider material stored in Helius transaction source: {forbidden}");
|
||||||
|
}
|
||||||
|
let projection = match resources.split_once("fn project_helius_transaction_signal") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("trait RawTransactionIngestStandardLogsView") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert!(!projection.contains(".transaction()"), "Helius nested full payload must not enter Worker projection");
|
||||||
|
for forbidden in ["Developer", "Business", "Professional", "paid_tier", "provider_tier"] {
|
||||||
|
assert!(!resources.contains(forbidden), "provider tier must not be coded in Worker: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_006_http_block_polling_is_bounded_run_local_stop_preemptible_and_redacted() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL",
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE",
|
||||||
|
"let mut next_scan_slot = start_slot",
|
||||||
|
"next_scan_slot = slot",
|
||||||
|
"tokio::time::sleep(self.poll_interval)",
|
||||||
|
"processing_frontier.observe_settled(slot)",
|
||||||
|
"source.http_block_polling_transaction_version_unqualified",
|
||||||
|
"source.http_block_polling_transaction_version_unsupported",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.006 HTTP polling hardening guard missing: {required}");
|
||||||
|
}
|
||||||
|
let source_impl = match resources.split_once("impl crate::RawTransactionIngestHttpBlockPollingSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl std::fmt::Debug for crate::RawTransactionIngestHttpBlockPollingSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert_eq!(source_impl.matches("get_block_observed(").count(), 1);
|
||||||
|
assert!(!source_impl.contains("tokio::spawn"), "HTTP polling must not spawn one task per tick");
|
||||||
|
assert!(!source_impl.contains("observe_pending("), "HTTP block polling is direct RAW and must not inflate hydration_pending");
|
||||||
|
let source_struct = match resources.split_once("pub struct RawTransactionIngestHttpBlockPollingSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl crate::RawTransactionIngestHttpBlockPollingSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["url:", "api_key", "credential", "secret", "transaction:", "payload:", "tier:"] {
|
||||||
|
assert!(!source_struct.contains(forbidden), "sensitive/raw material stored in HTTP polling source: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_009_hydration_retry_ownership_and_no_orphan_cleanup_are_explicit() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
assert_eq!(resources.matches("get_transaction_observed(").count(), 1, "Worker must delegate one hydration attempt to Transport");
|
||||||
|
for required in [
|
||||||
|
"coordinator.abort_all(&mut processing_frontier).await",
|
||||||
|
"fn discard_all_pending(&mut self)",
|
||||||
|
"processing_frontier.discard_all_pending()",
|
||||||
|
"source.hydration_pending_saturated",
|
||||||
|
"tasks.abort_all()",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.009 no-orphan/backpressure guard missing: {required}");
|
||||||
|
}
|
||||||
|
let hydration = match resources.split_once("struct RawTransactionIngestHydrationCoordinator {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("fn hydration_method_code(") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["tokio::time::sleep", "tokio::time::interval", "get_block_observed"] {
|
||||||
|
assert!(!hydration.contains(forbidden), "hydration coordinator introduced forbidden retry behavior: {forbidden}");
|
||||||
|
}
|
||||||
|
assert!(!resources.contains("unbounded_channel"), "Worker introduced an unbounded channel");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_007_multi_source_supervisor_is_fail_closed_joined_and_does_not_publish_source_identity() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES",
|
||||||
|
"RawTransactionIngestSourceInventory",
|
||||||
|
"std::sync::Mutex",
|
||||||
|
"tokio::task::JoinSet",
|
||||||
|
"source.configured_source_closed",
|
||||||
|
"source.task_join_failed",
|
||||||
|
"source_stop_sender.send_replace(true)",
|
||||||
|
"drain_live_source_tasks",
|
||||||
|
"processing_frontier_slot = std::option::Option::None",
|
||||||
|
"saturating_add",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.007 supervisor hardening guard missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"pub struct RawTransactionIngestSourceInventory",
|
||||||
|
"pub struct RawTransactionIngestSourceInventoryPublisher",
|
||||||
|
"pub fn source_key(",
|
||||||
|
"source_keys:",
|
||||||
|
"unbounded_channel",
|
||||||
|
"primary_source",
|
||||||
|
"standby_source",
|
||||||
|
"first_provider_wins",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "private source identity/scheduling policy leaked into public root: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_008_cross_source_convergence_is_bounded_conflict_checked_and_private() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let persistence = include_str!("../src/persistence.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"max_entries: usize",
|
||||||
|
"entries.len() >= self.max_entries",
|
||||||
|
"std::sync::Arc::strong_count(entry) == 1",
|
||||||
|
"known_state_value != &canonical_state",
|
||||||
|
"record_observation(observation)",
|
||||||
|
"persistence.additional_observation_not_recorded",
|
||||||
|
] {
|
||||||
|
assert!(persistence.contains(required), "required pre.008 persistence hardening guard missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"max_pending: usize",
|
||||||
|
"pending.len() >= self.max_pending",
|
||||||
|
"source.global_hydration_pending_saturated",
|
||||||
|
"tokio::sync::watch::channel",
|
||||||
|
"publish_and_remove",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.008 hydration hardening guard missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"pub struct RawTransactionIngestPersistenceConvergence",
|
||||||
|
"pub struct RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"pub fn persist_raw_transaction_ingest_converged_acquisition",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.008 private convergence implementation leaked publicly: {forbidden}");
|
||||||
|
}
|
||||||
|
assert!(!resources.contains("unbounded_channel"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_009_duplicate_storm_disagreement_and_starvation_guards_are_explicit() {
|
||||||
|
let admission = include_str!("../src/admission.rs");
|
||||||
|
let persistence = include_str!("../src/persistence.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"runtime_resources.hydration_pending_capacity_insufficient",
|
||||||
|
"runtime_resources.hydration_concurrency_insufficient",
|
||||||
|
"base + usize::from(hydration_entry_index < remainder)",
|
||||||
|
"hydration_in_flight_limit",
|
||||||
|
"tokio::sync::Semaphore::new(max_in_flight)",
|
||||||
|
"RawTransactionIngestHydrationLeaderGuard",
|
||||||
|
"source.global_hydration_pending_saturated",
|
||||||
|
"block_time: transaction.block_time()",
|
||||||
|
"slot: transaction.slot()",
|
||||||
|
"content_hash: transaction.payload().content_hash()",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
admission.contains(required) || persistence.contains(required) || resources.contains(required),
|
||||||
|
"required pre.009 adversarial guard missing: {required}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"VecDeque::new()",
|
||||||
|
"unbounded_channel",
|
||||||
|
"first_provider_wins",
|
||||||
|
"provider_priority",
|
||||||
|
"majority_vote",
|
||||||
|
"preferred_provider",
|
||||||
|
"overwrite_conflict",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!admission.contains(forbidden) && !persistence.contains(forbidden) && !resources.contains(forbidden),
|
||||||
|
"pre.009 introduced forbidden unbounded/provider-preference behavior: {forbidden}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_010_multi_source_health_is_conservative_counted_and_redacted() {
|
||||||
|
let snapshot = include_str!("../src/snapshot.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"source_total: usize",
|
||||||
|
"source_active: usize",
|
||||||
|
"source_reconnecting: usize",
|
||||||
|
"source_failed: usize",
|
||||||
|
"source_reconnecting > 0",
|
||||||
|
"source_failed > 0",
|
||||||
|
"source_active < source_total",
|
||||||
|
"WorkerHealth::Degraded",
|
||||||
|
"WorkerHealth::Unhealthy",
|
||||||
|
] {
|
||||||
|
assert!(snapshot.contains(required), "required pre.010 health guard missing: {required}");
|
||||||
|
}
|
||||||
|
for required in ["source_total = self.source_projections.len()", "with_source_counts(source_total, source_active, source_reconnecting, source_failed)"] {
|
||||||
|
assert!(resources.contains(required), "required pre.010 inventory count guard missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["source_key: [u8; 32]", "provider_url", "endpoint_url", "filter_fingerprint", "credential", "api_key", "token_header"] {
|
||||||
|
assert!(!snapshot.contains(forbidden), "pre.010 snapshot leaks source/provider material: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_011_shutdown_races_are_bounded_joined_atomic_and_counter_safe() {
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let supervisor = match runtime.split_once("async fn run_supervisor<Spawner>(") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("fn spawn_persistence(") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
let active_shutdown = match supervisor.split_once("let mut fault = supervise_until_stop(") {
|
||||||
|
std::option::Option::Some((_, value)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
let stop_position = active_shutdown.find("source_stop_sender.send_replace(true);");
|
||||||
|
let stopping_position = active_shutdown.find("begin_stopping(");
|
||||||
|
let drain_position = active_shutdown.find("drain_owned_work(");
|
||||||
|
let terminal_position = active_shutdown.find("match fault {");
|
||||||
|
assert!(matches!((stop_position, stopping_position), (std::option::Option::Some(stop), std::option::Option::Some(stopping)) if stop < stopping));
|
||||||
|
assert!(matches!((stopping_position, drain_position), (std::option::Option::Some(stopping), std::option::Option::Some(drain)) if stopping < drain));
|
||||||
|
assert!(matches!((drain_position, terminal_position), (std::option::Option::Some(drain), std::option::Option::Some(terminal)) if drain < terminal));
|
||||||
|
for required in [
|
||||||
|
"tokio::time::timeout(settings.shutdown_drain_timeout(), drain).await",
|
||||||
|
"persistence.abort_all()",
|
||||||
|
"children.abort_all()",
|
||||||
|
"while persistence.join_next().await.is_some() {}",
|
||||||
|
"while children.join_next().await.is_some() {}",
|
||||||
|
] {
|
||||||
|
assert!(runtime.contains(required), "required pre.011 bounded-drain guard missing: {required}");
|
||||||
|
}
|
||||||
|
let registry_publish = match resources.split_once("fn publish_and_remove(&self, key: &RawTransactionIngestHydrationKey") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("type RawTransactionIngestHydrationTasks") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
let notify_position = registry_publish.find("sender.send_replace(std::option::Option::Some(result));");
|
||||||
|
let remove_position = registry_publish.find("pending.remove(key);");
|
||||||
|
assert!(matches!((notify_position, remove_position), (std::option::Option::Some(notify), std::option::Option::Some(remove)) if notify < remove));
|
||||||
|
let inventory = match resources.split_once("impl RawTransactionIngestSourceInventory {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl RawTransactionIngestSourceInventoryPublisher") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for required in [
|
||||||
|
"checked_add(projection.hydration_pending())",
|
||||||
|
"checked_add(projection.source_continuity_gap_total())",
|
||||||
|
"checked_add(projection.source_reconnect_total())",
|
||||||
|
"checked_add(projection.source_replay_attempt_total())",
|
||||||
|
"source.inventory_entry_missing",
|
||||||
|
"source.inventory_key_mismatch",
|
||||||
|
"source.inventory_projection_missing",
|
||||||
|
] {
|
||||||
|
assert!(inventory.contains(required), "required pre.011 inventory guard missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(!inventory.contains("saturating_add"), "pre.011 source inventory must not silently saturate aggregate counters");
|
||||||
|
for source_marker in [
|
||||||
|
"impl crate::RawTransactionIngestYellowstoneSource",
|
||||||
|
"impl crate::RawTransactionIngestHeliusTransactionSource",
|
||||||
|
"impl crate::RawTransactionIngestHttpBlockPollingSource",
|
||||||
|
"impl crate::RawTransactionIngestStandardBlockSource",
|
||||||
|
"impl crate::RawTransactionIngestStandardLogsSource",
|
||||||
|
] {
|
||||||
|
let tail = match resources.split_once(source_marker) {
|
||||||
|
std::option::Option::Some((_, value)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert!(tail.contains("stop_receiver.changed()"), "pre.011 source lacks stop-preemptible wait: {source_marker}");
|
||||||
|
}
|
||||||
|
assert!(!runtime.contains("unbounded_channel"));
|
||||||
|
assert!(!resources.contains("unbounded_channel"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_cross_layer_security_closure_is_complete_and_source_neutral() {
|
||||||
|
let cross_layer = include_str!("cross_layer_completeness.rs");
|
||||||
|
for required in [
|
||||||
|
"v0_3_13_pre_012_legacy_v0_v1_is_proven_from_transport_through_worker_common_raw_to_store",
|
||||||
|
"v0_3_13_pre_012_yellowstone_non_regression_covers_identity_v1_reconnect_gap_and_worker_hydration",
|
||||||
|
"v0_3_13_pre_012_security_redaction_matrix_covers_all_live_sources_and_lower_layers",
|
||||||
|
] {
|
||||||
|
assert!(cross_layer.contains(required), "pre.012 security closure proof missing: {required}");
|
||||||
|
}
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"SourceSnapshotByKey",
|
||||||
|
"SourceHealthByKey",
|
||||||
|
"provider_health",
|
||||||
|
"endpoint_health",
|
||||||
|
"RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"RawTransactionIngestCanonicalState",
|
||||||
|
"source_key",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.012 public root leaked private/source-specific material: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
430
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
Normal file
430
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
Normal file
@@ -0,0 +1,430 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
// version: 22
|
||||||
|
|
||||||
|
//! External public-surface proofs for the RAW transaction ingest Worker foundation.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_kind_code_and_settings_are_consumable_from_crate_root() {
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE, "raw_transaction_ingest");
|
||||||
|
let kind_result = ksp_worker_api::WorkerKindCode::new(ksp_worker_raw_transaction_ingest_lib::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE);
|
||||||
|
assert!(kind_result.is_ok());
|
||||||
|
let kind = match kind_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(kind.as_str(), "raw_transaction_ingest");
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-mainnet-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let settings = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::with_defaults(network, worker_id);
|
||||||
|
assert_eq!(settings.network().as_str(), "mainnet");
|
||||||
|
assert_eq!(settings.worker_id().as_str(), "raw-ingest-mainnet-001");
|
||||||
|
assert_eq!(settings.admission_queue_capacity(), ksp_worker_raw_transaction_ingest_lib::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY);
|
||||||
|
assert_eq!(settings.persistence_concurrency(), ksp_worker_raw_transaction_ingest_lib::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY);
|
||||||
|
assert_eq!(settings.shutdown_drain_timeout(), ksp_worker_raw_transaction_ingest_lib::DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_invalid_network_and_worker_identity_remain_owned_by_lower_contracts() {
|
||||||
|
let network = ksp_store_lib::RawNetworkId::new("");
|
||||||
|
let worker_id = ksp_worker_api::WorkerId::new("invalid worker id");
|
||||||
|
assert!(network.is_err());
|
||||||
|
assert!(worker_id.is_err());
|
||||||
|
let network_error = match network {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(network_error.code(), ksp_store_lib::ERROR_CODE_RAW_MODEL_INVALID);
|
||||||
|
let worker_error = match worker_id {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(worker_error.code(), ksp_worker_api::ERROR_CODE_WORKER_ID_INVALID);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_runtime_bound_errors_are_stable_and_do_not_echo_values() {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-mainnet-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let result = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::new(network, worker_id, 0, 8, std::time::Duration::from_secs(5));
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID);
|
||||||
|
assert_eq!(error.message(), "invalid RAW transaction ingest Worker settings");
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].value(), "admission_queue_capacity");
|
||||||
|
assert!(!std::format!("{error:?}").contains("raw-ingest-mainnet-001"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle() {
|
||||||
|
let _start: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings,
|
||||||
|
std::sync::Arc<ksp_store_lib::Store>,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWorker::start;
|
||||||
|
let _request_stop: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle) -> bool =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle::request_stop;
|
||||||
|
let _wait_terminal: for<'a> fn(
|
||||||
|
&'a ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestTerminalFuture<'a> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle::wait_terminal;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
assert!(!root.contains("JoinHandle"));
|
||||||
|
assert!(!root.contains("JoinSet"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_007_store_and_content_conflict_error_codes_are_public_and_stable() {
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT.code(), "content_conflict");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED.code(), "store_failed");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_snapshot_surface_and_common_projection_are_public_and_stable() {
|
||||||
|
let _snapshot_source: fn(
|
||||||
|
&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle::snapshot_source;
|
||||||
|
let _worker_snapshot_source: fn(
|
||||||
|
&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle::worker_snapshot_source;
|
||||||
|
let _current: fn(
|
||||||
|
&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource::current;
|
||||||
|
let _wait: for<'a> fn(
|
||||||
|
&'a ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource,
|
||||||
|
ksp_worker_api::WorkerSnapshotSequence,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotFuture<'a> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource::wait_for_change;
|
||||||
|
fn require_worker_source<T: ksp_worker_api::WorkerSnapshotSource + std::marker::Send + std::marker::Sync>() {}
|
||||||
|
require_worker_source::<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource>();
|
||||||
|
let _state_type = std::any::type_name::<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSourceState>();
|
||||||
|
let _states = [
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSourceState::Active,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSourceState::Reconnecting,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSourceState::Closing,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSourceState::Closed,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSourceState::Failed,
|
||||||
|
];
|
||||||
|
let _source_state = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_state;
|
||||||
|
let _reconnect = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_reconnect_total;
|
||||||
|
let _replay = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_replay_attempt_total;
|
||||||
|
let _gap = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_continuity_gap_total;
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED.code(), "counter_exhausted");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_009_source_and_drain_timeout_error_codes_are_public_and_stable() {
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT.code(), "drain_timeout");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED.code(), "source_failed");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_002_runtime_resource_types_are_consumable_without_client_escape_hatch() {
|
||||||
|
let _source_new: fn(
|
||||||
|
ksp_onchain_transport_lib::YellowstoneGrpcChannel,
|
||||||
|
ksp_onchain_transport_lib::YellowstoneSubscribeRequest,
|
||||||
|
ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestYellowstoneSource> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestYellowstoneSource::new;
|
||||||
|
let _resources_new: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestYellowstoneSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::new;
|
||||||
|
let _start_with_resources: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings,
|
||||||
|
std::sync::Arc<ksp_store_lib::Store>,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWorker::start_with_runtime_resources;
|
||||||
|
let source = include_str!("../src/runtime_resources.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"pub fn channel(",
|
||||||
|
"pub fn http_pool(",
|
||||||
|
"pub fn subscribe_request(",
|
||||||
|
"pub fn hydration_role(",
|
||||||
|
"pub fn enqueue(",
|
||||||
|
"pub fn send(",
|
||||||
|
"pub fn inner(",
|
||||||
|
] {
|
||||||
|
assert!(!source.contains(forbidden), "runtime-resource implementation escape hatch present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_002_multi_source_runtime_resource_surface_is_bounded_and_source_neutral() {
|
||||||
|
let _source_count: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources) -> usize =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::source_count;
|
||||||
|
let _push_yellowstone: fn(
|
||||||
|
&mut ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestYellowstoneSource,
|
||||||
|
) -> ksp_core_lib::Result<()> = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::try_push_yellowstone_source;
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES, 32);
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
assert!(!root.contains("RawTransactionIngestLiveSource"));
|
||||||
|
assert!(!root.contains("source_key"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_003_standard_logs_runtime_resource_surface_is_typed_and_transport_owned() {
|
||||||
|
let _source_new: fn(
|
||||||
|
ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource::new;
|
||||||
|
let _resources_from_logs: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_standard_logs_source;
|
||||||
|
let _push_logs: fn(
|
||||||
|
&mut ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource,
|
||||||
|
) -> ksp_core_lib::Result<()> = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::try_push_standard_logs_source;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
assert!(root.contains("RawTransactionIngestStandardLogsSource"));
|
||||||
|
for forbidden in ["pub fn ws_endpoint(", "pub fn filter(", "pub fn http_pool(", "pub fn source_key("] {
|
||||||
|
assert!(!resources.contains(forbidden), "standard logs source implementation escape hatch present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_005_helius_transaction_runtime_resource_surface_is_typed_and_transport_owned() {
|
||||||
|
let _source_new: fn(
|
||||||
|
ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource::new;
|
||||||
|
let _resources_from_helius: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_helius_transaction_source;
|
||||||
|
let _push_helius: fn(
|
||||||
|
&mut ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource,
|
||||||
|
) -> ksp_core_lib::Result<()> = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::try_push_helius_transaction_source;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
assert!(root.contains("RawTransactionIngestHeliusTransactionSource"));
|
||||||
|
for forbidden in ["pub fn ws_endpoint(", "pub fn filter(", "pub fn http_pool(", "pub fn source_key("] {
|
||||||
|
assert!(!resources.contains(forbidden), "Helius transaction source implementation escape hatch present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_004_standard_block_runtime_resource_surface_is_typed_and_transport_owned() {
|
||||||
|
let _source_new: fn(
|
||||||
|
ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
ksp_onchain_transport_lib::SolanaBlockSubscribeFilter,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardBlockSource> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardBlockSource::new;
|
||||||
|
let _resources_from_block: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardBlockSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_standard_block_source;
|
||||||
|
let _push_block: fn(
|
||||||
|
&mut ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardBlockSource,
|
||||||
|
) -> ksp_core_lib::Result<()> = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::try_push_standard_block_source;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
assert!(root.contains("RawTransactionIngestStandardBlockSource"));
|
||||||
|
for forbidden in ["pub fn ws_endpoint(", "pub fn filter(", "pub fn source_key("] {
|
||||||
|
assert!(!resources.contains(forbidden), "standard block source implementation escape hatch present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_006_http_block_polling_runtime_resource_surface_is_bounded_and_transport_owned() {
|
||||||
|
let _source_new: fn(
|
||||||
|
ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource::new;
|
||||||
|
let _source_new_with_limits: fn(
|
||||||
|
ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
std::time::Duration,
|
||||||
|
u16,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource::new_with_limits;
|
||||||
|
let _resources_from_polling: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_http_block_polling_source;
|
||||||
|
let _push_polling: fn(
|
||||||
|
&mut ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource,
|
||||||
|
) -> ksp_core_lib::Result<()> = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::try_push_http_block_polling_source;
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL, std::time::Duration::from_secs(1));
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE, 128);
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL, std::time::Duration::from_millis(100));
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL, std::time::Duration::from_secs(30));
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE, 1);
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE, 1024);
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for forbidden in ["pub fn http_pool(", "pub fn polling_role(", "pub fn source_key(", "pub fn profile_fingerprint("] {
|
||||||
|
assert!(!resources.contains(forbidden), "HTTP polling source implementation escape hatch present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processing_only() {
|
||||||
|
let _hydration_pending: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> usize =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::hydration_pending;
|
||||||
|
let _processing_frontier_slot: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> std::option::Option<u64> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::processing_frontier_slot;
|
||||||
|
let _oldest_pending_slot: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> std::option::Option<u64> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::oldest_pending_slot;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in ["ReplayInfo", "from_slot", "repair", "backfill"] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.007 public root overclaims continuity/replay: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_007_source_inventory_and_logical_keys_remain_private() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in ["RawTransactionIngestSourceInventory", "RawTransactionIngestSourceInventoryPublisher", "RawTransactionIngestLiveSource", "source_key"] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.007 private multi-source implementation leaked through crate root: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_008_convergence_cache_registry_and_source_keys_remain_private() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"pub use self::persistence::RawTransactionIngestPersistenceConvergence",
|
||||||
|
"pub use self::persistence::persist_raw_transaction_ingest_converged_acquisition",
|
||||||
|
"RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"RawTransactionIngestHydrationKey",
|
||||||
|
"source_key",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.008 convergence implementation leaked through public crate root: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_009_fairness_budgets_semaphore_and_canonical_state_remain_private() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for private_name in [
|
||||||
|
"RawTransactionIngestCanonicalState",
|
||||||
|
"RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"RawTransactionIngestHydrationLeaderGuard",
|
||||||
|
"validate_hydration_fairness_capacity",
|
||||||
|
"hydration_pending_limit",
|
||||||
|
"hydration_in_flight_limit",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(&std::format!("pub use self::runtime_resources::{private_name}")));
|
||||||
|
assert!(!root.contains(&std::format!("pub use self::persistence::{private_name}")));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_010_multi_source_snapshot_counts_are_public_and_source_neutral() {
|
||||||
|
let _source_total: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> usize =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_total;
|
||||||
|
let _source_active: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> usize =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_active;
|
||||||
|
let _source_reconnecting: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> usize =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_reconnecting;
|
||||||
|
let _source_failed: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> usize =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_failed;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in ["SourceHealthByKey", "SourceSnapshotByKey", "provider_health", "endpoint_health"] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.010 public root leaked source-specific health material: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_011_race_hardening_adds_no_public_runtime_or_source_identity_surface() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestSourceInventory",
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestSourceInventoryPublisher",
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestHydrationLeaderGuard",
|
||||||
|
"publish_and_remove",
|
||||||
|
"source.inventory_key_mismatch",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.011 private race-hardening implementation leaked publicly: {forbidden}");
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED,
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "counter_exhausted")
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT,
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "drain_timeout")
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_completeness_closure_adds_no_public_implementation_surface() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
assert!(!root.contains("pub mod "));
|
||||||
|
for forbidden in [
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestLiveSource",
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestSourceInventory",
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"pub use self::persistence::RawTransactionIngestCanonicalState",
|
||||||
|
"pub use self::persistence::RawTransactionIngestPersistenceConvergence",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.012 private implementation leaked through public root: {forbidden}");
|
||||||
|
}
|
||||||
|
let cross_layer = include_str!("cross_layer_completeness.rs");
|
||||||
|
assert!(cross_layer.contains("v0_3_13_pre_012_all_five_live_source_families_share_one_worker_common_raw_store_pipeline"));
|
||||||
|
assert!(cross_layer.contains("v0_3_13_pre_012_legacy_v0_v1_is_proven_from_transport_through_worker_common_raw_to_store"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,237 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
// version: 20
|
||||||
|
|
||||||
|
//! Release-completeness canaries through the `pre.012` cross-layer completeness/security tranche.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_production_module_inventory_is_exact() -> std::io::Result<()> {
|
||||||
|
let source_root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src");
|
||||||
|
let entries = match std::fs::read_dir(source_root) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for entry in entries {
|
||||||
|
let entry = match entry {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let file_type = match entry.file_type() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if !file_type.is_file() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match entry.file_name().into_string() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => continue,
|
||||||
|
};
|
||||||
|
if name.ends_with(".rs") {
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
names.sort_unstable();
|
||||||
|
assert_eq!(
|
||||||
|
names,
|
||||||
|
std::vec!["admission.rs", "error.rs", "identity.rs", "lib.rs", "persistence.rs", "runtime.rs", "runtime_resources.rs", "settings.rs", "snapshot.rs",]
|
||||||
|
);
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_public_root_export_inventory_is_exact() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let mut exports = std::vec::Vec::new();
|
||||||
|
for line in root.lines() {
|
||||||
|
let line = line.trim();
|
||||||
|
if !line.starts_with("pub use ") {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let path = line.trim_start_matches("pub use ").trim_end_matches(';');
|
||||||
|
let name = match path.rsplit("::").next() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
exports.push(name);
|
||||||
|
}
|
||||||
|
exports.sort_unstable();
|
||||||
|
assert_eq!(
|
||||||
|
exports,
|
||||||
|
std::vec![
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY",
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL",
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE",
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY",
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT",
|
||||||
|
"RAW_TRANSACTION_INGEST_WORKER_KIND_CODE",
|
||||||
|
"RawTransactionIngestHandle",
|
||||||
|
"RawTransactionIngestHeliusTransactionSource",
|
||||||
|
"RawTransactionIngestHttpBlockPollingSource",
|
||||||
|
"RawTransactionIngestRuntimeResources",
|
||||||
|
"RawTransactionIngestSettings",
|
||||||
|
"RawTransactionIngestSnapshot",
|
||||||
|
"RawTransactionIngestSnapshotFuture",
|
||||||
|
"RawTransactionIngestSnapshotSource",
|
||||||
|
"RawTransactionIngestSourceState",
|
||||||
|
"RawTransactionIngestStandardBlockSource",
|
||||||
|
"RawTransactionIngestStandardLogsSource",
|
||||||
|
"RawTransactionIngestTerminalFuture",
|
||||||
|
"RawTransactionIngestWorker",
|
||||||
|
"RawTransactionIngestYellowstoneSource",
|
||||||
|
]
|
||||||
|
);
|
||||||
|
assert!(!root.contains("pub mod "));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_external_hardening_suite_is_present_and_scoped() {
|
||||||
|
let hardening = include_str!("hardening.rs");
|
||||||
|
for required in [
|
||||||
|
"pre_010_external_error_codes_are_stable_unique_and_domain_scoped",
|
||||||
|
"pre_010_debug_and_settings_errors_redact_worker_identity_and_invalid_values",
|
||||||
|
"v0_3_12_pre_002_manifest_dependency_surface_opens_only_transport_and_remains_backend_neutral",
|
||||||
|
"v0_3_12_pre_003_private_signal_debug_and_shape_do_not_expose_signature_filters_or_payload",
|
||||||
|
"v0_3_12_pre_004_hydration_provenance_and_remote_material_are_bounded_and_redacted",
|
||||||
|
"v0_3_12_pre_005_block_and_continuity_shapes_are_bounded_redacted_and_raw_separated",
|
||||||
|
"pre_010_source_visibility_contract_uses_crate_root_for_shared_items",
|
||||||
|
"pre_010_production_surface_has_no_historical_backfill_or_retriever_contract",
|
||||||
|
"v0_3_12_pre_002_production_sources_keep_transport_confined_to_runtime_resources",
|
||||||
|
"v0_3_12_pre_006_runtime_resource_contract_opens_one_supervised_transport_source",
|
||||||
|
"v0_3_12_pre_006_source_coalescence_is_bounded_stop_preemptible_and_redacted",
|
||||||
|
"v0_3_12_pre_007_processing_frontier_is_bounded_processing_only_and_redacted",
|
||||||
|
"v0_3_12_pre_008_reconnect_projection_is_source_neutral_bounded_and_contains_no_replay_material",
|
||||||
|
"v0_3_12_pre_009_hydration_retry_ownership_and_no_orphan_cleanup_are_explicit",
|
||||||
|
"pre_010_lower_layers_have_no_dependency_return_to_concrete_worker",
|
||||||
|
"v0_3_12_pre_002_public_root_exposes_contract_types_without_transport_implementation_paths",
|
||||||
|
"v0_3_13_pre_003_standard_logs_redaction_and_reference_only_contract_are_explicit",
|
||||||
|
"v0_3_13_pre_004_standard_block_redaction_version_and_null_guards_are_explicit",
|
||||||
|
"v0_3_13_pre_005_helius_transaction_redaction_full_reference_and_tier_neutrality_are_explicit",
|
||||||
|
"v0_3_13_pre_006_http_block_polling_is_bounded_run_local_stop_preemptible_and_redacted",
|
||||||
|
"v0_3_13_pre_007_multi_source_supervisor_is_fail_closed_joined_and_does_not_publish_source_identity",
|
||||||
|
"v0_3_13_pre_008_cross_source_convergence_is_bounded_conflict_checked_and_private",
|
||||||
|
"v0_3_13_pre_009_duplicate_storm_disagreement_and_starvation_guards_are_explicit",
|
||||||
|
"v0_3_13_pre_010_multi_source_health_is_conservative_counted_and_redacted",
|
||||||
|
"v0_3_13_pre_011_shutdown_races_are_bounded_joined_atomic_and_counter_safe",
|
||||||
|
] {
|
||||||
|
assert!(hardening.contains(required), "required pre.010 hardening canary missing: {required}");
|
||||||
|
}
|
||||||
|
let dependency_boundary = include_str!("dependency_boundary.rs");
|
||||||
|
assert!(dependency_boundary.contains("pre_002_manifest_dependency_surface_is_exact"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_006_source_surface_hardens_shutdown_and_faults_without_backend_edges"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_006_productive_source_uses_transport_session_bounded_coalescence_and_existing_admission"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_002_manifest_opens_only_the_onchain_transport_live_source_edge"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_003_transaction_and_status_adapters_are_private_and_transport_facade_only"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_004_hydration_contract_uses_only_transport_facade_common_raw_and_existing_ingress"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_005_block_and_continuity_adapters_remain_private_and_transport_facade_only"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_007_processing_frontier_remains_run_local_and_backend_neutral_after_continuity_extension"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_008_worker_observes_transport_reconnect_replay_and_faults_only_on_proven_retention_gap"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_003_standard_logs_source_reuses_transport_facades_and_common_hydration_only"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_004_standard_block_source_is_direct_raw_and_transport_facade_only"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_005_helius_transaction_source_reuses_transport_facade_and_common_hydration_only"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_006_http_block_polling_reuses_transport_facades_without_becoming_backfill"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_007_multi_source_supervisor_and_inventory_remain_private_bounded_and_source_neutral"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_008_cross_source_convergence_reuses_store_and_transport_facades_only"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_009_global_bounds_and_fairness_stay_inside_worker_facades"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_010_multi_source_snapshot_health_remains_source_neutral_and_backend_free"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_011_shutdown_race_hardening_stays_inside_worker_and_existing_facades"));
|
||||||
|
let public_api = include_str!("public_api.rs");
|
||||||
|
assert!(public_api.contains("pre_003_kind_code_and_settings_are_consumable_from_crate_root"));
|
||||||
|
assert!(public_api.contains("pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle"));
|
||||||
|
assert!(public_api.contains("pre_008_snapshot_surface_and_common_projection_are_public_and_stable"));
|
||||||
|
assert!(public_api.contains("pre_009_source_and_drain_timeout_error_codes_are_public_and_stable"));
|
||||||
|
assert!(public_api.contains("v0_3_12_pre_002_runtime_resource_types_are_consumable_without_client_escape_hatch"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_002_multi_source_runtime_resource_surface_is_bounded_and_source_neutral"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_003_standard_logs_runtime_resource_surface_is_typed_and_transport_owned"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_004_standard_block_runtime_resource_surface_is_typed_and_transport_owned"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_006_http_block_polling_runtime_resource_surface_is_bounded_and_transport_owned"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_007_source_inventory_and_logical_keys_remain_private"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_008_convergence_cache_registry_and_source_keys_remain_private"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_009_fairness_budgets_semaphore_and_canonical_state_remain_private"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_010_multi_source_snapshot_counts_are_public_and_source_neutral"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_011_race_hardening_adds_no_public_runtime_or_source_identity_surface"));
|
||||||
|
assert!(public_api.contains("v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processing_only"));
|
||||||
|
assert!(public_api.contains("pre_008_snapshot_surface_and_common_projection_are_public_and_stable"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_010_cross_layer_completeness_suite_is_present_and_exact() -> std::io::Result<()> {
|
||||||
|
let test_root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("tests");
|
||||||
|
let entries = match std::fs::read_dir(test_root) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for entry in entries {
|
||||||
|
let entry = match entry {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let file_type = match entry.file_type() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if !file_type.is_file() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match entry.file_name().into_string() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => continue,
|
||||||
|
};
|
||||||
|
if name.ends_with(".rs") {
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
names.sort_unstable();
|
||||||
|
assert_eq!(names, std::vec!["cross_layer_completeness.rs", "dependency_boundary.rs", "hardening.rs", "public_api.rs", "release_completeness.rs"]);
|
||||||
|
let cross_layer = include_str!("cross_layer_completeness.rs");
|
||||||
|
for required in [
|
||||||
|
"v0_3_12_pre_010_dependency_firewall_is_exact_across_transport_worker_common_raw_store",
|
||||||
|
"v0_3_12_pre_010_legacy_v0_fixture_matrix_is_exact_across_transport_worker_and_common_raw",
|
||||||
|
"v0_3_12_pre_010_redaction_scanner_matrix_is_present_across_all_layers",
|
||||||
|
"v0_3_12_pre_010_public_roots_keep_implementation_modules_private",
|
||||||
|
] {
|
||||||
|
assert!(cross_layer.contains(required), "required pre.010 cross-layer canary missing: {required}");
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_cross_layer_completeness_security_inventory_is_exact() {
|
||||||
|
let cross_layer = include_str!("cross_layer_completeness.rs");
|
||||||
|
for required in [
|
||||||
|
"v0_3_13_pre_012_all_five_live_source_families_share_one_worker_common_raw_store_pipeline",
|
||||||
|
"v0_3_13_pre_012_legacy_v0_v1_is_proven_from_transport_through_worker_common_raw_to_store",
|
||||||
|
"v0_3_13_pre_012_yellowstone_non_regression_covers_identity_v1_reconnect_gap_and_worker_hydration",
|
||||||
|
"v0_3_13_pre_012_security_redaction_matrix_covers_all_live_sources_and_lower_layers",
|
||||||
|
] {
|
||||||
|
assert!(cross_layer.contains(required), "required pre.012 cross-layer canary missing: {required}");
|
||||||
|
}
|
||||||
|
let dependency_boundary = include_str!("dependency_boundary.rs");
|
||||||
|
let hardening = include_str!("hardening.rs");
|
||||||
|
let public_api = include_str!("public_api.rs");
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_012_cross_layer_closure_does_not_expand_worker_dependency_graph"));
|
||||||
|
assert!(hardening.contains("v0_3_13_pre_012_cross_layer_security_closure_is_complete_and_source_neutral"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_012_completeness_closure_adds_no_public_implementation_surface"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,414 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
// version: 6
|
||||||
|
|
||||||
|
fn material(
|
||||||
|
network: &str,
|
||||||
|
signature_byte: u8,
|
||||||
|
transaction_data: &str,
|
||||||
|
) -> std::option::Option<(ksp_store_lib::RawNetworkId, ksp_raw_transaction_lib::RawTransactionMaterial)> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new(network) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let signature = ksp_store_lib::RawTransactionSignature::new([signature_byte; 64]);
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
network.clone(),
|
||||||
|
signature,
|
||||||
|
42,
|
||||||
|
std::option::Option::Some(1_700_000_000),
|
||||||
|
transaction_data,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
return std::option::Option::Some((network, material));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn provenance() -> std::option::Option<ksp_store_lib::RawAcquisitionProvenance> {
|
||||||
|
let provider = match ksp_store_lib::RawProvenanceCode::new("deterministic-harness") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let protocol = match ksp_store_lib::RawProvenanceCode::new("internal") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let method = match ksp_store_lib::RawProvenanceCode::new("admission") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_lib::RawTimestamp::from_unix_millis(1_700_000_000_000) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(ksp_store_lib::RawAcquisitionProvenance::new(
|
||||||
|
provider,
|
||||||
|
protocol,
|
||||||
|
method,
|
||||||
|
ksp_store_lib::RawAcquisitionOrigin::Live,
|
||||||
|
received_at,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_006_bounded_admission_applies_async_backpressure_until_one_slot_is_consumed() {
|
||||||
|
let (network, first_material) = match material("mainnet", 1, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (_, second_material) = match material("mainnet", 2, "BAUG") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
|
||||||
|
let first_ingress = crate::RawTransactionIngress {
|
||||||
|
material: first_material,
|
||||||
|
network: network.clone(),
|
||||||
|
provenance: first_provenance,
|
||||||
|
source_key: [1; 32],
|
||||||
|
};
|
||||||
|
assert!(sender.send(first_ingress).await.is_ok());
|
||||||
|
let second_sender = sender.clone();
|
||||||
|
let second_network = network.clone();
|
||||||
|
let completed = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
|
||||||
|
let completed_task = std::sync::Arc::clone(&completed);
|
||||||
|
let second_task = tokio::spawn(async move {
|
||||||
|
let second_ingress = crate::RawTransactionIngress {
|
||||||
|
material: second_material,
|
||||||
|
network: second_network,
|
||||||
|
provenance: second_provenance,
|
||||||
|
source_key: [2; 32],
|
||||||
|
};
|
||||||
|
let admitted = second_sender.send(second_ingress).await.is_ok();
|
||||||
|
completed_task.store(admitted, std::sync::atomic::Ordering::Release);
|
||||||
|
return admitted;
|
||||||
|
});
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
assert!(!completed.load(std::sync::atomic::Ordering::Acquire));
|
||||||
|
let first = admission.receive(&network).await;
|
||||||
|
assert!(matches!(first, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
for _ in 0..64 {
|
||||||
|
if completed.load(std::sync::atomic::Ordering::Acquire) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert!(completed.load(std::sync::atomic::Ordering::Acquire));
|
||||||
|
let joined = second_task.await;
|
||||||
|
assert!(matches!(joined, std::result::Result::Ok(true)));
|
||||||
|
let second = admission.receive(&network).await;
|
||||||
|
assert!(matches!(second, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_006_stop_preempts_a_blocked_sender_without_silent_post_stop_admission() {
|
||||||
|
let (network, first_material) = match material("mainnet", 3, "BwgJ") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (_, second_material) = match material("mainnet", 4, "CgsM") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (stop_sender, stop_receiver) = tokio::sync::watch::channel(false);
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
|
||||||
|
let first_ingress = crate::RawTransactionIngress {
|
||||||
|
material: first_material,
|
||||||
|
network: network.clone(),
|
||||||
|
provenance: first_provenance,
|
||||||
|
source_key: [3; 32],
|
||||||
|
};
|
||||||
|
assert!(sender.send(first_ingress).await.is_ok());
|
||||||
|
let blocked_sender = sender.clone();
|
||||||
|
let blocked_network = network.clone();
|
||||||
|
let blocked_task = tokio::spawn(async move {
|
||||||
|
let mut blocked_stop = stop_receiver;
|
||||||
|
let blocked_ingress = crate::RawTransactionIngress {
|
||||||
|
material: second_material,
|
||||||
|
network: blocked_network,
|
||||||
|
provenance: second_provenance,
|
||||||
|
source_key: [4; 32],
|
||||||
|
};
|
||||||
|
return tokio::select! {
|
||||||
|
biased;
|
||||||
|
_changed = blocked_stop.changed() => false,
|
||||||
|
result = blocked_sender.send(blocked_ingress) => result.is_ok(),
|
||||||
|
};
|
||||||
|
});
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
stop_sender.send_replace(true);
|
||||||
|
let blocked = blocked_task.await;
|
||||||
|
assert!(matches!(blocked, std::result::Result::Ok(false)));
|
||||||
|
let first = admission.receive(&network).await;
|
||||||
|
assert!(matches!(first, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
admission.close();
|
||||||
|
let end = admission.receive(&network).await;
|
||||||
|
assert!(matches!(end, std::result::Result::Ok(std::option::Option::None)));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_observation_key_domain_and_common_raw_assembly_golden_are_exact() {
|
||||||
|
let (network, material) = match material("mainnet", 7, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let ingress = crate::RawTransactionIngress { material, network: network.clone(), provenance: provenance.clone(), source_key: [9; 32] };
|
||||||
|
let acquisition = super::canonicalize_ingress(&network, ingress);
|
||||||
|
assert!(acquisition.is_ok());
|
||||||
|
let acquisition = match acquisition {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(acquisition.transaction().reference().network(), &network);
|
||||||
|
assert_eq!(acquisition.transaction().reference().signature().as_bytes(), &[7; 64]);
|
||||||
|
assert_eq!(acquisition.transaction().slot(), 42);
|
||||||
|
assert_eq!(acquisition.transaction().payload().format_id().as_str(), ksp_raw_transaction_lib::RAW_TRANSACTION_FORMAT_ID);
|
||||||
|
assert_eq!(acquisition.transaction().payload().format_version(), ksp_raw_transaction_lib::RAW_TRANSACTION_FORMAT_VERSION);
|
||||||
|
assert_eq!(acquisition.transaction().payload().bytes(), br#"{"transaction":["AQID","base64"]}"#);
|
||||||
|
assert_eq!(acquisition.observation().provenance(), &provenance);
|
||||||
|
assert_eq!(acquisition.observation().transaction(), acquisition.transaction().reference());
|
||||||
|
assert_eq!(
|
||||||
|
acquisition.observation().observation_key().as_bytes(),
|
||||||
|
&[
|
||||||
|
248, 209, 97, 220, 168, 168, 167, 42, 13, 161, 16, 252, 135, 136, 46, 231, 19, 104, 72, 70, 28, 106, 163, 43, 9, 227, 10, 97, 136, 185, 93, 204,
|
||||||
|
],
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_network_guards_reject_ingress_or_material_mismatch_without_echoing_values() {
|
||||||
|
let (mainnet, mainnet_material) = match material("mainnet", 8, "DQ4P") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (devnet, devnet_material) = match material("devnet", 9, "EBES") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let wrong_ingress = crate::RawTransactionIngress { material: mainnet_material, network: devnet, provenance: first_provenance, source_key: [10; 32] };
|
||||||
|
let wrong_ingress = super::canonicalize_ingress(&mainnet, wrong_ingress);
|
||||||
|
assert!(wrong_ingress.is_err());
|
||||||
|
let wrong_material = crate::RawTransactionIngress {
|
||||||
|
material: devnet_material,
|
||||||
|
network: mainnet.clone(),
|
||||||
|
provenance: second_provenance,
|
||||||
|
source_key: [11; 32],
|
||||||
|
};
|
||||||
|
let wrong_material = super::canonicalize_ingress(&mainnet, wrong_material);
|
||||||
|
assert!(wrong_material.is_err());
|
||||||
|
for result in [wrong_ingress, wrong_material] {
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => continue,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
let rendered = std::format!("{error:?}");
|
||||||
|
assert!(!rendered.contains("mainnet"));
|
||||||
|
assert!(!rendered.contains("devnet"));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_full_queue_dequeue_marks_source_neutral_backpressure_observation() {
|
||||||
|
let (network, material) = match material("mainnet", 13, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
|
||||||
|
let ingress = crate::RawTransactionIngress { material, network: network.clone(), provenance, source_key: [13; 32] };
|
||||||
|
assert!(sender.send(ingress).await.is_ok());
|
||||||
|
let first = admission.receive(&network).await;
|
||||||
|
assert!(matches!(first, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
assert!(admission.take_backpressure_wait_observed());
|
||||||
|
admission.close();
|
||||||
|
let second = admission.receive(&network).await;
|
||||||
|
assert!(matches!(second, std::result::Result::Ok(std::option::Option::None)));
|
||||||
|
assert!(!admission.take_backpressure_wait_observed());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_008_distinct_source_keys_produce_distinct_idempotent_observation_keys() {
|
||||||
|
let (network, first_material) = match material("mainnet", 51, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (_, second_material) = match material("mainnet", 51, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(2);
|
||||||
|
let first_ingress = crate::RawTransactionIngress {
|
||||||
|
material: first_material,
|
||||||
|
network: network.clone(),
|
||||||
|
provenance: first_provenance,
|
||||||
|
source_key: [11_u8; 32],
|
||||||
|
};
|
||||||
|
let second_ingress = crate::RawTransactionIngress {
|
||||||
|
material: second_material,
|
||||||
|
network: network.clone(),
|
||||||
|
provenance: second_provenance,
|
||||||
|
source_key: [12_u8; 32],
|
||||||
|
};
|
||||||
|
if sender.send(first_ingress).await.is_err() || sender.send(second_ingress).await.is_err() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
std::mem::drop(sender);
|
||||||
|
let first = match admission.receive(&network).await {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
let second = match admission.receive(&network).await {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
assert_eq!(first.transaction().reference(), second.transaction().reference());
|
||||||
|
assert_eq!(first.transaction().payload().content_hash(), second.transaction().payload().content_hash());
|
||||||
|
assert_ne!(first.observation().observation_key(), second.observation().observation_key());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_009_duplicate_storm_cannot_starve_second_ready_source_on_bounded_admission() {
|
||||||
|
let (network, initial_material) = match material("mainnet", 60, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (_, peer_material) = match material("mainnet", 99, "BwgJ") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let initial_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let storm_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let peer_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
|
||||||
|
assert!(
|
||||||
|
sender
|
||||||
|
.send(crate::RawTransactionIngress {
|
||||||
|
material: initial_material,
|
||||||
|
network: network.clone(),
|
||||||
|
provenance: initial_provenance,
|
||||||
|
source_key: [60_u8; 32],
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.is_ok()
|
||||||
|
);
|
||||||
|
let storm_sender = sender.clone();
|
||||||
|
let storm_network = network.clone();
|
||||||
|
let storm_task = tokio::spawn(async move {
|
||||||
|
for _ in 0..8 {
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
storm_network.clone(),
|
||||||
|
ksp_store_lib::RawTransactionSignature::new([61_u8; 64]),
|
||||||
|
42,
|
||||||
|
std::option::Option::Some(1_700_000_000),
|
||||||
|
"BAUG",
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
if storm_sender
|
||||||
|
.send(crate::RawTransactionIngress {
|
||||||
|
material,
|
||||||
|
network: storm_network.clone(),
|
||||||
|
provenance: storm_provenance.clone(),
|
||||||
|
source_key: [61_u8; 32],
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
let peer_sender = sender.clone();
|
||||||
|
let peer_network = network.clone();
|
||||||
|
let peer_task = tokio::spawn(async move {
|
||||||
|
return peer_sender
|
||||||
|
.send(crate::RawTransactionIngress { material: peer_material, network: peer_network, provenance: peer_provenance, source_key: [99_u8; 32] })
|
||||||
|
.await
|
||||||
|
.is_ok();
|
||||||
|
});
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
assert_eq!(admission.queue_depth(), 1);
|
||||||
|
let first = match admission.receive(&network).await {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
assert_eq!(first.transaction().reference().signature().as_bytes(), &[60_u8; 64]);
|
||||||
|
let second = match admission.receive(&network).await {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
assert_eq!(second.transaction().reference().signature().as_bytes(), &[61_u8; 64]);
|
||||||
|
let third = match admission.receive(&network).await {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
assert_eq!(third.transaction().reference().signature().as_bytes(), &[99_u8; 64]);
|
||||||
|
for _ in 0..7 {
|
||||||
|
let next = admission.receive(&network).await;
|
||||||
|
assert!(matches!(next, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
assert!(admission.queue_depth() <= 1);
|
||||||
|
}
|
||||||
|
let peer_joined = peer_task.await;
|
||||||
|
assert!(matches!(peer_joined, std::result::Result::Ok(true)));
|
||||||
|
let storm_joined = storm_task.await;
|
||||||
|
assert!(matches!(storm_joined, std::result::Result::Ok(true)));
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,408 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/persistence.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
#[derive(Clone, Copy)]
|
||||||
|
enum PortResponse {
|
||||||
|
Outcome(ksp_store_lib::RawEntityWriteOutcome, ksp_store_lib::RawObservationWriteOutcome),
|
||||||
|
Conflict,
|
||||||
|
StoreFailure,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct FakePersistencePort {
|
||||||
|
calls: std::sync::atomic::AtomicUsize,
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
normal_mode_seen: std::sync::atomic::AtomicBool,
|
||||||
|
observation_calls: std::sync::atomic::AtomicUsize,
|
||||||
|
observation_response: std::option::Option<ksp_store_lib::RawObservationWriteOutcome>,
|
||||||
|
response: PortResponse,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FakePersistencePort {
|
||||||
|
fn new(network: ksp_store_lib::RawNetworkId, response: PortResponse) -> Self {
|
||||||
|
return Self {
|
||||||
|
calls: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
network,
|
||||||
|
normal_mode_seen: std::sync::atomic::AtomicBool::new(false),
|
||||||
|
observation_calls: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
observation_response: std::option::Option::None,
|
||||||
|
response,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn with_observation_response(mut self, observation_response: ksp_store_lib::RawObservationWriteOutcome) -> Self {
|
||||||
|
self.observation_response = std::option::Option::Some(observation_response);
|
||||||
|
return self;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistencePort for FakePersistencePort {
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool {
|
||||||
|
return &self.network == network;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
_transaction: ksp_store_lib::RawTransaction,
|
||||||
|
_observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>> {
|
||||||
|
self.calls.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
self.normal_mode_seen.store(mode == ksp_store_lib::RawTransactionAcquisitionMode::Normal, std::sync::atomic::Ordering::Release);
|
||||||
|
let response = self.response;
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
return match response {
|
||||||
|
PortResponse::Outcome(entity, observation) => std::result::Result::Ok(ksp_store_lib::RawAcquisitionWriteOutcome::new(entity, observation)),
|
||||||
|
PortResponse::Conflict => std::result::Result::Err(ksp_core_lib::Error::new(ksp_store_lib::ERROR_CODE_RAW_CONFLICT, "synthetic conflict")),
|
||||||
|
PortResponse::StoreFailure => std::result::Result::Err(ksp_core_lib::Error::new(
|
||||||
|
ksp_core_lib::ErrorCode::new("synthetic_store", "write_failed"),
|
||||||
|
"synthetic remote failure text that must not escape",
|
||||||
|
)),
|
||||||
|
};
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
fn record_observation<'a>(
|
||||||
|
&'a self,
|
||||||
|
_observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawObservationWriteOutcome>> {
|
||||||
|
self.observation_calls.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
let observation_response = self.observation_response;
|
||||||
|
let response = self.response;
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
if let std::option::Option::Some(outcome) = observation_response {
|
||||||
|
return std::result::Result::Ok(outcome);
|
||||||
|
}
|
||||||
|
return match response {
|
||||||
|
PortResponse::Outcome(_, observation) => std::result::Result::Ok(observation),
|
||||||
|
PortResponse::Conflict => std::result::Result::Err(ksp_core_lib::Error::new(ksp_store_lib::ERROR_CODE_RAW_CONFLICT, "synthetic conflict")),
|
||||||
|
PortResponse::StoreFailure => std::result::Result::Err(ksp_core_lib::Error::new(
|
||||||
|
ksp_core_lib::ErrorCode::new("synthetic_store", "write_failed"),
|
||||||
|
"synthetic remote failure text that must not escape",
|
||||||
|
)),
|
||||||
|
};
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn acquisition(network_name: &str, signature_byte: u8) -> std::option::Option<ksp_raw_transaction_lib::RawTransactionAcquisition> {
|
||||||
|
return acquisition_with_observation_key(network_name, signature_byte, signature_byte);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn acquisition_with_observation_key(
|
||||||
|
network_name: &str,
|
||||||
|
signature_byte: u8,
|
||||||
|
observation_byte: u8,
|
||||||
|
) -> std::option::Option<ksp_raw_transaction_lib::RawTransactionAcquisition> {
|
||||||
|
return acquisition_with_shape(network_name, signature_byte, observation_byte, 42, std::option::Option::Some(1_700_000_000), "AQID");
|
||||||
|
}
|
||||||
|
|
||||||
|
fn acquisition_with_shape(
|
||||||
|
network_name: &str,
|
||||||
|
signature_byte: u8,
|
||||||
|
observation_byte: u8,
|
||||||
|
slot: u64,
|
||||||
|
block_time: std::option::Option<i64>,
|
||||||
|
transaction_data: &str,
|
||||||
|
) -> std::option::Option<ksp_raw_transaction_lib::RawTransactionAcquisition> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new(network_name) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let signature = ksp_store_lib::RawTransactionSignature::new([signature_byte; 64]);
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
network,
|
||||||
|
signature,
|
||||||
|
slot,
|
||||||
|
block_time,
|
||||||
|
transaction_data,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
let transaction = match ksp_raw_transaction_lib::canonicalize_raw_transaction(material) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let provider = match ksp_store_lib::RawProvenanceCode::new("deterministic-harness") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let protocol = match ksp_store_lib::RawProvenanceCode::new("internal") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let method = match ksp_store_lib::RawProvenanceCode::new("persistence") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_lib::RawTimestamp::from_unix_millis(1_700_000_000_000) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let provenance = ksp_store_lib::RawAcquisitionProvenance::new(provider, protocol, method, ksp_store_lib::RawAcquisitionOrigin::Live, received_at);
|
||||||
|
return std::option::Option::Some(ksp_raw_transaction_lib::assemble_raw_transaction_acquisition(
|
||||||
|
transaction,
|
||||||
|
ksp_store_lib::RawObservationKey::new([observation_byte; 32]),
|
||||||
|
provenance,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn network(name: &str) -> std::option::Option<ksp_store_lib::RawNetworkId> {
|
||||||
|
return match ksp_store_lib::RawNetworkId::new(name) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_normal_mode_classifies_new_idempotent_and_purged_outcomes() {
|
||||||
|
let cases = [
|
||||||
|
(
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
crate::RawTransactionIngestEntityPersistence::Inserted,
|
||||||
|
crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent, ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent),
|
||||||
|
crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
crate::RawTransactionIngestObservationPersistence::AlreadyPresent,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::SkippedPurged, ksp_store_lib::RawObservationWriteOutcome::NotRecorded),
|
||||||
|
crate::RawTransactionIngestEntityPersistence::SkippedPurged,
|
||||||
|
crate::RawTransactionIngestObservationPersistence::NotRecorded,
|
||||||
|
),
|
||||||
|
];
|
||||||
|
for (index, (response, expected_entity, expected_observation)) in cases.into_iter().enumerate() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, response);
|
||||||
|
let acquisition = match acquisition("mainnet", index as u8 + 1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
let outcome = match result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(outcome.entity(), expected_entity);
|
||||||
|
assert_eq!(outcome.observation(), expected_observation);
|
||||||
|
assert!(port.normal_mode_seen.load(std::sync::atomic::Ordering::Acquire));
|
||||||
|
assert_eq!(port.calls.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_content_conflict_maps_to_terminal_worker_code_without_remote_text() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, PortResponse::Conflict);
|
||||||
|
let acquisition = match acquisition("mainnet", 9) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT);
|
||||||
|
assert!(error.context().is_empty());
|
||||||
|
assert!(!std::format!("{error:?}").contains("synthetic conflict"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_store_failure_keeps_only_safe_lower_error_code() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, PortResponse::StoreFailure);
|
||||||
|
let acquisition = match acquisition("mainnet", 10) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED);
|
||||||
|
assert_eq!(error.context().len(), 2);
|
||||||
|
assert_eq!(error.context()[0].value(), "synthetic_store");
|
||||||
|
assert_eq!(error.context()[1].value(), "write_failed");
|
||||||
|
assert!(!std::format!("{error:?}").contains("synthetic remote failure text"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_rehydrated_or_impossible_store_outcomes_are_runtime_invalid() {
|
||||||
|
let cases = [
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Rehydrated, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent),
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::SkippedPurged, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
];
|
||||||
|
for (index, response) in cases.into_iter().enumerate() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, response);
|
||||||
|
let acquisition = match acquisition("mainnet", index as u8 + 20) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => continue,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].value(), "persistence.store_outcome_invalid");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_store_network_mismatch_is_rejected_before_write() {
|
||||||
|
let network = match network("devnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(
|
||||||
|
network,
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
);
|
||||||
|
let acquisition = match acquisition("mainnet", 30) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
assert_eq!(error.context()[0].value(), "persistence.store_network_mismatch");
|
||||||
|
assert_eq!(port.calls.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_008_same_canonical_identity_uses_atomic_entity_once_then_additional_observation() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(
|
||||||
|
network,
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
);
|
||||||
|
let convergence = crate::RawTransactionIngestPersistenceConvergence::new(8);
|
||||||
|
let first = match acquisition_with_observation_key("mainnet", 41, 1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second = match acquisition_with_observation_key("mainnet", 41, 2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_outcome = crate::persist_raw_transaction_ingest_converged_acquisition(&port, first, &convergence).await;
|
||||||
|
let second_outcome = crate::persist_raw_transaction_ingest_converged_acquisition(&port, second, &convergence).await;
|
||||||
|
assert!(first_outcome.is_ok());
|
||||||
|
assert!(second_outcome.is_ok());
|
||||||
|
assert_eq!(port.calls.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert_eq!(port.observation_calls.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
let second_outcome = match second_outcome {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(second_outcome.entity(), crate::RawTransactionIngestEntityPersistence::AlreadyPresent);
|
||||||
|
assert_eq!(second_outcome.observation(), crate::RawTransactionIngestObservationPersistence::Inserted);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_008_replayed_same_observation_is_idempotent_without_raw_resubmission() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(
|
||||||
|
network,
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
)
|
||||||
|
.with_observation_response(ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent);
|
||||||
|
let convergence = crate::RawTransactionIngestPersistenceConvergence::new(8);
|
||||||
|
let first = match acquisition_with_observation_key("mainnet", 42, 3) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second = match acquisition_with_observation_key("mainnet", 42, 3) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_result = crate::persist_raw_transaction_ingest_converged_acquisition(&port, first, &convergence).await;
|
||||||
|
assert!(first_result.is_ok());
|
||||||
|
let second_result = crate::persist_raw_transaction_ingest_converged_acquisition(&port, second, &convergence).await;
|
||||||
|
let second_outcome = match second_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(port.calls.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert_eq!(port.observation_calls.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert_eq!(second_outcome.observation(), crate::RawTransactionIngestObservationPersistence::AlreadyPresent);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_009_source_skew_and_provider_disagreement_fail_before_additional_observation() {
|
||||||
|
let cases = [
|
||||||
|
(43_u64, std::option::Option::Some(1_700_000_000_i64), "AQID"),
|
||||||
|
(42_u64, std::option::Option::Some(1_700_000_001_i64), "AQID"),
|
||||||
|
(42_u64, std::option::Option::Some(1_700_000_000_i64), "BAUG"),
|
||||||
|
];
|
||||||
|
for (index, (slot, block_time, transaction_data)) in cases.into_iter().enumerate() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(
|
||||||
|
network,
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
);
|
||||||
|
let convergence = crate::RawTransactionIngestPersistenceConvergence::new(8);
|
||||||
|
let first = match acquisition_with_shape("mainnet", 51, 10, 42, std::option::Option::Some(1_700_000_000), "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let divergent = match acquisition_with_shape("mainnet", 51, index as u8 + 20, slot, block_time, transaction_data) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert!(crate::persist_raw_transaction_ingest_converged_acquisition(&port, first, &convergence).await.is_ok());
|
||||||
|
let conflict = crate::persist_raw_transaction_ingest_converged_acquisition(&port, divergent, &convergence).await;
|
||||||
|
let error = match conflict {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT);
|
||||||
|
assert_eq!(port.calls.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert_eq!(port.observation_calls.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,996 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
// version: 11
|
||||||
|
|
||||||
|
struct ActiveTaskGuard {
|
||||||
|
active: std::sync::Arc<std::sync::atomic::AtomicUsize>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ActiveTaskGuard {
|
||||||
|
fn new(active: std::sync::Arc<std::sync::atomic::AtomicUsize>) -> Self {
|
||||||
|
active.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return Self { active };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::ops::Drop for ActiveTaskGuard {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
self.active.fetch_sub(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn settings(network: &str) -> std::option::Option<crate::RawTransactionIngestSettings> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new(network) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-runtime-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(crate::RawTransactionIngestSettings::with_defaults(network, worker_id));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_for_active_count(active: &std::sync::Arc<std::sync::atomic::AtomicUsize>, expected: usize) -> bool {
|
||||||
|
for _ in 0..64 {
|
||||||
|
if active.load(std::sync::atomic::Ordering::Acquire) == expected {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
return active.load(std::sync::atomic::Ordering::Acquire) == expected;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_current_runtime_is_required_before_spawn() {
|
||||||
|
let result = super::current_runtime_handle();
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].value(), "start.runtime_unavailable");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_store_network_mismatch_is_rejected_without_echoing_network_values() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let other = match ksp_store_lib::RawNetworkId::new("devnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let result = super::validate_store_network(&settings, &other);
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
let rendered = std::format!("{error:?}");
|
||||||
|
assert!(!rendered.contains("mainnet"));
|
||||||
|
assert!(!rendered.contains("devnet"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_004_immediate_stop_is_idempotent_and_reaches_stopped_terminal() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation(settings, tokio::runtime::Handle::current(), std::option::Option::None) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(handle.snapshot_source().current().worker_snapshot().state(), ksp_worker_api::WorkerState::Starting);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
assert!(!handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
assert!(terminal.is_ok());
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert!(terminal.is_terminal());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_004_running_lifecycle_stops_through_cloned_handle() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation(settings, tokio::runtime::Handle::current(), std::option::Option::None) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
assert_eq!(handle.snapshot_source().current().worker_snapshot().state(), ksp_worker_api::WorkerState::Running);
|
||||||
|
let clone = handle.clone();
|
||||||
|
assert!(clone.request_stop());
|
||||||
|
assert!(!handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_004_dropping_last_control_handle_causes_private_runtime_exit() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation(settings, tokio::runtime::Handle::current(), std::option::Option::None) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let observer = handle.snapshot_source();
|
||||||
|
std::mem::drop(handle);
|
||||||
|
loop {
|
||||||
|
let current = observer.current();
|
||||||
|
if current.worker_snapshot().state().is_terminal() {
|
||||||
|
assert_eq!(current.worker_snapshot().state(), ksp_worker_api::WorkerState::Stopped);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let _changed = observer.wait_for_change(current.worker_snapshot().sequence()).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_005_supervisor_joins_all_cooperative_source_tasks_before_terminal() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_active = std::sync::Arc::clone(&active);
|
||||||
|
let handle = match super::start_foundation_with_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::None,
|
||||||
|
move |children, stop_receiver, _admission_sender| {
|
||||||
|
for _ in 0..3 {
|
||||||
|
let active = std::sync::Arc::clone(&source_active);
|
||||||
|
let mut child_stop = stop_receiver.clone();
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let _guard = ActiveTaskGuard::new(active);
|
||||||
|
loop {
|
||||||
|
let changed = child_stop.changed().await;
|
||||||
|
if changed.is_err() || *child_stop.borrow() {
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_active_count(&active, 3).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_005_supervisor_reaps_completed_source_task_and_still_joins_live_child_on_stop() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let completed = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_active = std::sync::Arc::clone(&active);
|
||||||
|
let source_completed = std::sync::Arc::clone(&completed);
|
||||||
|
let handle = match super::start_foundation_with_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::None,
|
||||||
|
move |children, stop_receiver, _admission_sender| {
|
||||||
|
let completed = std::sync::Arc::clone(&source_completed);
|
||||||
|
let _completed_abort_handle = children.spawn(async move {
|
||||||
|
completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
let active = std::sync::Arc::clone(&source_active);
|
||||||
|
let mut child_stop = stop_receiver.clone();
|
||||||
|
let _active_abort_handle = children.spawn(async move {
|
||||||
|
let _guard = ActiveTaskGuard::new(active);
|
||||||
|
loop {
|
||||||
|
let changed = child_stop.changed().await;
|
||||||
|
if changed.is_err() || *child_stop.borrow() {
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_active_count(&active, 1).await);
|
||||||
|
for _ in 0..64 {
|
||||||
|
if completed.load(std::sync::atomic::Ordering::Acquire) == 1 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert_eq!(completed.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Copy)]
|
||||||
|
enum RuntimePortResponse {
|
||||||
|
Conflict,
|
||||||
|
StoreFailure,
|
||||||
|
StoreFailureBlocked,
|
||||||
|
SuccessBlocked,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct RuntimePortActiveGuard<'a> {
|
||||||
|
active: &'a std::sync::atomic::AtomicUsize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::ops::Drop for RuntimePortActiveGuard<'_> {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
self.active.fetch_sub(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
struct RuntimePersistencePort {
|
||||||
|
active: std::sync::atomic::AtomicUsize,
|
||||||
|
completed: std::sync::atomic::AtomicUsize,
|
||||||
|
max_active: std::sync::atomic::AtomicUsize,
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
normal_mode_seen: std::sync::atomic::AtomicBool,
|
||||||
|
notify: tokio::sync::Notify,
|
||||||
|
released: std::sync::atomic::AtomicBool,
|
||||||
|
response: RuntimePortResponse,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RuntimePersistencePort {
|
||||||
|
fn new(network: ksp_store_lib::RawNetworkId, response: RuntimePortResponse, released: bool) -> Self {
|
||||||
|
return Self {
|
||||||
|
active: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
completed: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
max_active: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
network,
|
||||||
|
normal_mode_seen: std::sync::atomic::AtomicBool::new(false),
|
||||||
|
notify: tokio::sync::Notify::new(),
|
||||||
|
released: std::sync::atomic::AtomicBool::new(released),
|
||||||
|
response,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn release(&self) {
|
||||||
|
self.released.store(true, std::sync::atomic::Ordering::Release);
|
||||||
|
self.notify.notify_waiters();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn update_max_active(&self, current: usize) {
|
||||||
|
let mut observed = self.max_active.load(std::sync::atomic::Ordering::Acquire);
|
||||||
|
while current > observed {
|
||||||
|
match self.max_active.compare_exchange_weak(observed, current, std::sync::atomic::Ordering::AcqRel, std::sync::atomic::Ordering::Acquire) {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => observed = value,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistencePort for RuntimePersistencePort {
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool {
|
||||||
|
return &self.network == network;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
_transaction: ksp_store_lib::RawTransaction,
|
||||||
|
_observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>> {
|
||||||
|
self.normal_mode_seen.store(mode == ksp_store_lib::RawTransactionAcquisitionMode::Normal, std::sync::atomic::Ordering::Release);
|
||||||
|
let response = self.response;
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
if matches!(response, RuntimePortResponse::StoreFailureBlocked | RuntimePortResponse::SuccessBlocked) {
|
||||||
|
let current = self.active.fetch_add(1, std::sync::atomic::Ordering::AcqRel) + 1;
|
||||||
|
let _active_guard = RuntimePortActiveGuard { active: &self.active };
|
||||||
|
self.update_max_active(current);
|
||||||
|
while !self.released.load(std::sync::atomic::Ordering::Acquire) {
|
||||||
|
self.notify.notified().await;
|
||||||
|
}
|
||||||
|
if matches!(response, RuntimePortResponse::SuccessBlocked) {
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Ok(ksp_store_lib::RawAcquisitionWriteOutcome::new(
|
||||||
|
ksp_store_lib::RawEntityWriteOutcome::Inserted,
|
||||||
|
ksp_store_lib::RawObservationWriteOutcome::Inserted,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if matches!(response, RuntimePortResponse::Conflict) {
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(ksp_store_lib::ERROR_CODE_RAW_CONFLICT, "synthetic conflict"));
|
||||||
|
}
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(
|
||||||
|
ksp_core_lib::ErrorCode::new("synthetic_store", "write_failed"),
|
||||||
|
"synthetic store failure",
|
||||||
|
));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
fn record_observation<'a>(
|
||||||
|
&'a self,
|
||||||
|
_observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawObservationWriteOutcome>> {
|
||||||
|
let response = self.response;
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
if matches!(response, RuntimePortResponse::StoreFailureBlocked | RuntimePortResponse::SuccessBlocked) {
|
||||||
|
let current = self.active.fetch_add(1, std::sync::atomic::Ordering::AcqRel) + 1;
|
||||||
|
let _active_guard = RuntimePortActiveGuard { active: &self.active };
|
||||||
|
self.update_max_active(current);
|
||||||
|
while !self.released.load(std::sync::atomic::Ordering::Acquire) {
|
||||||
|
self.notify.notified().await;
|
||||||
|
}
|
||||||
|
if matches!(response, RuntimePortResponse::SuccessBlocked) {
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Ok(ksp_store_lib::RawObservationWriteOutcome::Inserted);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if matches!(response, RuntimePortResponse::Conflict) {
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(ksp_store_lib::ERROR_CODE_RAW_CONFLICT, "synthetic conflict"));
|
||||||
|
}
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(
|
||||||
|
ksp_core_lib::ErrorCode::new("synthetic_store", "write_failed"),
|
||||||
|
"synthetic store failure",
|
||||||
|
));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn runtime_ingress(network: &ksp_store_lib::RawNetworkId, signature_byte: u8) -> std::option::Option<crate::RawTransactionIngress> {
|
||||||
|
let signature = ksp_store_lib::RawTransactionSignature::new([signature_byte; 64]);
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
network.clone(),
|
||||||
|
signature,
|
||||||
|
42,
|
||||||
|
std::option::Option::Some(1_700_000_000),
|
||||||
|
"AQID",
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
let provider = match ksp_store_lib::RawProvenanceCode::new("deterministic-harness") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let protocol = match ksp_store_lib::RawProvenanceCode::new("internal") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let method = match ksp_store_lib::RawProvenanceCode::new("runtime") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_lib::RawTimestamp::from_unix_millis(1_700_000_000_000) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let provenance = ksp_store_lib::RawAcquisitionProvenance::new(provider, protocol, method, ksp_store_lib::RawAcquisitionOrigin::Live, received_at);
|
||||||
|
return std::option::Option::Some(crate::RawTransactionIngress { material, network: network.clone(), provenance, source_key: [signature_byte; 32] });
|
||||||
|
}
|
||||||
|
|
||||||
|
fn settings_with_persistence_concurrency(concurrency: usize) -> std::option::Option<crate::RawTransactionIngestSettings> {
|
||||||
|
return settings_with_runtime_limits(8, concurrency, std::time::Duration::from_secs(5));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn settings_with_runtime_limits(
|
||||||
|
admission_queue_capacity: usize,
|
||||||
|
persistence_concurrency: usize,
|
||||||
|
shutdown_drain_timeout: std::time::Duration,
|
||||||
|
) -> std::option::Option<crate::RawTransactionIngestSettings> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-runtime-persistence-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return match crate::RawTransactionIngestSettings::new(network, worker_id, admission_queue_capacity, persistence_concurrency, shutdown_drain_timeout) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_for_completed(port: &std::sync::Arc<RuntimePersistencePort>, expected: usize) -> bool {
|
||||||
|
for _ in 0..128 {
|
||||||
|
if port.completed.load(std::sync::atomic::Ordering::Acquire) == expected {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
return port.completed.load(std::sync::atomic::Ordering::Acquire) == expected;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_for_max_active(port: &std::sync::Arc<RuntimePersistencePort>, expected: usize) -> bool {
|
||||||
|
for _ in 0..128 {
|
||||||
|
if port.max_active.load(std::sync::atomic::Ordering::Acquire) == expected {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
return port.max_active.load(std::sync::atomic::Ordering::Acquire) == expected;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_runtime_bounds_in_flight_store_persistence_to_configured_concurrency() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
for signature_byte in 1..=4 {
|
||||||
|
let ingress = match runtime_ingress(&network, signature_byte) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 2).await);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 2);
|
||||||
|
assert_eq!(port.max_active.load(std::sync::atomic::Ordering::Acquire), 2);
|
||||||
|
port.release();
|
||||||
|
assert!(wait_for_completed(&port, 4).await);
|
||||||
|
assert!(port.normal_mode_seen.load(std::sync::atomic::Ordering::Acquire));
|
||||||
|
assert_eq!(port.max_active.load(std::sync::atomic::Ordering::Acquire), 2);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_content_conflict_becomes_terminal_after_private_drain() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::Conflict, true));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 11) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT));
|
||||||
|
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
let snapshot = source.current();
|
||||||
|
assert_eq!(snapshot.content_conflict_total(), 1);
|
||||||
|
assert_eq!(snapshot.store_failure_total(), 0);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_store_failure_becomes_terminal_without_exposing_remote_error_text() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::StoreFailure, true));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 12) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED));
|
||||||
|
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert!(!std::format!("{handle:?}").contains("synthetic store failure"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_008_runtime_snapshots_count_successful_pipeline_and_retain_terminal() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, true));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
for signature_byte in 21..=23 {
|
||||||
|
let ingress = match runtime_ingress(&network, signature_byte) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let concrete_source = handle.snapshot_source();
|
||||||
|
let common_source = handle.worker_snapshot_source();
|
||||||
|
assert!(wait_for_completed(&port, 3).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
let snapshot = concrete_source.current();
|
||||||
|
assert_eq!(snapshot.worker_snapshot().state(), ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
assert_eq!(snapshot.admitted_total(), 3);
|
||||||
|
assert_eq!(snapshot.canonicalized_total(), 3);
|
||||||
|
assert_eq!(snapshot.persisted_total(), 3);
|
||||||
|
assert_eq!(snapshot.entity_inserted_total(), 3);
|
||||||
|
assert_eq!(snapshot.entity_already_present_total(), 0);
|
||||||
|
assert_eq!(snapshot.entity_skipped_purged_total(), 0);
|
||||||
|
assert_eq!(snapshot.observation_inserted_total(), 3);
|
||||||
|
assert_eq!(snapshot.observation_already_present_total(), 0);
|
||||||
|
assert_eq!(snapshot.content_conflict_total(), 0);
|
||||||
|
assert_eq!(snapshot.store_failure_total(), 0);
|
||||||
|
assert_eq!(snapshot.source_failure_total(), 0);
|
||||||
|
assert_eq!(snapshot.backpressure_wait_total(), 0);
|
||||||
|
assert_eq!(snapshot.admission_queue_depth(), 0);
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::current(&common_source);
|
||||||
|
assert_eq!(&common, snapshot.worker_snapshot());
|
||||||
|
let retained = concrete_source.wait_for_change(snapshot.worker_snapshot().sequence()).await;
|
||||||
|
assert_eq!(retained, snapshot);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_008_fault_snapshots_count_classified_store_failures_and_project_unhealthy() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::StoreFailure, true));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 24) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED));
|
||||||
|
let snapshot = source.current();
|
||||||
|
assert_eq!(snapshot.worker_snapshot().state(), terminal);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
assert_eq!(snapshot.admitted_total(), 1);
|
||||||
|
assert_eq!(snapshot.canonicalized_total(), 1);
|
||||||
|
assert_eq!(snapshot.persisted_total(), 0);
|
||||||
|
assert_eq!(snapshot.store_failure_total(), 1);
|
||||||
|
assert_eq!(snapshot.content_conflict_total(), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_source_failure_is_counted_and_late_stop_cannot_replace_terminal_fault() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation_with_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::None,
|
||||||
|
move |children: &mut tokio::task::JoinSet<ksp_core_lib::Result<()>>, _stop_receiver, _admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED));
|
||||||
|
let snapshot = source.current();
|
||||||
|
assert_eq!(snapshot.source_failure_total(), 1);
|
||||||
|
assert_eq!(snapshot.backpressure_wait_total(), 0);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
assert!(!handle.request_stop());
|
||||||
|
let retained = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(retained, terminal);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_stop_does_not_hide_store_failure_observed_during_bounded_drain() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::StoreFailureBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 41) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.ingress_invalid")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 1).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
port.release();
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED));
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
assert_eq!(snapshot.store_failure_total(), 1);
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_saturation_is_observable_without_drop_or_unbounded_admission() {
|
||||||
|
let settings = match settings_with_runtime_limits(1, 1, std::time::Duration::from_secs(5)) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let source_stage = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_stage_for_task = source_stage.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
for signature_byte in 51..=53 {
|
||||||
|
let ingress = match runtime_ingress(&network, signature_byte) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.ingress_invalid")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
source_stage_for_task.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 1).await);
|
||||||
|
for _ in 0..128 {
|
||||||
|
if source_stage.load(std::sync::atomic::Ordering::Acquire) == 2 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert_eq!(source_stage.load(std::sync::atomic::Ordering::Acquire), 2);
|
||||||
|
port.release();
|
||||||
|
assert!(wait_for_completed(&port, 3).await);
|
||||||
|
for _ in 0..128 {
|
||||||
|
if source_stage.load(std::sync::atomic::Ordering::Acquire) == 3 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert_eq!(source_stage.load(std::sync::atomic::Ordering::Acquire), 3);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
assert_eq!(snapshot.persisted_total(), 3);
|
||||||
|
assert_eq!(snapshot.source_failure_total(), 0);
|
||||||
|
assert!(snapshot.backpressure_wait_total() >= 1);
|
||||||
|
assert_eq!(snapshot.admission_queue_depth(), 0);
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_drain_timeout_aborts_and_joins_all_owned_source_and_persistence_tasks() {
|
||||||
|
let settings = match settings_with_runtime_limits(1, 1, crate::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let source_active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_active_for_task = source_active.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 61) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.ingress_invalid")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
let _guard = ActiveTaskGuard::new(source_active_for_task);
|
||||||
|
return std::future::pending::<ksp_core_lib::Result<()>>().await;
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 1).await);
|
||||||
|
assert!(wait_for_active_count(&source_active, 1).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT));
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
assert_eq!(snapshot.admission_queue_depth(), 0);
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
assert_eq!(source_active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_011_source_counter_exhaustion_stays_terminal_and_is_not_collapsed_to_source_failed() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation_with_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::None,
|
||||||
|
move |children: &mut tokio::task::JoinSet<ksp_core_lib::Result<()>>, _stop_receiver, _admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
return std::result::Result::Err(crate::counter_exhausted_error("source_reconnect_total"));
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED));
|
||||||
|
let snapshot = source.current();
|
||||||
|
assert_eq!(snapshot.source_failure_total(), 1);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().state(), terminal);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
let terminal_sequence = snapshot.worker_snapshot().sequence();
|
||||||
|
assert!(!handle.request_stop());
|
||||||
|
let retained = source.current();
|
||||||
|
assert_eq!(retained.worker_snapshot().state(), terminal);
|
||||||
|
assert_eq!(retained.worker_snapshot().sequence(), terminal_sequence);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_011_drain_timeout_prevents_late_persistence_completion_after_terminal() {
|
||||||
|
let settings = match settings_with_runtime_limits(1, 1, crate::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let source_active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_active_for_task = std::sync::Arc::clone(&source_active);
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 81) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.ingress_invalid")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
let _guard = ActiveTaskGuard::new(source_active_for_task);
|
||||||
|
return std::future::pending::<ksp_core_lib::Result<()>>().await;
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 1).await);
|
||||||
|
assert!(wait_for_active_count(&source_active, 1).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT));
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
let terminal_sequence = snapshot.worker_snapshot().sequence();
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
assert_eq!(source_active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
port.release();
|
||||||
|
for _ in 0..64 {
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
let retained = handle.snapshot_source().current();
|
||||||
|
assert_eq!(retained.worker_snapshot().state(), terminal);
|
||||||
|
assert_eq!(retained.worker_snapshot().sequence(), terminal_sequence);
|
||||||
|
assert_eq!(retained.in_flight_persistence(), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,105 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/settings.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
fn identities() -> std::option::Option<(ksp_store_lib::RawNetworkId, ksp_worker_api::WorkerId)> {
|
||||||
|
let network_result = ksp_store_lib::RawNetworkId::new("mainnet");
|
||||||
|
assert!(network_result.is_ok());
|
||||||
|
let network = match network_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let worker_id_result = ksp_worker_api::WorkerId::new("raw-ingest-mainnet-001");
|
||||||
|
assert!(worker_id_result.is_ok());
|
||||||
|
let worker_id = match worker_id_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some((network, worker_id));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_defaults_are_exact_and_preserve_typed_identity() {
|
||||||
|
let (network, worker_id) = match identities() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let settings = crate::RawTransactionIngestSettings::with_defaults(network.clone(), worker_id.clone());
|
||||||
|
assert_eq!(settings.network(), &network);
|
||||||
|
assert_eq!(settings.worker_id(), &worker_id);
|
||||||
|
assert_eq!(settings.admission_queue_capacity(), 256);
|
||||||
|
assert_eq!(settings.persistence_concurrency(), 8);
|
||||||
|
assert_eq!(settings.shutdown_drain_timeout(), std::time::Duration::from_secs(5));
|
||||||
|
assert_eq!(crate::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY, 256);
|
||||||
|
assert_eq!(crate::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY, 8);
|
||||||
|
assert_eq!(crate::DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT, std::time::Duration::from_secs(5));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_inclusive_bounds_accept_exact_minimum_and_maximum() {
|
||||||
|
for (queue, concurrency, drain) in [
|
||||||
|
(
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT,
|
||||||
|
),
|
||||||
|
] {
|
||||||
|
let (network, worker_id) = match identities() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let settings = crate::RawTransactionIngestSettings::new(network, worker_id, queue, concurrency, drain);
|
||||||
|
assert!(settings.is_ok());
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_invalid_runtime_bounds_return_stable_field_scoped_error() {
|
||||||
|
let invalid = [
|
||||||
|
(0, 8, std::time::Duration::from_secs(5), "admission_queue_capacity"),
|
||||||
|
(65_537, 8, std::time::Duration::from_secs(5), "admission_queue_capacity"),
|
||||||
|
(256, 0, std::time::Duration::from_secs(5), "persistence_concurrency"),
|
||||||
|
(256, 65, std::time::Duration::from_secs(5), "persistence_concurrency"),
|
||||||
|
(256, 8, std::time::Duration::from_millis(99), "shutdown_drain_timeout"),
|
||||||
|
(256, 8, std::time::Duration::from_millis(30_001), "shutdown_drain_timeout"),
|
||||||
|
];
|
||||||
|
for (queue, concurrency, drain, field) in invalid {
|
||||||
|
let (network, worker_id) = match identities() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::RawTransactionIngestSettings::new(network, worker_id, queue, concurrency, drain);
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => continue,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID);
|
||||||
|
assert_eq!(error.code().domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(error.code().code(), "settings_invalid");
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].key(), "field");
|
||||||
|
assert_eq!(error.context()[0].value(), field);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_debug_keeps_worker_identity_redacted() {
|
||||||
|
let (network, worker_id) = match identities() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let settings = crate::RawTransactionIngestSettings::with_defaults(network, worker_id);
|
||||||
|
let rendered = std::format!("{settings:?}");
|
||||||
|
assert!(rendered.contains("mainnet"));
|
||||||
|
assert!(rendered.contains("WorkerId(..)"));
|
||||||
|
assert!(!rendered.contains("raw-ingest-mainnet-001"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,251 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/snapshot.rs
|
||||||
|
// version: 5
|
||||||
|
|
||||||
|
fn snapshot_foundation_with_source_total(
|
||||||
|
source_total: usize,
|
||||||
|
) -> std::option::Option<(crate::RawTransactionIngestSnapshotPublisher, crate::RawTransactionIngestSnapshotSource)> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-snapshot-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let settings = crate::RawTransactionIngestSettings::with_defaults(network, worker_id.clone());
|
||||||
|
let kind = match ksp_worker_api::WorkerKindCode::new(crate::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let mut lifecycle = ksp_worker_api::WorkerLifecycle::new(worker_id, kind);
|
||||||
|
if lifecycle.start().is_err() {
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
return std::option::Option::Some(crate::RawTransactionIngestSnapshotPublisher::new(&settings, &lifecycle, source_total));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn snapshot_foundation() -> std::option::Option<(crate::RawTransactionIngestSnapshotPublisher, crate::RawTransactionIngestSnapshotSource)> {
|
||||||
|
return snapshot_foundation_with_source_total(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_initial_snapshot_and_common_projection_are_exact() {
|
||||||
|
let (publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let _publisher = publisher;
|
||||||
|
let concrete = source.current();
|
||||||
|
assert_eq!(concrete.worker_snapshot().sequence().value(), 0);
|
||||||
|
assert_eq!(concrete.worker_snapshot().state(), ksp_worker_api::WorkerState::Starting);
|
||||||
|
assert_eq!(concrete.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unknown);
|
||||||
|
assert_eq!(concrete.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Unknown);
|
||||||
|
assert_eq!(concrete.admission_queue_capacity(), crate::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY);
|
||||||
|
assert_eq!(concrete.admission_queue_depth(), 0);
|
||||||
|
assert_eq!(concrete.persistence_concurrency(), crate::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY);
|
||||||
|
assert_eq!(concrete.in_flight_persistence(), 0);
|
||||||
|
assert_eq!(concrete.admitted_total(), 0);
|
||||||
|
assert_eq!(concrete.canonicalized_total(), 0);
|
||||||
|
assert_eq!(concrete.persisted_total(), 0);
|
||||||
|
assert_eq!(concrete.entity_inserted_total(), 0);
|
||||||
|
assert_eq!(concrete.entity_already_present_total(), 0);
|
||||||
|
assert_eq!(concrete.entity_skipped_purged_total(), 0);
|
||||||
|
assert_eq!(concrete.observation_inserted_total(), 0);
|
||||||
|
assert_eq!(concrete.observation_already_present_total(), 0);
|
||||||
|
assert_eq!(concrete.content_conflict_total(), 0);
|
||||||
|
assert_eq!(concrete.store_failure_total(), 0);
|
||||||
|
assert_eq!(concrete.source_failure_total(), 0);
|
||||||
|
assert_eq!(concrete.backpressure_wait_total(), 0);
|
||||||
|
assert_eq!(concrete.hydration_pending(), 0);
|
||||||
|
assert_eq!(concrete.processing_frontier_slot(), std::option::Option::None);
|
||||||
|
assert_eq!(concrete.oldest_pending_slot(), std::option::Option::None);
|
||||||
|
assert_eq!(concrete.source_state(), std::option::Option::None);
|
||||||
|
assert_eq!(concrete.source_total(), 0);
|
||||||
|
assert_eq!(concrete.source_active(), 0);
|
||||||
|
assert_eq!(concrete.source_reconnecting(), 0);
|
||||||
|
assert_eq!(concrete.source_failed(), 0);
|
||||||
|
assert_eq!(concrete.source_reconnect_total(), 0);
|
||||||
|
assert_eq!(concrete.source_replay_attempt_total(), 0);
|
||||||
|
assert_eq!(concrete.source_continuity_gap_total(), 0);
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::current(&source);
|
||||||
|
assert_eq!(&common, concrete.worker_snapshot());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_checked_counter_exhaustion_is_stable_and_never_wraps() {
|
||||||
|
let result = super::checked_counter(u64::MAX, "admitted_total");
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED);
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].key(), "field");
|
||||||
|
assert_eq!(error.context()[0].value(), "admitted_total");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_008_slow_concrete_and_common_listeners_coalesce_to_same_latest_sequence() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let observed = source.current().worker_snapshot().sequence();
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Running, 0, 0).is_ok());
|
||||||
|
assert!(publisher.record_admission_success(ksp_worker_api::WorkerState::Running, 1, 1, false).is_ok());
|
||||||
|
let concrete = source.wait_for_change(observed).await;
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::wait_for_change(&source, observed).await;
|
||||||
|
assert_eq!(concrete.worker_snapshot().sequence(), common.sequence());
|
||||||
|
assert_eq!(concrete.worker_snapshot().sequence().value(), 2);
|
||||||
|
assert_eq!(concrete.worker_snapshot().state(), ksp_worker_api::WorkerState::Running);
|
||||||
|
assert_eq!(concrete.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(concrete.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Active);
|
||||||
|
assert_eq!(concrete.admission_queue_depth(), 1);
|
||||||
|
assert_eq!(concrete.in_flight_persistence(), 1);
|
||||||
|
assert_eq!(concrete.admitted_total(), 1);
|
||||||
|
assert_eq!(concrete.canonicalized_total(), 1);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_008_terminal_latest_value_is_retained_after_publisher_drop() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Running, 0, 0).is_ok());
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Stopping, 0, 0).is_ok());
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Stopped, 0, 0).is_ok());
|
||||||
|
std::mem::drop(publisher);
|
||||||
|
let retained = source.current();
|
||||||
|
assert_eq!(retained.worker_snapshot().state(), ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(retained.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(retained.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
let same_terminal = source.wait_for_change(retained.worker_snapshot().sequence()).await;
|
||||||
|
assert_eq!(same_terminal, retained);
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::wait_for_change(&source, retained.worker_snapshot().sequence()).await;
|
||||||
|
assert_eq!(common, retained.worker_snapshot().clone());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_009_source_failure_and_backpressure_counters_advance_without_changing_projection_contract() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Running, 0, 0).is_ok());
|
||||||
|
assert!(publisher.record_admission_success(ksp_worker_api::WorkerState::Running, 0, 1, true).is_ok());
|
||||||
|
assert!(publisher.record_source_failure(ksp_worker_api::WorkerState::Running, 0, 1).is_ok());
|
||||||
|
let concrete = source.current();
|
||||||
|
assert_eq!(concrete.admitted_total(), 1);
|
||||||
|
assert_eq!(concrete.canonicalized_total(), 1);
|
||||||
|
assert_eq!(concrete.backpressure_wait_total(), 1);
|
||||||
|
assert_eq!(concrete.source_failure_total(), 1);
|
||||||
|
assert_eq!(concrete.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(concrete.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Active);
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::current(&source);
|
||||||
|
assert_eq!(&common, concrete.worker_snapshot());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_007_processing_frontier_projection_is_latest_value_and_activity_aware() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let initial = source.current();
|
||||||
|
assert_eq!(initial.hydration_pending(), 0);
|
||||||
|
assert_eq!(initial.processing_frontier_slot(), std::option::Option::None);
|
||||||
|
assert_eq!(initial.oldest_pending_slot(), std::option::Option::None);
|
||||||
|
let projection = crate::RawTransactionIngestProcessingFrontierProjection::new(2, std::option::Option::Some(40), std::option::Option::Some(41));
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, projection).is_ok());
|
||||||
|
let pending = source.current();
|
||||||
|
assert_eq!(pending.hydration_pending(), 2);
|
||||||
|
assert_eq!(pending.processing_frontier_slot(), std::option::Option::Some(40));
|
||||||
|
assert_eq!(pending.oldest_pending_slot(), std::option::Option::Some(41));
|
||||||
|
assert_eq!(pending.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Active);
|
||||||
|
let projection = crate::RawTransactionIngestProcessingFrontierProjection::new(0, std::option::Option::Some(45), std::option::Option::None);
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, projection).is_ok());
|
||||||
|
let settled = source.current();
|
||||||
|
assert_eq!(settled.hydration_pending(), 0);
|
||||||
|
assert_eq!(settled.processing_frontier_slot(), std::option::Option::Some(45));
|
||||||
|
assert_eq!(settled.oldest_pending_slot(), std::option::Option::None);
|
||||||
|
assert_eq!(settled.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_source_continuity_projection_preserves_processing_frontier_and_distinguishes_replay() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let projection = crate::RawTransactionIngestProcessingFrontierProjection::new(2, std::option::Option::Some(40), std::option::Option::Some(41))
|
||||||
|
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting), 0, 1, 0);
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, projection).is_ok());
|
||||||
|
let reconnecting = source.current();
|
||||||
|
assert_eq!(reconnecting.hydration_pending(), 2);
|
||||||
|
assert_eq!(reconnecting.processing_frontier_slot(), std::option::Option::Some(40));
|
||||||
|
assert_eq!(reconnecting.oldest_pending_slot(), std::option::Option::Some(41));
|
||||||
|
assert_eq!(reconnecting.source_state(), std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting));
|
||||||
|
assert_eq!(reconnecting.source_reconnect_total(), 0);
|
||||||
|
assert_eq!(reconnecting.source_replay_attempt_total(), 1);
|
||||||
|
assert_eq!(reconnecting.source_continuity_gap_total(), 0);
|
||||||
|
assert_eq!(reconnecting.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Active);
|
||||||
|
let projection = crate::RawTransactionIngestProcessingFrontierProjection::new(0, std::option::Option::Some(45), std::option::Option::None)
|
||||||
|
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Active), 1, 1, 0);
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, projection).is_ok());
|
||||||
|
let resumed = source.current();
|
||||||
|
assert_eq!(resumed.hydration_pending(), 0);
|
||||||
|
assert_eq!(resumed.processing_frontier_slot(), std::option::Option::Some(45));
|
||||||
|
assert_eq!(resumed.source_state(), std::option::Option::Some(crate::RawTransactionIngestSourceState::Active));
|
||||||
|
assert_eq!(resumed.source_reconnect_total(), 1);
|
||||||
|
assert_eq!(resumed.source_replay_attempt_total(), 1);
|
||||||
|
assert_eq!(resumed.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_010_multi_source_counts_and_health_are_conservative_and_source_neutral() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation_with_source_total(3) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let initial = source.current();
|
||||||
|
assert_eq!(initial.source_total(), 3);
|
||||||
|
assert_eq!(initial.source_active(), 0);
|
||||||
|
assert_eq!(initial.source_reconnecting(), 0);
|
||||||
|
assert_eq!(initial.source_failed(), 0);
|
||||||
|
assert_eq!(initial.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unknown);
|
||||||
|
let reconnecting = crate::RawTransactionIngestProcessingFrontierProjection::new(1, std::option::Option::Some(50), std::option::Option::Some(50))
|
||||||
|
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting), 0, 1, 0)
|
||||||
|
.with_source_counts(3, 2, 1, 0);
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, reconnecting).is_ok());
|
||||||
|
let degraded = source.current();
|
||||||
|
assert_eq!(degraded.source_total(), 3);
|
||||||
|
assert_eq!(degraded.source_active(), 2);
|
||||||
|
assert_eq!(degraded.source_reconnecting(), 1);
|
||||||
|
assert_eq!(degraded.source_failed(), 0);
|
||||||
|
assert_eq!(degraded.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Degraded);
|
||||||
|
assert_eq!(degraded.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Active);
|
||||||
|
let active = crate::RawTransactionIngestProcessingFrontierProjection::new(0, std::option::Option::Some(51), std::option::Option::None)
|
||||||
|
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Active), 1, 1, 0)
|
||||||
|
.with_source_counts(3, 3, 0, 0);
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, active).is_ok());
|
||||||
|
let healthy = source.current();
|
||||||
|
assert_eq!(healthy.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(healthy.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
let failed = crate::RawTransactionIngestProcessingFrontierProjection::new(0, std::option::Option::Some(51), std::option::Option::None)
|
||||||
|
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Failed), 1, 1, 0)
|
||||||
|
.with_source_counts(3, 2, 0, 1);
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, failed).is_ok());
|
||||||
|
let unhealthy = source.current();
|
||||||
|
assert_eq!(unhealthy.source_failed(), 1);
|
||||||
|
assert_eq!(unhealthy.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
return;
|
||||||
|
}
|
||||||
393
deltas/0.3.11/pre.001.md
Normal file
393
deltas/0.3.11/pre.001.md
Normal file
@@ -0,0 +1,393 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.001.md -->
|
||||||
|
<!-- version: 2 -->
|
||||||
|
|
||||||
|
# Delta 0.3.11-pre.001 — audit/sizing fondation Worker RawTransaction ingest
|
||||||
|
|
||||||
|
## Base
|
||||||
|
|
||||||
|
```text
|
||||||
|
archive : khadhroony-solana-project-v0.3.10.zip
|
||||||
|
SHA-256 : befafea61304aaea15c94db7c8b3525c58b7c1622bf880a9110373bcc0ae6ac0
|
||||||
|
ZIP bytes : 7993096
|
||||||
|
ZIP entries : 1901
|
||||||
|
unzip -t : PASS
|
||||||
|
workspace members : 20
|
||||||
|
workspace.package.version base : 0.3.10
|
||||||
|
stable delta : deltas/0.3.10/rel.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Le ZIP a une racine unique et a été contrôlé sans entrée absolue, path traversal, symlink, `.git/`, `target/`, `node_modules/`, `Cargo.lock` ou `.env`.
|
||||||
|
|
||||||
|
Préconditions du prompt `030` confirmées :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-raw-transaction-lib présent et documenté
|
||||||
|
ksp-worker-api présent
|
||||||
|
ksp-job-backfill-lib présent et consommateur de ksp-raw-transaction-lib
|
||||||
|
ksp-worker-raw-transaction-ingest-lib absent
|
||||||
|
```
|
||||||
|
|
||||||
|
## Type de livraison
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-general-0.3.11-pre.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le type `general` est requis par le bump `Cargo.toml` de toute prerelease non-fix (`VER-ID-009`). L'archive d'échange contient uniquement les fichiers ajoutés/modifiés par cette tranche et son delta.
|
||||||
|
|
||||||
|
## Baseline exécutée avant modification
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 772 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
L'audit mécanique était vert mais la lecture humaine des règles a trouvé deux divergences normatives actives non détectées par les scripts.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Exécuter intégralement le gate d'ouverture demandé par `prompts/030-V0_3_11_START_PROMPT.md` : lecture/audit de la stable `v0.3.10`, contrôle renforcé des règles, inventaire Worker/Common RAW/Store/Backfill, fraîcheur des dépendances runtime, brainstorming public/private et supervisor, définition du harness déterministe, threat model, sizing et planification, sans commencer l'implémentation lourde du Worker.
|
||||||
|
|
||||||
|
## Réconciliation normative
|
||||||
|
|
||||||
|
### D1–D4
|
||||||
|
|
||||||
|
Défauts hérités trouvés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
KSP-DURABLE-002 : D2 encore nommé Core canonique
|
||||||
|
KSP-DURABLE-006 : instructions encore qualifiées de faits Core
|
||||||
|
DEP-STORE-004 : D2 encore nommé Core canonique
|
||||||
|
```
|
||||||
|
|
||||||
|
Les règles plus récentes et l'architecture `008` imposent déjà :
|
||||||
|
|
||||||
|
```text
|
||||||
|
D1 RAW
|
||||||
|
D2 STRUCTURAL
|
||||||
|
D3 DECODED
|
||||||
|
D4 DOMAIN
|
||||||
|
```
|
||||||
|
|
||||||
|
Correction : normalisation des trois règles actives. Les archives/deltas/plans historiques ne sont pas réécrits.
|
||||||
|
|
||||||
|
### Nom du Worker RAW
|
||||||
|
|
||||||
|
Les règles `KSP-WORKER-005/006/007/010` utilisaient encore :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-retriever
|
||||||
|
```
|
||||||
|
|
||||||
|
Le nom durable retenu par les architectures, ROADMAP, CHANGELOG, handoff `0.3.10` et prompt `030` est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Correction : normalisation du nom dans ces quatre règles, sans changement de leur responsabilité fonctionnelle.
|
||||||
|
|
||||||
|
Un scan des règles actives après correction ne trouve plus `D2 Core canonique`, `faits Core distincts` ni `ksp-worker-raw-retriever`.
|
||||||
|
|
||||||
|
## Inventaire et décisions de fondation
|
||||||
|
|
||||||
|
### Frontières préservées
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-api reste runtime-neutral et générique
|
||||||
|
ksp-raw-transaction-lib reste source-neutral sans runtime/Transport/Config/Worker/Job/backend
|
||||||
|
Worker persistera via ksp-store-lib default-features=false
|
||||||
|
Store reste vérité de convergence (network, signature)
|
||||||
|
Backfill et Worker restent producteurs indépendants
|
||||||
|
aucun paramètre historique métier au start Worker
|
||||||
|
aucun edge Worker -> Config/Job/backend physique
|
||||||
|
```
|
||||||
|
|
||||||
|
### Surface publique minimale retenue
|
||||||
|
|
||||||
|
```text
|
||||||
|
RAW_TRANSACTION_INGEST_WORKER_KIND_CODE
|
||||||
|
RawTransactionIngestSettings
|
||||||
|
RawTransactionIngestWorker
|
||||||
|
RawTransactionIngestHandle
|
||||||
|
RawTransactionIngestSnapshot
|
||||||
|
RawTransactionIngestSnapshotSource / future associée
|
||||||
|
ErrorCodes statiques Worker
|
||||||
|
```
|
||||||
|
|
||||||
|
Settings de fondation :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network
|
||||||
|
worker_id
|
||||||
|
admission_queue_capacity
|
||||||
|
persistence_concurrency
|
||||||
|
shutdown_drain_timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
Bornes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission queue : 1..=65_536, default 256
|
||||||
|
persistence concurrency : 1..=64, default 8
|
||||||
|
shutdown drain : 100 ms..=30 s, default 5 s
|
||||||
|
```
|
||||||
|
|
||||||
|
### Types reportés
|
||||||
|
|
||||||
|
Ne sont pas figés publiquement avant une vraie source :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionSourceId
|
||||||
|
SourceSettings
|
||||||
|
SourceCapability
|
||||||
|
SourceRole
|
||||||
|
ContinuityState
|
||||||
|
run/source frontiers
|
||||||
|
provider-specific settings
|
||||||
|
retry/replay/gap repair
|
||||||
|
```
|
||||||
|
|
||||||
|
### Runtime
|
||||||
|
|
||||||
|
```text
|
||||||
|
caller-owned active Tokio runtime
|
||||||
|
Worker-owned private supervisor/tasks
|
||||||
|
JoinHandle/JoinSet privés
|
||||||
|
start(settings, Arc<Store>) -> Handle
|
||||||
|
request_stop idempotent
|
||||||
|
wait_terminal async sans Tokio nominal public
|
||||||
|
mpsc admission borné
|
||||||
|
watch stop privé
|
||||||
|
watch snapshot latest-value
|
||||||
|
aucun unbounded channel
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pipeline
|
||||||
|
|
||||||
|
```text
|
||||||
|
private deterministic ingress
|
||||||
|
-> ksp-raw-transaction-lib canonicalization
|
||||||
|
-> Worker-owned deterministic observation key/provenance
|
||||||
|
-> common RawTransactionAcquisition
|
||||||
|
-> ksp-store-lib atomic Normal persistence
|
||||||
|
-> explicit outcomes/counters
|
||||||
|
-> concrete/common latest-value snapshot
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun cache mémoire n'est utilisé comme vérité de correction ; Store décide new/idempotent/conflict.
|
||||||
|
|
||||||
|
### Fault/shutdown
|
||||||
|
|
||||||
|
```text
|
||||||
|
content conflict -> terminal fault, no overwrite, no winner
|
||||||
|
Store error -> terminal safe fault
|
||||||
|
source harness failure -> terminal safe fault
|
||||||
|
stop -> close new admissions + bounded drain admitted work
|
||||||
|
expiry drain -> abort private remaining tasks + join + terminal drain_timeout
|
||||||
|
terminal publié seulement après absence de tâche privée survivante
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fraîcheur externe
|
||||||
|
|
||||||
|
Vérifié le 8 septembre 2026 :
|
||||||
|
|
||||||
|
```text
|
||||||
|
tokio latest stable = 1.53.1 ; workspace = ^1.53
|
||||||
|
futures-util latest stable = 0.3.34 ; workspace = ^0.3
|
||||||
|
sha2 latest stable = 0.11.0 ; workspace = ^0.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun bump workspace n'est requis.
|
||||||
|
|
||||||
|
Dépendances prévues pour la crate foundation lorsqu'elle est créée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-core-lib
|
||||||
|
ksp-worker-api
|
||||||
|
ksp-raw-transaction-lib
|
||||||
|
ksp-store-lib default-features=false
|
||||||
|
ksp-logging-lib
|
||||||
|
sha2
|
||||||
|
tokio features macros,rt,sync,time
|
||||||
|
```
|
||||||
|
|
||||||
|
Non retenus en fondation :
|
||||||
|
|
||||||
|
```text
|
||||||
|
futures-util : pas d'usage nécessaire après choix JoinSet/mpsc/watch/select!
|
||||||
|
ksp-onchain-transport-lib : aucune source live en 0.3.11
|
||||||
|
ksp-config-lib : edge interdit
|
||||||
|
SDK provider : interdit/inutile
|
||||||
|
```
|
||||||
|
|
||||||
|
## Harness et threat model
|
||||||
|
|
||||||
|
Le plan `032` fixe un harness privé sans réseau couvrant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
start/stop
|
||||||
|
source failure
|
||||||
|
queue saturation/backpressure
|
||||||
|
bounded persistence concurrency
|
||||||
|
new/idempotent/new observation
|
||||||
|
content conflict
|
||||||
|
Store failure
|
||||||
|
stop pendant admission/persistence
|
||||||
|
drain timeout
|
||||||
|
no orphan tasks
|
||||||
|
slow/no snapshot listener
|
||||||
|
latest-value concrete/common
|
||||||
|
redaction et absence de surface Job/Config/provider
|
||||||
|
```
|
||||||
|
|
||||||
|
## Sizing recalibré
|
||||||
|
|
||||||
|
Le forecast initial est scindé pour respecter le budget de 15–20 minutes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
pre.002 crate + dependency firewall
|
||||||
|
pre.003 identity/settings
|
||||||
|
pre.004 lifecycle/start/handle/terminal
|
||||||
|
pre.005 supervisor/task ownership + harness
|
||||||
|
pre.006 bounded admission + common RAW
|
||||||
|
pre.007 Store persistence/idempotence/conflict
|
||||||
|
pre.008 snapshots + Worker API projection
|
||||||
|
pre.009 shutdown/backpressure/fault hardening
|
||||||
|
pre.010 public/release/security hardening
|
||||||
|
pre.011 gate technique final
|
||||||
|
pre.012 réconciliation documentaire
|
||||||
|
pre.013 préparation publication
|
||||||
|
rel.001
|
||||||
|
```
|
||||||
|
|
||||||
|
Décision : `0.3.11` reste clôturable dans une seule session **à condition** de ne pas introduire source live, Transport productif, Config, migration Store ou continuity/gap repair. Toute tranche réellement trop grosse est scindée avant exécution.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
deltas/0.3.11/pre.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/rules/RULES_KSP.md
|
||||||
|
docs/rules/RULES_DEPENDENCIES.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Versions :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml : 505 -> 506
|
||||||
|
workspace.package.version : 0.3.10 -> 0.3.11-pre.1
|
||||||
|
RULES_KSP.md : 40 -> 41
|
||||||
|
RULES_DEPENDENCIES.md : 18 -> 19
|
||||||
|
```
|
||||||
|
|
||||||
|
Le bump Cargo est requis par `VER-ID-009` pour une prerelease non-fix, même sans modification Rust.
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation après modification
|
||||||
|
|
||||||
|
Audits statiques réellement exécutés sur le delta complet :
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 775 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
Un scan sémantique complémentaire des règles actives confirme l'absence des trois formulations obsolètes corrigées. La comparaison byte-à-byte contre la stable confirme exactement 3 fichiers ajoutés, 3 fichiers modifiés et aucun fichier supprimé.
|
||||||
|
|
||||||
|
## Preuve opérateur stable distincte
|
||||||
|
|
||||||
|
Le journal opérateur fourni pour `0.3.10` montre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check : sans erreur
|
||||||
|
cargo check --workspace : sans erreur
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings : sans erreur
|
||||||
|
ksp-raw-transaction-lib : 26 tests verts
|
||||||
|
ksp-job-backfill-lib : 77 tests verts
|
||||||
|
cargo tree normal des deux crates : fourni
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette preuve concerne la stable d'entrée ; elle ne devient pas une exécution locale de `0.3.11-pre.001`.
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test --workspace --all-targets --all-features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
Motif : `cargo` n'est pas installé dans l'environnement d'assemblage. `pre.001` ne contient aucun Rust Worker, mais l'opérateur doit rejouer les gates applicables après application du delta.
|
||||||
|
|
||||||
|
## Questions fermées
|
||||||
|
|
||||||
|
```text
|
||||||
|
settings techniques et bornes V1
|
||||||
|
public/private surface de fondation
|
||||||
|
absence de SourceCapability/Role publics avant 0.3.12
|
||||||
|
start + Arc<Store> + handle
|
||||||
|
terminal future runtime-neutral
|
||||||
|
supervisor/task ownership privé
|
||||||
|
mpsc/watch exacts
|
||||||
|
harness déterministe sans réseau
|
||||||
|
absence de frontier/timestamp public en fondation
|
||||||
|
Store conflict/failure policy
|
||||||
|
Tokio direct + features minimales retenues
|
||||||
|
absence de futures-util/Transport en foundation
|
||||||
|
sizing et maintien de la release
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions reportées
|
||||||
|
|
||||||
|
```text
|
||||||
|
premier contrat public SourceId/capability/role
|
||||||
|
Yellowstone source settings/adapters
|
||||||
|
source required/optional policy
|
||||||
|
retry/reconnect
|
||||||
|
run frontier/continuity
|
||||||
|
gap repair
|
||||||
|
hydration productive
|
||||||
|
WS/Helius/HTTP live
|
||||||
|
multi-provider coalescence
|
||||||
|
hot reconfiguration desired/effective
|
||||||
|
provider smokes
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune question reportée ne bloque `pre.002`.
|
||||||
|
|
||||||
|
## Non inclus
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune nouvelle crate Worker créée
|
||||||
|
aucun Rust modifié
|
||||||
|
aucun Transport/Config/Store code modifié
|
||||||
|
aucune migration
|
||||||
|
aucune source live
|
||||||
|
aucun CHANGELOG/ROADMAP modifié
|
||||||
|
aucun prompt 0.3.12 produit
|
||||||
|
```
|
||||||
164
deltas/0.3.11/pre.002.md
Normal file
164
deltas/0.3.11/pre.002.md
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.002` — squelette Worker RAW + dependency firewall
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.001
|
||||||
|
workspace.package.version = 0.3.11-pre.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour `pre.001` est vert sur `cargo fmt --all`, les audits Rust/Markdown, `cargo check --workspace` et Clippy workspace strict `--all-targets --all-features -- -D warnings`. Aucun `cargo test` supplémentaire n'a été communiqué pour ce gate et aucun n'est inventé.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Créer uniquement le squelette de `ksp-worker-raw-transaction-ingest-lib`, l'inscrire au workspace et verrouiller son graphe de dépendances. Aucun settings, identity, `start`, task, channel, Store write, canonicalisation ou snapshot runtime n'est matérialisé dans cette tranche.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Cette tranche est une prerelease non-fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.11-pre.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le `Cargo.toml` racine passe du header `506` au header `507` et le workspace passe de 20 à 21 membres.
|
||||||
|
|
||||||
|
## Graphe de production exact
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
-> ksp-core-lib
|
||||||
|
-> ksp-logging-lib
|
||||||
|
-> ksp-raw-transaction-lib
|
||||||
|
-> ksp-store-lib default-features=false
|
||||||
|
-> ksp-worker-api
|
||||||
|
-> sha2
|
||||||
|
-> tokio features=macros,rt,sync,time
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces edges matérialisent uniquement la frontière décidée en `pre.001`. Le code runtime qui les consommera reste réservé à ses tranches dédiées.
|
||||||
|
|
||||||
|
## Firewall explicite
|
||||||
|
|
||||||
|
Absents du manifest de production :
|
||||||
|
|
||||||
|
```text
|
||||||
|
futures-util
|
||||||
|
ksp-config-lib
|
||||||
|
ksp-interface-lib
|
||||||
|
ksp-job-api
|
||||||
|
ksp-job-backfill-lib
|
||||||
|
ksp-onchain-transport-lib
|
||||||
|
ksp-program-api
|
||||||
|
ksp-store-api direct
|
||||||
|
ksp-store-postgres-lib
|
||||||
|
ksp-wallet-lib
|
||||||
|
reqwest
|
||||||
|
Solana protocol SDK direct
|
||||||
|
tokio-tungstenite
|
||||||
|
tonic
|
||||||
|
yellowstone-grpc-proto
|
||||||
|
```
|
||||||
|
|
||||||
|
Il n'existe ni `[dev-dependencies]`, ni `[build-dependencies]` dans cette tranche.
|
||||||
|
|
||||||
|
## Crate-root volontairement vide de comportement
|
||||||
|
|
||||||
|
`src/lib.rs` contient uniquement les lints workspace et la rustdoc de responsabilité. Il n'expose encore :
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun pub mod
|
||||||
|
aucun type public
|
||||||
|
aucune fonction publique
|
||||||
|
aucun start/spawn
|
||||||
|
aucun appel Tokio
|
||||||
|
aucun appel Store/common RAW/Worker API
|
||||||
|
```
|
||||||
|
|
||||||
|
Le test externe `dependency_boundary.rs` verrouille le manifest exact, les edges interdits et l'absence de comportement runtime prématuré au crate-root.
|
||||||
|
|
||||||
|
## Documentation volontairement différée
|
||||||
|
|
||||||
|
`README.md` et `USAGE.md` ne sont pas créés à ce stade car la crate ne possède encore aucune API consommable. Leur réconciliation version-neutral reste réservée à `pre.012`, après freeze de la fondation runtime.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
deltas/0.3.11/pre.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 776 file(s))
|
||||||
|
|
||||||
|
contrôle TOML/statique : workspace 0.3.11-pre.2, 21 membres, manifest Worker exact, Store sans default features, Tokio macros/rt/sync/time
|
||||||
|
comparaison byte-à-byte avec pre.001 : 4 fichiers ajoutés, 3 modifiés, 0 supprimé
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions conservées
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune source live dans 0.3.11
|
||||||
|
aucun Transport productif avant 0.3.12
|
||||||
|
Store uniquement via ksp-store-lib default-features=false
|
||||||
|
aucun backend Store direct
|
||||||
|
aucun Job/Backfill/Config dans le Worker
|
||||||
|
futures-util absent tant qu'un usage réel n'est pas démontré
|
||||||
|
Tokio direct parce que le Worker concret possédera ses tâches runtime
|
||||||
|
aucune API publique avant la tranche qui en possède le contrat
|
||||||
|
```
|
||||||
|
|
||||||
|
## Suite
|
||||||
|
|
||||||
|
Après gate opérateur vert, `pre.003` introduit uniquement l'identité Worker concrète et `RawTransactionIngestSettings` avec defaults, bornes et validation. Aucun spawn n'y est autorisé.
|
||||||
100
deltas/0.3.11/pre.003-fix.001.md
Normal file
100
deltas/0.3.11/pre.003-fix.001.md
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.003-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.003-fix.001` — propagation d'erreur explicite des settings
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.003
|
||||||
|
workspace.package.version = 0.3.11-pre.3
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent, mais `cargo clippy --workspace --all-targets --all-features -- -D warnings` rejette deux usages de l'opérateur `?` dans `RawTransactionIngestSettings::new`.
|
||||||
|
|
||||||
|
Diagnostics :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs:42:9
|
||||||
|
clippy::question_mark_used
|
||||||
|
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs:48:9
|
||||||
|
clippy::question_mark_used
|
||||||
|
```
|
||||||
|
|
||||||
|
La règle normative `RUST-ERR-002` interdit également l'opérateur `?` en code de production. Les tests de la crate communiqués dans le même journal restent verts : 4 tests unitaires, 3 tests dependency-boundary et 3 tests public API, sans échec.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger strictement `pre.003` sans modifier son contrat fonctionnel : remplacer les deux propagations `?` par un contrôle de flux explicite retournant exactement la même erreur. Aucun type, borne, default, code d'erreur, dépendance, test, lifecycle ou comportement runtime n'est ajouté.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de production. Conformément à `VER-ID-007` et `VER-ID-010` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.003-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.3.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Dans `RawTransactionIngestSettings::new`, les validations de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission_queue_capacity
|
||||||
|
persistence_concurrency
|
||||||
|
```
|
||||||
|
|
||||||
|
utilisent désormais `if let Result::Err(error)` puis `return Err(error)`. La validation de `shutdown_drain_timeout` était déjà explicite et reste inchangée.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.003-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : aucun opérateur `?` restant dans settings.rs
|
||||||
|
scan statique : aucune modification de surface publique/settings hors propagation d'erreur
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.004` reste bloquée jusqu'à validation opérateur verte de ce fix. Aucun périmètre de `pre.003` n'est redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
185
deltas/0.3.11/pre.003.md
Normal file
185
deltas/0.3.11/pre.003.md
Normal file
@@ -0,0 +1,185 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.003.md -->
|
||||||
|
<!-- version: 3 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.003` — identité Worker + settings bornés
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.002
|
||||||
|
workspace.package.version = 0.3.11-pre.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour `pre.002` est entièrement vert : `cargo fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, tests de la crate et arbres Cargo normal/features. Les 3 tests `pre.002` passent et le firewall de dépendances reste conforme.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement l'identité de la verticale Worker RAW transaction ingest et ses settings runtime techniques bornés. Aucun lifecycle, `start`, spawn, task, channel, supervisor, persistence Store, canonicalisation ou source live n'est introduit.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Cette tranche est une prerelease non-fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.11-pre.3
|
||||||
|
```
|
||||||
|
|
||||||
|
## Identité Worker
|
||||||
|
|
||||||
|
La verticale possède le code stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RAW_TRANSACTION_INGEST_WORKER_KIND_CODE = "raw_transaction_ingest"
|
||||||
|
```
|
||||||
|
|
||||||
|
La transformation de ce texte en `WorkerKindCode` reste sous validation de `ksp-worker-api`; aucun second validateur d'identité n'est créé dans la crate concrète.
|
||||||
|
|
||||||
|
## `RawTransactionIngestSettings`
|
||||||
|
|
||||||
|
Surface exacte :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network: ksp_store_lib::RawNetworkId
|
||||||
|
worker_id: ksp_worker_api::WorkerId
|
||||||
|
admission_queue_capacity: usize
|
||||||
|
persistence_concurrency: usize
|
||||||
|
shutdown_drain_timeout: Duration
|
||||||
|
```
|
||||||
|
|
||||||
|
Les champs sont privés et exposés par getters. Deux constructions existent : paramètres explicites bornés et defaults V1 sur des identités déjà validées.
|
||||||
|
|
||||||
|
Bornes exactes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission queue : 1..=65_536, default 256
|
||||||
|
persistence concurrency : 1..=64, default 8
|
||||||
|
shutdown drain : 100 ms..=30 s, default 5 s
|
||||||
|
```
|
||||||
|
|
||||||
|
Les bornes sont inclusives. Aucun champ historique, provider-specific, retry, replay, gap repair, source capability ou frontier n'est ajouté.
|
||||||
|
|
||||||
|
## Erreur settings
|
||||||
|
|
||||||
|
La crate ouvre un seul code propre à cette tranche :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.settings_invalid
|
||||||
|
```
|
||||||
|
|
||||||
|
Il couvre uniquement les limites techniques possédées par le Worker concret. Le contexte d'erreur contient seulement le nom statique du champ invalide :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission_queue_capacity
|
||||||
|
persistence_concurrency
|
||||||
|
shutdown_drain_timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune valeur caller-supplied n'est copiée dans le diagnostic.
|
||||||
|
|
||||||
|
Les erreurs de `RawNetworkId`, `WorkerId` et `WorkerKindCode` restent celles de leurs crates propriétaires.
|
||||||
|
|
||||||
|
## Redaction
|
||||||
|
|
||||||
|
`Debug` du settings rend le network logique mais conserve le `WorkerId(..)` redacted fourni par `ksp-worker-api`; le texte réel du Worker ID n'est pas rendu.
|
||||||
|
|
||||||
|
## Frontière runtime maintenue
|
||||||
|
|
||||||
|
Toujours absents :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestWorker
|
||||||
|
start/spawn
|
||||||
|
JoinHandle/JoinSet
|
||||||
|
mpsc/watch
|
||||||
|
supervisor
|
||||||
|
Store write
|
||||||
|
common RAW conversion
|
||||||
|
Transport
|
||||||
|
source live/provider SDK
|
||||||
|
```
|
||||||
|
|
||||||
|
Le manifest de production est inchangé depuis `pre.002`.
|
||||||
|
|
||||||
|
## Tests ajoutés/étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
unit_tests/settings.rs
|
||||||
|
defaults exacts
|
||||||
|
bornes min/max inclusives
|
||||||
|
six cas hors bornes
|
||||||
|
code/contexte d'erreur stable
|
||||||
|
redaction Debug du WorkerId
|
||||||
|
|
||||||
|
tests/public_api.rs
|
||||||
|
kind + settings consommables depuis crate root
|
||||||
|
validation network/Worker identity reste lower-layer-owned
|
||||||
|
erreur de borne publique stable et sans echo d'identité
|
||||||
|
|
||||||
|
tests/dependency_boundary.rs
|
||||||
|
firewall pre.002 conservé
|
||||||
|
crate-root pre.003 limité à identity/settings
|
||||||
|
absence de comportement runtime prématuré
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/identity.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/settings.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
deltas/0.3.11/pre.003.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Premier passage : quatre écarts de forme détectés par l'audit Rust (rustdoc `pub(crate)`, ordre de constantes, ligne vide de fonction), puis un espacement Markdown documentaire détecté après ajout de la validation. Ils ont tous été corrigés avant livraison.
|
||||||
|
|
||||||
|
Gate final local :
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 777 file(s))
|
||||||
|
|
||||||
|
contrôle statique runtime-boundary : clean
|
||||||
|
comparaison byte-à-byte avec pre.002 : 6 fichiers ajoutés, 5 modifiés, 0 supprimé
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Suite
|
||||||
|
|
||||||
|
Après gate opérateur vert, `pre.004` introduit le `start` caller-runtime-owned, le handle concret, le lifecycle/terminal et le stop idempotent. Aucun supervisor complet ni persistence réelle ne doit être anticipé avant leurs tranches dédiées.
|
||||||
135
deltas/0.3.11/pre.004.md
Normal file
135
deltas/0.3.11/pre.004.md
Normal file
@@ -0,0 +1,135 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.004.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.004` — start, handle, lifecycle et terminal Worker
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.003-fix.001
|
||||||
|
workspace.package.version = 0.3.11-pre.3.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert après le fix `pre.003-fix.001` sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict et les dix tests de la crate Worker.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.004` du plan `032` : start synchrone sur runtime Tokio caller-owned, handle clonable, stop idempotent, lifecycle générique `ksp-worker-api` et future terminal boxed, sans ouvrir le supervisor de tâches enfants, l'admission, la canonicalisation, la persistence, les snapshots concrets ou une source live.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.004
|
||||||
|
workspace.package.version : 0.3.11-pre.4
|
||||||
|
```
|
||||||
|
|
||||||
|
## Contrat public ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID
|
||||||
|
RawTransactionIngestWorker
|
||||||
|
RawTransactionIngestHandle
|
||||||
|
RawTransactionIngestTerminalFuture
|
||||||
|
RawTransactionIngestWorker::start(settings, Arc<Store>)
|
||||||
|
RawTransactionIngestHandle::request_stop()
|
||||||
|
RawTransactionIngestHandle::wait_terminal()
|
||||||
|
```
|
||||||
|
|
||||||
|
`start` exige un runtime Tokio courant, vérifie le réseau du Store avant spawn et conserve le Store par `Arc` sans le fermer ni l'utiliser encore pour une écriture.
|
||||||
|
|
||||||
|
`request_stop()` partage `WorkerStopToken` et retourne `true` uniquement lors de la première demande acceptée. Le task racine est réveillé par un `watch<bool>` privé ; la disparition de tous les handles ferme ce channel et provoque également une terminaison coopérative.
|
||||||
|
|
||||||
|
`wait_terminal()` retourne une future boxed `Send`. Elle attend un `WorkerState` terminal puis la fermeture du sender d'état par le task racine ; aucun `JoinHandle` public n'est nécessaire.
|
||||||
|
|
||||||
|
## Lifecycle de fondation
|
||||||
|
|
||||||
|
```text
|
||||||
|
Starting -> Running -> Stopping -> Stopped
|
||||||
|
Starting -> Stopping -> Stopped si stop immédiat
|
||||||
|
```
|
||||||
|
|
||||||
|
Les transitions utilisent `WorkerLifecycle`. Un invariant impossible devient `Faulted(worker_raw_transaction_ingest.runtime_invalid)` sans payload, worker id, réseau ou erreur externe dans le diagnostic.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
JoinSet / tasks enfants
|
||||||
|
mpsc / admission
|
||||||
|
canonicalisation common RAW
|
||||||
|
observation key
|
||||||
|
persistence Store
|
||||||
|
snapshots concrets / WorkerSnapshotSource
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
```
|
||||||
|
|
||||||
|
Les méthodes de snapshot prévues par la surface finale restent différées à `pre.008`. `pre.005` introduit le supervisor et l'ownership des tâches enfants autour du wake-up de stop déjà minimalement nécessaire ici.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
Les tests couvrent statiquement ou via harness runtime privé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
runtime Tokio courant obligatoire
|
||||||
|
network Store mismatch sûr
|
||||||
|
stop immédiat idempotent
|
||||||
|
Running -> stop via clone de handle
|
||||||
|
drop du dernier handle -> sortie du task racine
|
||||||
|
surface publique start/handle/terminal future
|
||||||
|
absence de JoinHandle public et du scope pre.005+
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
deltas/0.3.11/pre.004.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique des interdits/scope runtime
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.004` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.005` reste bloquée jusqu'à validation opérateur verte de cette tranche.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Le supervisor/tasks enfants, l'admission, la persistence et les snapshots restent dans leurs tranches planifiées.
|
||||||
125
deltas/0.3.11/pre.005.md
Normal file
125
deltas/0.3.11/pre.005.md
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.005.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.005` — supervisor privé et ownership des tâches
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.004
|
||||||
|
workspace.package.version = 0.3.11-pre.4
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, les 16 tests de la crate Worker, doc-tests et les arbres Cargo normal/features.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.005` du plan `032` : supervisor privé, `JoinSet` privé, ownership et join des tâches enfants, intégration du stop au seam source déterministe et preuve qu'aucune tâche enfant coopérative ne survit au terminal. Aucun admission `mpsc`, aucune canonicalisation RAW et aucune persistence Store réelle.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.005
|
||||||
|
workspace.package.version : 0.3.11-pre.5
|
||||||
|
```
|
||||||
|
|
||||||
|
## Runtime privé ajouté
|
||||||
|
|
||||||
|
Le task racine devient le supervisor du run. Il possède un `tokio::task::JoinSet<()>` privé et un seam privé de spawn de tâches source.
|
||||||
|
|
||||||
|
La production appelle ce seam sans source. Les tests injectent des tâches déterministes qui reçoivent un clone du `watch<bool>` de stop.
|
||||||
|
|
||||||
|
Le supervisor :
|
||||||
|
|
||||||
|
```text
|
||||||
|
reap des enfants terminés via join_next()
|
||||||
|
stop prioritaire via select! biased
|
||||||
|
join de tous les enfants restants après stop/fermeture du contrôle
|
||||||
|
Stopped publié uniquement après join complet
|
||||||
|
JoinError -> supervision non clean ; Faulted à la fermeture de supervision
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun `JoinHandle` ou `JoinSet` n'est ajouté à la surface publique.
|
||||||
|
|
||||||
|
## Arrêt coopératif
|
||||||
|
|
||||||
|
Le timeout de drain configuré existe déjà dans les settings mais n'est pas encore appliqué ici. `pre.005` attend coopérativement les enfants ; le timeout, `abort_all` et les races stop/fault sont réservés au hardening `pre.009`.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
mpsc / admission
|
||||||
|
PrivateRawTransactionIngress
|
||||||
|
canonicalisation common RAW
|
||||||
|
observation key
|
||||||
|
persistence Store
|
||||||
|
snapshots concrets / WorkerSnapshotSource
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
abort/timeout de drain
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
supervisor joint trois tâches source coopératives avant Stopped
|
||||||
|
compteur d'enfants actifs == 0 au terminal
|
||||||
|
reap d'une source terminée pendant Running
|
||||||
|
join d'une source encore active lors du stop
|
||||||
|
canary statique : JoinSet présent mais privé
|
||||||
|
canary statique : aucun scope pre.006+ ouvert
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.005.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique des interdits/scope runtime
|
||||||
|
Markdown table audit: clean (340 table(s), 780 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.005` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.006` reste bloquée jusqu'à validation opérateur verte de cette tranche.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. L'admission bornée et la canonicalisation common RAW restent dans `pre.006`, la persistence dans `pre.007`, les snapshots dans `pre.008` et le drain forcé/races dans `pre.009`.
|
||||||
97
deltas/0.3.11/pre.006-fix.001.md
Normal file
97
deltas/0.3.11/pre.006-fix.001.md
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.006-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.006-fix.001` — correction syntaxique du test `tokio::select!`
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.006
|
||||||
|
workspace.package.version = 0.3.11-pre.6
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent, mais que `cargo clippy --workspace --all-targets --all-features -- -D warnings` et `cargo test -p ksp-worker-raw-transaction-ingest-lib` ne peuvent compiler le test `unit_tests/admission.rs`.
|
||||||
|
|
||||||
|
Diagnostic exact :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/../unit_tests/admission.rs:152:13
|
||||||
|
error: no rules expected `result`
|
||||||
|
```
|
||||||
|
|
||||||
|
La cause est syntaxique : deux branches adjacentes du `tokio::select!` du test de préemption du sender bloqué n'étaient pas séparées par des virgules.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger strictement le test de `pre.006` sans modifier le contrat fonctionnel, le runtime de production, l'admission, la canonicalisation Common RAW, l'observation key, le golden ou les dépendances.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.006-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.6.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Le `tokio::select!` du test `pre_006_bounded_sender_is_preempted_by_stop_while_queue_is_full` sépare désormais explicitement ses branches :
|
||||||
|
|
||||||
|
```text
|
||||||
|
_changed = blocked_stop.changed() => false,
|
||||||
|
result = blocked_sender.send(blocked_ingress) => result.is_ok(),
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun comportement de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.006-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : le `tokio::select!` fautif contient deux branches séparées par des virgules
|
||||||
|
scan statique : aucun fichier de production de la crate Worker n'est modifié
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.007` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre de `pre.006` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
180
deltas/0.3.11/pre.006.md
Normal file
180
deltas/0.3.11/pre.006.md
Normal file
@@ -0,0 +1,180 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.006.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.006` — admission bornée et canonicalisation Common RAW
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.005
|
||||||
|
workspace.package.version = 0.3.11-pre.5
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, les 18 tests de la crate Worker, doc-tests et les arbres Cargo normal/features.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.006` du plan `032` : admission `mpsc` centrale bornée, backpressure async, ingress privé, canonicalisation via `ksp-raw-transaction-lib`, observation key Worker déterministe et assembly du `RawTransactionAcquisition`. Aucune persistence Store et aucune source réseau.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.006
|
||||||
|
workspace.package.version : 0.3.11-pre.6
|
||||||
|
```
|
||||||
|
|
||||||
|
## Admission privée bornée
|
||||||
|
|
||||||
|
Le supervisor crée un seul channel central dimensionné par `RawTransactionIngestSettings::admission_queue_capacity()` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
tokio::sync::mpsc::channel(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le seam source reçoit le sender `mpsc` borné et le signal de stop. Le contrat prouvé est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
send().await pour le backpressure
|
||||||
|
select! biased : stop avant send dans le harness source
|
||||||
|
aucun unbounded_channel
|
||||||
|
receiver close au shutdown
|
||||||
|
```
|
||||||
|
|
||||||
|
Au shutdown, le receiver ferme les nouvelles admissions et consume les entrées déjà en queue avant le join final des tâches enfants.
|
||||||
|
|
||||||
|
## Ingress et Common RAW
|
||||||
|
|
||||||
|
L'ingress crate-private, absent de l'API publique, transporte :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionMaterial
|
||||||
|
RawNetworkId attendu
|
||||||
|
RawAcquisitionProvenance
|
||||||
|
source key opaque [u8; 32]
|
||||||
|
```
|
||||||
|
|
||||||
|
Le pipeline est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network guard
|
||||||
|
ksp_raw_transaction_lib::canonicalize_raw_transaction(...)
|
||||||
|
material network guard
|
||||||
|
observation key Worker
|
||||||
|
ksp_raw_transaction_lib::assemble_raw_transaction_acquisition(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune canonicalisation RAW v1 n'est réimplémentée dans le Worker.
|
||||||
|
|
||||||
|
## Observation key V1
|
||||||
|
|
||||||
|
Domaine exact :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp.raw_transaction_ingest.observation.v1\0
|
||||||
|
```
|
||||||
|
|
||||||
|
Le SHA-256 encode avec longueur `u64` big-endian :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network
|
||||||
|
signature
|
||||||
|
source key
|
||||||
|
```
|
||||||
|
|
||||||
|
Golden harness :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network : mainnet
|
||||||
|
signature : 64 x 0x07
|
||||||
|
source key : 32 x 0x09
|
||||||
|
SHA-256 : f8d161dca8a8a72a0da110fc87882ee7136848461c6aa32b09e30a6188b95dcc
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun timestamp local, run id ou ordre d'admission ne participe à la clé.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
capacity=1 : second send bloqué jusqu'à consommation d'un slot
|
||||||
|
stop : sender bloqué rejeté sans admission post-stop
|
||||||
|
payload common RAW canonique exact
|
||||||
|
format id/version exacts
|
||||||
|
observation-key golden exact
|
||||||
|
assembly transaction/provenance/reference exact
|
||||||
|
mismatch network ingress/material rejetés sans fuite de valeurs
|
||||||
|
dependency boundary : mpsc/common RAW requis, persistence/Transport/snapshot toujours interdits
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
deltas/0.3.11/pre.006.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionWrite / persistence Store
|
||||||
|
RawTransactionAcquisitionMode::Normal
|
||||||
|
persistence concurrency
|
||||||
|
Store outcome mapping / idempotence / conflict
|
||||||
|
snapshots concrets
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
retry / reconnect / gap repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 table(s), 781 file(s))
|
||||||
|
|
||||||
|
scan production : aucun ?, unwrap, expect, panic, unbounded_channel
|
||||||
|
scan scope : aucune persistence Store, aucun Transport, aucun snapshot concret
|
||||||
|
pipeline : exactement 1 mpsc::channel, 1 canonicalize_raw_transaction, 1 assemble_raw_transaction_acquisition
|
||||||
|
observation golden indépendant : f8d161dca8a8a72a0da110fc87882ee7136848461c6aa32b09e30a6188b95dcc
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.006` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.007` reste bloquée jusqu'à validation opérateur verte de l'admission bornée, du backpressure, du golden observation et de l'assembly Common RAW.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. La persistence réelle et les outcomes Store restent dans `pre.007`, les snapshots dans `pre.008`, et le hardening fault/drain dans `pre.009`.
|
||||||
161
deltas/0.3.11/pre.007.md
Normal file
161
deltas/0.3.11/pre.007.md
Normal file
@@ -0,0 +1,161 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.007.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.007` — persistence Store bornée et outcomes déterministes
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.006-fix.001
|
||||||
|
workspace.package.version = 0.3.11-pre.6.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, les 22 tests de la crate Worker et les doc-tests. Aucun `cargo tree` n'a été rejoué après le fix, celui-ci n'ayant modifié aucune dépendance ni feature.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.007` du plan `032` : persistence atomique via la façade Store, mode `Normal`, concurrence bornée, classification entity/observation, idempotence/purge et terminaux dédiés content conflict/Store failure. Aucun snapshot concret et aucune source réseau.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.007
|
||||||
|
workspace.package.version : 0.3.11-pre.7
|
||||||
|
```
|
||||||
|
|
||||||
|
## Port Store privé
|
||||||
|
|
||||||
|
La crate ajoute un port `RawTransactionIngestPersistencePort` crate-private, implémenté par `ksp_store_lib::Store` et utilisé exclusivement avec :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionWrite::persist_raw_transaction_acquisition(..., RawTransactionAcquisitionMode::Normal)
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun backend Store concret n'entre dans le graphe ou dans le code Worker.
|
||||||
|
|
||||||
|
## Outcomes
|
||||||
|
|
||||||
|
Classification exacte :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Inserted + Inserted -> Inserted / Inserted
|
||||||
|
AlreadyPresent + Inserted -> AlreadyPresent / Inserted
|
||||||
|
AlreadyPresent + AlreadyPresent -> AlreadyPresent / AlreadyPresent
|
||||||
|
SkippedPurged + NotRecorded -> SkippedPurged / NotRecorded
|
||||||
|
```
|
||||||
|
|
||||||
|
`Rehydrated` et les couples impossibles deviennent `runtime_invalid`.
|
||||||
|
|
||||||
|
## Faults Store
|
||||||
|
|
||||||
|
Deux codes publics deviennent actifs :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.content_conflict
|
||||||
|
worker_raw_transaction_ingest.store_failed
|
||||||
|
```
|
||||||
|
|
||||||
|
Le conflit Store canonique est projeté sur `content_conflict`. Toute autre erreur Store est réduite à `store_failed` en conservant uniquement son domain/code sûr, jamais son texte arbitraire.
|
||||||
|
|
||||||
|
## Concurrence bornée
|
||||||
|
|
||||||
|
Le supervisor possède un `JoinSet` privé de persistences et n'admet une nouvelle acquisition que lorsque :
|
||||||
|
|
||||||
|
```text
|
||||||
|
in_flight_persistence < persistence_concurrency
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune queue secondaire ou persistence non bornée n'est introduite. Le harness bloque quatre writes avec une limite de deux et exige `max_active == 2`.
|
||||||
|
|
||||||
|
## Drain
|
||||||
|
|
||||||
|
Sur stop ou fault : nouvelles admissions fermées, acquisitions déjà admises drainées/persistées, persistences rejointes, sources rejointes, puis terminal. Le timeout/abort et la priorité complète des races terminales restent dans `pre.009`.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
mode Normal exact
|
||||||
|
new/idempotent/purged + observation distincte
|
||||||
|
Rehydrated/couples impossibles rejetés
|
||||||
|
network mismatch avant write
|
||||||
|
content conflict redacted
|
||||||
|
Store failure redacted
|
||||||
|
persistence concurrency réellement bornée
|
||||||
|
terminaux runtime content_conflict/store_failed
|
||||||
|
ErrorCode publics stables
|
||||||
|
dependency boundary persistence privée / backend direct interdit
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/persistence.rs
|
||||||
|
deltas/0.3.11/pre.007.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
snapshots concrets / compteurs publics
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
backend Store direct
|
||||||
|
ForceRehydrate
|
||||||
|
source_failed / counter_exhausted
|
||||||
|
drain timeout / abort forcé
|
||||||
|
retry / reconnect / gap repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean
|
||||||
|
|
||||||
|
scan production : aucun unwrap, expect, panic, unbounded_channel
|
||||||
|
scan scope : aucun backend Store direct, aucun Transport, aucun snapshot concret
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.007` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'a changé dans cette tranche ; aucun `cargo tree` intermédiaire n'est donc requis :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.008` reste bloquée jusqu'à validation opérateur verte de la persistence `Normal`, des outcomes/idempotences, de la concurrence bornée et des terminaux Store/conflict.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Les snapshots/counters restent dans `pre.008`; le hardening fault/shutdown reste dans `pre.009`.
|
||||||
100
deltas/0.3.11/pre.008-fix.001.md
Normal file
100
deltas/0.3.11/pre.008-fix.001.md
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.008-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.008-fix.001` — correction Clippy du traitement de publication snapshot
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.008
|
||||||
|
workspace.package.version = 0.3.11-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown, `cargo check --workspace` et tous les tests de `ksp-worker-raw-transaction-ingest-lib` passent, mais que `cargo clippy --workspace --all-targets --all-features -- -D warnings` rejette un `if` imbriqué dans `src/runtime.rs`.
|
||||||
|
|
||||||
|
Diagnostic exact :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs:176:17
|
||||||
|
clippy::collapsible_if
|
||||||
|
```
|
||||||
|
|
||||||
|
Le code traitait une erreur de publication snapshot puis testait séparément `fault.is_none()`. Clippy exige la forme conditionnelle fusionnée.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger strictement `pre.008` sans modifier son comportement : même erreur, même priorité du premier fault, mêmes snapshots, mêmes compteurs, même lifecycle et même runtime. Aucune dépendance, feature ou surface publique n'est modifiée.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de production. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.008-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.8.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Le traitement fautif devient la forme équivalente acceptée par Clippy :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
if let Result::Err(error) = published
|
||||||
|
&& fault.is_none()
|
||||||
|
{
|
||||||
|
fault = Option::Some(error.code());
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
La sémantique reste identique : une erreur de publication snapshot ne remplace jamais un fault déjà fixé.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.008-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : le nested-if signalé n'existe plus
|
||||||
|
scan statique : aucune dépendance/feature ni surface publique modifiée
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'étant modifiée, aucun `cargo tree` n'est requis pour ce fix.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.009` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre de `pre.008` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
167
deltas/0.3.11/pre.008.md
Normal file
167
deltas/0.3.11/pre.008.md
Normal file
@@ -0,0 +1,167 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.008.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.008` — snapshots latest-value et projection Worker API
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.007
|
||||||
|
workspace.package.version = 0.3.11-pre.7
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 31 tests de la crate Worker et les doc-tests. Aucun `cargo tree` intermédiaire n'était requis : `pre.007` n'avait modifié aucune dépendance ni feature.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.008` du plan `032` : snapshot concret latest-value, compteurs checked, projection directe vers `ksp-worker-api::WorkerSnapshotSource`, coalescence des listeners et terminal retenu. Aucun hardening timeout/abort et aucune source réseau.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.008
|
||||||
|
workspace.package.version : 0.3.11-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
## Snapshot concret
|
||||||
|
|
||||||
|
La crate expose :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestSnapshot
|
||||||
|
RawTransactionIngestSnapshotFuture
|
||||||
|
RawTransactionIngestSnapshotSource
|
||||||
|
```
|
||||||
|
|
||||||
|
Le snapshot contient uniquement la projection `WorkerSnapshot`, les capacités/profondeurs techniques bornées et les compteurs sûrs prévus par le plan.
|
||||||
|
|
||||||
|
## Un seul flux latest-value
|
||||||
|
|
||||||
|
Un unique :
|
||||||
|
|
||||||
|
```text
|
||||||
|
tokio::sync::watch::channel<RawTransactionIngestSnapshot>
|
||||||
|
```
|
||||||
|
|
||||||
|
porte l'état observable du run. Le watch terminal spécialisé des tranches précédentes disparaît : `wait_terminal()`, `snapshot_source()` et `worker_snapshot_source()` lisent tous ce même flux.
|
||||||
|
|
||||||
|
`RawTransactionIngestSnapshotSource` implémente directement `ksp_worker_api::WorkerSnapshotSource`. La sequence `WorkerSnapshotSequence` est donc identique pour la vue concrète et la projection commune.
|
||||||
|
|
||||||
|
## Compteurs checked
|
||||||
|
|
||||||
|
Les compteurs `u64` effectivement incrémentés dans cette tranche utilisent `checked_add(1)`. Aucun wrap silencieux n'est admis. Le nouveau code public est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.counter_exhausted
|
||||||
|
```
|
||||||
|
|
||||||
|
Les compteurs sont mis à jour au point d'ownership du supervisor : dequeue/canonicalisation et récolte de persistence. `admission_queue_depth` provient directement du receiver `mpsc` borné ; `in_flight_persistence` provient du `JoinSet` privé existant.
|
||||||
|
|
||||||
|
`source_failure_total` et `backpressure_wait_total` font partie du snapshot sûr mais restent à zéro en `pre.008`; leur comportement effectif appartient à `pre.009`.
|
||||||
|
|
||||||
|
## Projection common Worker
|
||||||
|
|
||||||
|
Mapping de fondation :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Starting -> health Unknown
|
||||||
|
Running -> health Healthy
|
||||||
|
Stopping / Stopped -> conserve le dernier health non terminal
|
||||||
|
Faulted -> health Unhealthy
|
||||||
|
queue > 0 ou persistence > 0 -> activity Active
|
||||||
|
sinon Running/Stopping/terminal -> activity Idle
|
||||||
|
```
|
||||||
|
|
||||||
|
La state machine reste celle de `ksp-worker-api::WorkerLifecycle`.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
snapshot initial exact
|
||||||
|
projection common/concrete même sequence
|
||||||
|
counter exhaustion stable sans wrap
|
||||||
|
listener lent coalescé vers la dernière valeur
|
||||||
|
terminal latest-value retenu après fermeture publisher
|
||||||
|
surface publique snapshot_source/worker_snapshot_source
|
||||||
|
WorkerSnapshotSource Send + Sync
|
||||||
|
pipeline runtime réussi -> counters exacts
|
||||||
|
Store failure -> health Unhealthy + store_failure_total
|
||||||
|
absence de watch terminal séparé
|
||||||
|
absence de nouveau dependency edge
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/snapshot.rs
|
||||||
|
deltas/0.3.11/pre.008.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
backend Store direct
|
||||||
|
source_failed public
|
||||||
|
source failure behavior
|
||||||
|
backpressure wait instrumentation productive
|
||||||
|
drain timeout / abort forcé
|
||||||
|
fault/stop precedence finalisée
|
||||||
|
retry / reconnect / gap repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean
|
||||||
|
|
||||||
|
scan production : aucun unwrap, expect, panic, unbounded_channel
|
||||||
|
scan scope : aucun backend Store direct, aucun Transport, aucune nouvelle dépendance
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.008` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'a changé dans cette tranche ; aucun `cargo tree` intermédiaire n'est requis :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.009` reste bloquée jusqu'à validation opérateur verte des snapshots common/concrete, des compteurs checked et du terminal latest-value retenu.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Le hardening shutdown/backpressure/fault races reste dans `pre.009`.
|
||||||
100
deltas/0.3.11/pre.009-fix.001.md
Normal file
100
deltas/0.3.11/pre.009-fix.001.md
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.009-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.009-fix.001` — correction de compilation des tests de hardening
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.009
|
||||||
|
workspace.package.version = 0.3.11-pre.9
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent, mais que `cargo clippy --workspace --all-targets --all-features -- -D warnings` et `cargo test -p ksp-worker-raw-transaction-ingest-lib` ne peuvent compiler deux tests ajoutés par `pre.009`.
|
||||||
|
|
||||||
|
Diagnostics exacts :
|
||||||
|
|
||||||
|
```text
|
||||||
|
unit_tests/admission.rs : appel à RawTransactionAdmission::receive sans expected_network et traitement Option au lieu de Result<Option<_>>
|
||||||
|
unit_tests/runtime.rs : type du paramètre children non inférable dans le closure source-failure
|
||||||
|
```
|
||||||
|
|
||||||
|
Le code de production compile ; les défauts sont strictement localisés dans les tests de `pre.009`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement les tests de hardening de `pre.009`, sans modifier le runtime de production, l'admission, la persistence, les snapshots, les terminaux, les compteurs, les dépendances ou les features.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche des fichiers Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.009-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.9.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modifications
|
||||||
|
|
||||||
|
Le test `pre_009_full_queue_dequeue_marks_source_neutral_backpressure_observation` conserve désormais une copie du `RawNetworkId`, appelle explicitement `receive(&network)` et vérifie la forme `Result<Option<_>>` avec `matches!`, sans `unwrap`, `expect` ni `?`.
|
||||||
|
|
||||||
|
Le test `pre_009_source_failure_is_counted_and_late_stop_cannot_replace_terminal_fault` annote explicitement le paramètre `children` du closure avec le type privé attendu par le harness :
|
||||||
|
|
||||||
|
```text
|
||||||
|
&mut tokio::task::JoinSet<ksp_core_lib::Result<()>>
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun fichier Rust de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.009-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : les deux appels receive utilisent expected_network et traitent Result<Option<_>>
|
||||||
|
scan statique : le closure source-failure annote explicitement JoinSet<ksp_core_lib::Result<()>>
|
||||||
|
scan statique : aucun fichier src/ de production ni manifest de crate Worker modifié
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'étant modifiée, aucun `cargo tree` n'est requis pour ce fix.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.010` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre fonctionnel de `pre.009` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
97
deltas/0.3.11/pre.009-fix.002.md
Normal file
97
deltas/0.3.11/pre.009-fix.002.md
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.009-fix.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.009-fix.002` — complément de compilation du test source-failure
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.009-fix.001
|
||||||
|
workspace.package.version = 0.3.11-pre.9.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent, mais que `cargo clippy --workspace --all-targets --all-features -- -D warnings` et `cargo test -p ksp-worker-raw-transaction-ingest-lib` ne peuvent compiler un test de hardening `pre.009`.
|
||||||
|
|
||||||
|
Diagnostic exact :
|
||||||
|
|
||||||
|
```text
|
||||||
|
unit_tests/runtime.rs : appel de start_foundation_with_source_spawner avec 3 arguments alors que le harness en exige 4 ; store_guard manque
|
||||||
|
```
|
||||||
|
|
||||||
|
Le code de production compile ; le défaut est strictement localisé dans le test source-failure.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement l'appel du harness dans ce test, sans modifier le runtime de production, l'admission, la persistence, les snapshots, les terminaux, les compteurs, les dépendances ou les features.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.009-fix.002
|
||||||
|
workspace.package.version : 0.3.11-pre.9.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Le test `pre_009_source_failure_is_counted_and_late_stop_cannot_replace_terminal_fault` fournit désormais explicitement le troisième argument du harness :
|
||||||
|
|
||||||
|
```text
|
||||||
|
std::option::Option::None
|
||||||
|
```
|
||||||
|
|
||||||
|
Ce test ne possède pas de Store réel et ne teste aucune persistence ; l'absence de `store_guard` est donc explicite et conforme au rôle du harness.
|
||||||
|
|
||||||
|
Aucun fichier Rust de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.009-fix.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : le test source-failure fournit explicitement store_guard = None
|
||||||
|
scan statique : aucun fichier src/ de production ni manifest de crate Worker modifié
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'étant modifiée, aucun `cargo tree` n'est requis pour ce fix.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.010` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre fonctionnel de `pre.009` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
96
deltas/0.3.11/pre.009-fix.003.md
Normal file
96
deltas/0.3.11/pre.009-fix.003.md
Normal file
@@ -0,0 +1,96 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.009-fix.003.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.009-fix.003` — conformité Clippy du test source-failure
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.009-fix.002
|
||||||
|
workspace.package.version = 0.3.11-pre.9.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown, `cargo check --workspace` et les 45 tests de `ksp-worker-raw-transaction-ingest-lib` passent. Le seul échec restant est le gate Clippy strict :
|
||||||
|
|
||||||
|
```text
|
||||||
|
clippy::implicit_return
|
||||||
|
unit_tests/runtime.rs : le bloc async de la source faultée retourne implicitement Result::Err(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le code de production et le comportement du Worker ne sont pas en cause.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Rendre le test source-failure conforme à la règle d'explicitation des retours, sans modifier le runtime de production, l'admission, la persistence, les snapshots, les terminaux, les compteurs, les dépendances ou les features.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.009-fix.003
|
||||||
|
workspace.package.version : 0.3.11-pre.9.fix.3
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Dans `pre_009_source_failure_is_counted_and_late_stop_cannot_replace_terminal_fault`, le future injecté dans le `JoinSet` utilise désormais un retour explicite :
|
||||||
|
|
||||||
|
```text
|
||||||
|
return Result::Err(...);
|
||||||
|
```
|
||||||
|
|
||||||
|
La valeur retournée, le code d'erreur injecté et toute la sémantique du test restent inchangés.
|
||||||
|
|
||||||
|
Aucun fichier Rust de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.009-fix.003.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : le bloc async source-failure contient un return explicite
|
||||||
|
scan statique : aucun fichier src/ de production ni manifest de crate Worker modifié
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'étant modifiée, aucun `cargo tree` n'est requis pour ce fix.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.010` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre fonctionnel de `pre.009` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
135
deltas/0.3.11/pre.009.md
Normal file
135
deltas/0.3.11/pre.009.md
Normal file
@@ -0,0 +1,135 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.009.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.009` — hardening shutdown, backpressure et races fault/stop
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.008-fix.001
|
||||||
|
workspace.package.version = 0.3.11-pre.8.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 38 tests de la crate Worker et les doc-tests. Aucune dépendance/feature n'ayant changé, aucun `cargo tree` intermédiaire n'était requis.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Fermer uniquement le hardening `pre.009` du plan `032` : faults source sûrs, saturation observable, ordering stop/fault déterministe, deadline de drain, abort+join au timeout et absence de tâches orphelines. Aucune source live n'est introduite.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.009
|
||||||
|
workspace.package.version : 0.3.11-pre.9
|
||||||
|
```
|
||||||
|
|
||||||
|
## Faults source
|
||||||
|
|
||||||
|
Les tâches source privées retournent `ksp_core_lib::Result<()>`. Une erreur source ou un `JoinError` devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.source_failed
|
||||||
|
```
|
||||||
|
|
||||||
|
Le détail distant/provider n'est pas propagé et `source_failure_total` est incrémenté via checked-add.
|
||||||
|
|
||||||
|
## Shutdown borné
|
||||||
|
|
||||||
|
Le drain complet est enveloppé par `shutdown_drain_timeout`. En cas de timeout :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission.close()
|
||||||
|
persistence.abort_all()
|
||||||
|
sources.abort_all()
|
||||||
|
join complet des persistences
|
||||||
|
join complet des sources
|
||||||
|
Faulted(worker_raw_transaction_ingest.drain_timeout)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le terminal n'est donc jamais publié avant la récolte de toutes les tâches possédées.
|
||||||
|
|
||||||
|
## Ordering stop/fault
|
||||||
|
|
||||||
|
Le stop reste prioritaire dans le `select!`, mais un Store fault découvert pendant le drain n'est pas masqué. Un fault terminal déjà décidé n'est pas remplacé par un stop tardif. Le timeout de drain prévaut lorsqu'il empêche la clôture coopérative.
|
||||||
|
|
||||||
|
## Backpressure source-neutral
|
||||||
|
|
||||||
|
L'admission conserve une unique queue `mpsc` bornée. Le dequeue constate si cette queue était pleine à sa capacité configurée et incrémente `backpressure_wait_total`. Aucun drop, aucune queue secondaire et aucun `unbounded_channel` ne sont ajoutés.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
source failure terminal + compteur
|
||||||
|
late stop ne remplace pas source_failed
|
||||||
|
stop puis Store fault pendant drain
|
||||||
|
saturation capacity=1 sans drop
|
||||||
|
backpressure_wait_total observable
|
||||||
|
drain timeout + abort/join des sources et persistences
|
||||||
|
aucune tâche active après terminal timeout
|
||||||
|
projection snapshot source/backpressure
|
||||||
|
error codes source_failed/drain_timeout publics
|
||||||
|
dependency firewall inchangé
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.009.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/snapshot.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'est ajoutée, supprimée ou modifiée. Le `Cargo.toml` de la crate Worker reste identique à la base.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique des frontières runtime/dependencies
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.009` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'ayant changé, aucun `cargo tree` intermédiaire n'est requis :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.010` reste bloquée jusqu'à validation opérateur verte du hardening shutdown/backpressure/fault races.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Les sources live restent explicitement hors scope de `0.3.11`.
|
||||||
78
deltas/0.3.11/pre.010-fix.001.md
Normal file
78
deltas/0.3.11/pre.010-fix.001.md
Normal file
@@ -0,0 +1,78 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.010-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.010-fix.001` — correction du shadowing dans la canarie hardening
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.010
|
||||||
|
workspace.package.version = 0.3.11-pre.10
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent. Le gate Clippy/compilation des tests externes échoue sur `tests/hardening.rs` avec `E0618` : la variable locale `worker_id` masque la fonction helper `worker_id(...)`, puis le second appel tente d'appeler la valeur `WorkerId` comme une fonction.
|
||||||
|
|
||||||
|
Le code de production, les dépendances et le comportement du Worker ne sont pas en cause.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement cette collision de nom dans la canarie `pre.010`, sans modifier la surface publique, le runtime, les dépendances, les features ou les règles de hardening.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.010-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.10.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Dans `pre_010_debug_and_settings_errors_redact_worker_identity_and_invalid_values`, la variable locale issue du premier parse est renommée de `worker_id` vers `parsed_worker_id`. La fonction helper `worker_id(...)` reste ainsi appelable pour le second identifiant de test.
|
||||||
|
|
||||||
|
Aucun fichier Rust de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.010-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'est ajoutée, supprimée ou modifiée. Le `Cargo.toml` de la crate Worker reste identique à `pre.010`.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucun `cargo tree` n'est requis pour ce fix :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.011` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre fonctionnel de `pre.010` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
126
deltas/0.3.11/pre.010.md
Normal file
126
deltas/0.3.11/pre.010.md
Normal file
@@ -0,0 +1,126 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.010.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.010` — hardening public, release et sécurité
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.009-fix.003
|
||||||
|
workspace.package.version = 0.3.11-pre.9.fix.3
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 45 tests de la crate Worker et les doc-tests. Aucune dépendance/feature n'ayant changé, aucun `cargo tree` intermédiaire n'était requis.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Fermer uniquement le hardening externe `pre.010` du plan `032` : surface publique crate-root exacte, dépendances exactes, absence de surface historique, redaction, codes d'erreur, inventaire des modules et scans Config/secret/backend/Transport. Aucun comportement runtime n'est ajouté.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.010
|
||||||
|
workspace.package.version : 0.3.11-pre.10
|
||||||
|
```
|
||||||
|
|
||||||
|
## Hardening externe
|
||||||
|
|
||||||
|
Nouvelle suite `tests/hardening.rs` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
7 ErrorCode publics uniques et domain-scoped
|
||||||
|
WorkerId redacted dans Debug
|
||||||
|
settings invalides sans echo de valeur ni WorkerId
|
||||||
|
manifest productif exact
|
||||||
|
aucun dev/build dependency
|
||||||
|
crate::Item pour les impl partagées
|
||||||
|
aucune surface Backfill/historical/checkpoint/discovery/hydration/retriever
|
||||||
|
aucun Config productif
|
||||||
|
aucun secret-like material productif
|
||||||
|
aucun backend Store direct
|
||||||
|
aucun Transport productif
|
||||||
|
aucune dépendance retour depuis les couches basses
|
||||||
|
aucun type runtime/backend/live-source exposé par la crate-root
|
||||||
|
```
|
||||||
|
|
||||||
|
## Release completeness
|
||||||
|
|
||||||
|
Nouvelle suite `tests/release_completeness.rs` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
8 modules de production exacts
|
||||||
|
24 exports publics crate-root exacts
|
||||||
|
aucun pub mod
|
||||||
|
présence des canaries dependency/public/hardening requises
|
||||||
|
```
|
||||||
|
|
||||||
|
Le corpus attendu après cette tranche est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
35 unit tests
|
||||||
|
3 dependency-boundary tests
|
||||||
|
7 public API tests
|
||||||
|
8 hardening tests
|
||||||
|
3 release-completeness tests
|
||||||
|
56 tests Worker au total
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
deltas/0.3.11/pre.010.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Code de production
|
||||||
|
|
||||||
|
Aucun fichier `crates/ksp-worker-raw-transaction-ingest-lib/src/*.rs` n'est modifié.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'est ajoutée, supprimée ou modifiée. Le `Cargo.toml` de la crate Worker reste identique à la base.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scans statiques API/dependency/historical/Config/secret/backend/Transport
|
||||||
|
comparaison byte-for-byte des sources productives et du manifest Worker
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.010` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'ayant changé, aucun `cargo tree` intermédiaire n'est requis :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.011` reste bloquée jusqu'à validation opérateur verte des canaries public/release/security.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Les sources live restent hors scope de `0.3.11`.
|
||||||
45
deltas/0.3.11/pre.011.md
Normal file
45
deltas/0.3.11/pre.011.md
Normal file
@@ -0,0 +1,45 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.011.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.011`
|
||||||
|
|
||||||
|
## Objet
|
||||||
|
|
||||||
|
Ouvrir le gate technique final de `0.3.11` sans ajouter de scope fonctionnel.
|
||||||
|
|
||||||
|
## Changements
|
||||||
|
|
||||||
|
- passage de `workspace.package.version` à `0.3.11-pre.11` ;
|
||||||
|
- clôture documentaire du gate `pre.010-fix.001` ;
|
||||||
|
- matérialisation du gate final workspace, all-targets/all-features ;
|
||||||
|
- réintroduction volontaire des arbres Cargo Worker normal/features et de `cargo tree --duplicates` pour la validation finale ;
|
||||||
|
- aucun changement de code productif, test, dépendance ou feature.
|
||||||
|
|
||||||
|
## Non-changements
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune source live
|
||||||
|
aucun Transport
|
||||||
|
aucun Config
|
||||||
|
aucun backend Store direct
|
||||||
|
aucune API publique nouvelle
|
||||||
|
aucun comportement runtime nouveau
|
||||||
|
aucune dépendance/feature nouvelle
|
||||||
|
aucune réconciliation README/USAGE
|
||||||
|
aucune publication metadata
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate requis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo test --workspace --all-targets --all-features
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
169
deltas/0.3.11/pre.012.md
Normal file
169
deltas/0.3.11/pre.012.md
Normal file
@@ -0,0 +1,169 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.012.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.012` — réconciliation documentaire finale
|
||||||
|
|
||||||
|
## 1. Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.011
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate technique final opérateur du 8 septembre 2026 est fermé avant cette tranche : rustfmt check, audits, `cargo check`, Clippy strict, 56 tests Worker, tests workspace all-targets/all-features, arbres Cargo Worker normal/features et `cargo tree --duplicates` ont été exécutés avec succès sur `0.3.11-pre.11`.
|
||||||
|
|
||||||
|
## 2. Objectif
|
||||||
|
|
||||||
|
`pre.012` est le couloir de réconciliation documentaire finale de `0.3.11`.
|
||||||
|
|
||||||
|
La tranche :
|
||||||
|
|
||||||
|
- transforme les README/USAGE du Worker en références durables ;
|
||||||
|
- remplace les formulations où le Worker était encore décrit comme futur ;
|
||||||
|
- distingue explicitement la fondation source-neutral matérialisée des sources live `0.3.12+` ;
|
||||||
|
- réaligne les graphes d'architecture sur l'absence réelle de Transport dans la fondation ;
|
||||||
|
- ferme plan et validation sur les preuves réellement exécutées ;
|
||||||
|
- ne finalise ni `CHANGELOG.md`, ni `ROADMAP.md`, ni le prompt `0.3.12`.
|
||||||
|
|
||||||
|
## 3. Version workspace
|
||||||
|
|
||||||
|
`pre.012` est documentaire mais n'est pas un fix. Le contrat Cargo KSP impose donc le bump prerelease mécanique :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.11
|
||||||
|
->
|
||||||
|
0.3.11-pre.12
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun `Cargo.toml` de crate, edge de dépendance, feature, build setting, runtime ou configuration n'est modifié.
|
||||||
|
|
||||||
|
## 4. Contrat Worker documenté
|
||||||
|
|
||||||
|
La documentation durable fixe l'état actuel suivant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Worker concret ksp-worker-raw-transaction-ingest-lib
|
||||||
|
consumer de ksp-worker-api
|
||||||
|
settings réseau/Worker bornés
|
||||||
|
start/stop sur runtime Tokio caller-owned
|
||||||
|
supervision privée des tâches
|
||||||
|
mpsc central borné et backpressure
|
||||||
|
canonicalisation/assembly via ksp-raw-transaction-lib
|
||||||
|
observation key déterministe
|
||||||
|
persistence atomique via ksp-store-lib, mode Normal
|
||||||
|
concurrence Store bornée
|
||||||
|
snapshots latest-value concrets + WorkerSnapshotSource
|
||||||
|
classification settings/runtime/store/content/source/counter/drain
|
||||||
|
shutdown deadline + abort/join avant terminal timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
La même documentation rend explicite ce qui n'appartient pas encore à la fondation :
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune source HTTP/WS/Yellowstone productive
|
||||||
|
aucun edge ksp-onchain-transport-lib
|
||||||
|
aucune lecture Config
|
||||||
|
aucun backend Store direct
|
||||||
|
aucune API publique enqueue/source registration
|
||||||
|
aucun replay/gap-repair live
|
||||||
|
aucune campagne historique/backfill
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. Graphe de dépendances réconcilié
|
||||||
|
|
||||||
|
Le graphe normal Worker confirmé par `pre.011` est documenté comme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
-> ksp-core-lib
|
||||||
|
-> ksp-logging-lib
|
||||||
|
-> ksp-raw-transaction-lib
|
||||||
|
-> ksp-store-lib # default-features = false
|
||||||
|
-> ksp-worker-api
|
||||||
|
-> sha2
|
||||||
|
-> tokio
|
||||||
|
```
|
||||||
|
|
||||||
|
L'edge Worker -> `ksp-onchain-transport-lib` reste réservé aux adapters live ultérieurs et n'est plus présenté comme une dépendance déjà matérialisée.
|
||||||
|
|
||||||
|
## 6. Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/README.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
|
||||||
|
deltas/0.3.11/pre.012.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 7. Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
README.md
|
||||||
|
docs/architecture/004-COMPONENT_INVENTORY.md
|
||||||
|
docs/architecture/005-DEPENDENCY_GRAPH.md
|
||||||
|
docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md
|
||||||
|
docs/architecture/010-APPS_SERVICES_SCENARIOS_AND_CONTROL.md
|
||||||
|
docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md
|
||||||
|
docs/plans/002-FUNCTIONAL_RELEASE_SEQUENCE.md
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 8. Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## 9. Surfaces volontairement inchangées
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/**
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/**
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/**
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
prompts/**
|
||||||
|
config/**
|
||||||
|
```
|
||||||
|
|
||||||
|
## 10. Validations exécutées pendant l'assemblage
|
||||||
|
|
||||||
|
Sur l'état exact destiné à l'archive :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 table(s), 795 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
La revue de diff confirme également qu'aucun fichier `src/`, test, manifest de crate, Config, `CHANGELOG.md`, `ROADMAP.md` ou prompt n'est modifié.
|
||||||
|
|
||||||
|
## 11. Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne dispose pas de Cargo/Rustc. Les validations Cargo propres au delta restent donc opérateur :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate technique lourd (`clippy`, workspace all-targets/all-features, arbres Cargo) n'est pas rejoué car il appartient à `pre.011` et aucun code/test/dépendance/feature n'est modifié.
|
||||||
|
|
||||||
|
## 12. Décisions prises
|
||||||
|
|
||||||
|
- la fondation `0.3.11` est documentée comme Worker concret utilisable pour lifecycle/observabilité, mais sans source réseau productive ;
|
||||||
|
- l'admission reste privée : aucune API publique d'enqueue n'est inventée pour anticiper `0.3.12` ;
|
||||||
|
- les graphes séparent l'état matériel `0.3.11` du futur edge Transport ;
|
||||||
|
- le Job Backfill reste indépendant et historique ;
|
||||||
|
- `pre.013` reste seule propriétaire du prompt `0.3.12`, de `CHANGELOG.md` et de `ROADMAP.md`.
|
||||||
|
|
||||||
|
## 13. Questions ouvertes
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune question bloquante pour la clôture documentaire 0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Les choix précis de source/adapters live restent des décisions de `0.3.12+`, sans être figés par cette tranche documentaire.
|
||||||
164
deltas/0.3.11/pre.013.md
Normal file
164
deltas/0.3.11/pre.013.md
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.013.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.013` — préparation de publication
|
||||||
|
|
||||||
|
## 1. Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.012
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.012` est vert sur rustfmt check, audits Rust/Markdown, `cargo check --workspace`, 56 tests Worker et doc-tests.
|
||||||
|
|
||||||
|
## 2. Objectif
|
||||||
|
|
||||||
|
`pre.013` est la dernière prerelease de préparation de publication de `0.3.11`.
|
||||||
|
|
||||||
|
Conformément au couloir KSP de fermeture, elle modifie uniquement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompt de démarrage 0.3.12
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
Cargo.toml racine mécanique
|
||||||
|
delta pre.013
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun README, USAGE, plan, validation, architecture, code, test, Config ou manifest de crate n'est modifié.
|
||||||
|
|
||||||
|
## 3. Version workspace
|
||||||
|
|
||||||
|
La prerelease non-fix synchronise Cargo :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.12
|
||||||
|
->
|
||||||
|
0.3.11-pre.13
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. Préparation de la release suivante
|
||||||
|
|
||||||
|
Le nouveau prompt :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/031-V0_3_12_START_PROMPT.md
|
||||||
|
```
|
||||||
|
|
||||||
|
ouvre `0.3.12` exclusivement depuis le futur stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Sa mission est bornée à Yellowstone + hydration HTTP + continuité propre au run du Worker. Son `pre.001` impose avant codage :
|
||||||
|
|
||||||
|
```text
|
||||||
|
audit stable réel 0.3.11
|
||||||
|
audit Yellowstone/Solana/provider courant
|
||||||
|
matrice Transaction/Status/Block/BlockMeta/Slot/from_slot/ReplayInfo
|
||||||
|
architecture d'injection Transport sans Worker -> Config
|
||||||
|
stratégie hydration et provenance
|
||||||
|
frontier/reconnect/replay semantics
|
||||||
|
threat model live
|
||||||
|
smokes réellement accessibles
|
||||||
|
graphe Cargo cible
|
||||||
|
sizing et rescission éventuelle
|
||||||
|
plan 033 + validation 029
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. CHANGELOG
|
||||||
|
|
||||||
|
`CHANGELOG.md` reçoit l'entrée de synthèse `0.3.11` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
fondation Worker source-neutral
|
||||||
|
admission/persistence/snapshots/shutdown hardening
|
||||||
|
dependency firewall
|
||||||
|
absence volontaire de source live dans 0.3.11
|
||||||
|
gate technique final réel
|
||||||
|
handoff 0.3.12-0.3.14
|
||||||
|
prompt 031
|
||||||
|
```
|
||||||
|
|
||||||
|
## 6. ROADMAP
|
||||||
|
|
||||||
|
L'entrée `0.3.11` passe de `[ ]` à `[X]` et décrit l'état effectivement livré. Les entrées `0.3.12`–`0.3.14` restent ouvertes et conservent leur découpage live existant.
|
||||||
|
|
||||||
|
## 7. Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/031-V0_3_12_START_PROMPT.md
|
||||||
|
deltas/0.3.11/pre.013.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 8. Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 9. Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## 10. Surfaces explicitement inchangées
|
||||||
|
|
||||||
|
```text
|
||||||
|
README.md
|
||||||
|
docs/**
|
||||||
|
crates/**
|
||||||
|
config/**
|
||||||
|
tests/**
|
||||||
|
```
|
||||||
|
|
||||||
|
Hormis le nouveau prompt situé sous `prompts/`, aucun autre prompt n'est modifié.
|
||||||
|
|
||||||
|
## 11. Validations exécutées pendant l'assemblage
|
||||||
|
|
||||||
|
Sur l'état exact destiné à l'archive :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 table(s), 797 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
La revue de diff confirme également :
|
||||||
|
|
||||||
|
```text
|
||||||
|
2 fichiers ajoutés
|
||||||
|
3 fichiers modifiés
|
||||||
|
0 fichier supprimé
|
||||||
|
aucun changement README/docs/crates/config/tests
|
||||||
|
prompt 031 : 16 sections normatives présentes
|
||||||
|
```
|
||||||
|
|
||||||
|
## 12. Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne dispose pas de Cargo/Rustc. Les contrôles Cargo minimum de la prerelease restent opérateur :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun test ou Clippy lourd n'est requis par le contenu de cette tranche de publication, puisqu'aucun code/test/dépendance/feature n'est modifié.
|
||||||
|
|
||||||
|
## 13. Décisions prises
|
||||||
|
|
||||||
|
- `0.3.11` est prêt pour une publication stable mécanique après validation de `pre.013` ;
|
||||||
|
- `0.3.12` ne commence qu'après `0.3.11-rel.001` et le tag stable `v0.3.11` ;
|
||||||
|
- le prompt suivant n'autorise aucun codage Yellowstone avant son `pre.001` d'audit/sizing ;
|
||||||
|
- aucune correction d'un couloir antérieur n'est absorbée dans cette préparation de publication.
|
||||||
|
|
||||||
|
## 14. Questions ouvertes
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune question bloquante pour la préparation de publication 0.3.11
|
||||||
|
```
|
||||||
176
deltas/0.3.11/rel.001.md
Normal file
176
deltas/0.3.11/rel.001.md
Normal file
@@ -0,0 +1,176 @@
|
|||||||
|
<!-- file: deltas/0.3.11/rel.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-rel.001` — publication stable
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
Base directe attendue :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.013
|
||||||
|
workspace.package.version = 0.3.11-pre.13
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.013`, exécuté le **8 septembre 2026**, est propre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check: PASS
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 table(s), 797 file(s))
|
||||||
|
cargo check --workspace: PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
Les couloirs de fermeture antérieurs sont déjà fermés :
|
||||||
|
|
||||||
|
- `pre.011` : gate technique final sur `0.3.11-pre.11`, avec rustfmt check, audits Rust/Markdown, `cargo check --workspace`, Clippy workspace/all-targets/all-features `-D warnings`, 56 tests Worker, `cargo test --workspace --all-targets --all-features`, arbres Cargo Worker normal/features et `cargo tree --duplicates` ;
|
||||||
|
- `pre.012` : réconciliation documentaire finale avec README/USAGE durables du Worker et architecture/références réalignées sur l'absence volontaire de source live dans `0.3.11` ;
|
||||||
|
- `pre.013` : préparation minimale de publication avec `CHANGELOG.md`, `ROADMAP.md` et `prompts/031-V0_3_12_START_PROMPT.md`, dont les 16 sections normatives ont été vérifiées conformes à `docs/rules/PROMPT_STRUCTURE.md`.
|
||||||
|
|
||||||
|
## Objet
|
||||||
|
|
||||||
|
Publier mécaniquement la version stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Conformément à `VER-LIFECYCLE-012`, cette tranche ne corrige ni code, ni test, ni documentation durable, ni architecture, ni configuration, ni dépendance, ni prompt.
|
||||||
|
|
||||||
|
Tout défaut nouveau renvoie vers une prerelease appropriée ; `rel.001` n'est pas une tranche de rattrapage.
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
### `Cargo.toml`
|
||||||
|
|
||||||
|
Le header est incrémenté parce que le fichier est réellement modifié :
|
||||||
|
|
||||||
|
```text
|
||||||
|
517 -> 518
|
||||||
|
```
|
||||||
|
|
||||||
|
La version workspace devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.13 -> 0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune autre ligne du `Cargo.toml` racine n'est modifiée.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/rel.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Surfaces explicitement inchangées
|
||||||
|
|
||||||
|
```text
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
README.md
|
||||||
|
RULES.md
|
||||||
|
prompts/031-V0_3_12_START_PROMPT.md
|
||||||
|
crates/**
|
||||||
|
docs/**
|
||||||
|
config/**
|
||||||
|
```
|
||||||
|
|
||||||
|
La surface préparée en `pre.013` reste byte-identique hors mécanique Cargo et ajout du présent delta.
|
||||||
|
|
||||||
|
## Surface stable publiée
|
||||||
|
|
||||||
|
`0.3.11` stabilise `ksp-worker-raw-transaction-ingest-lib` comme fondation source-neutral du Worker continu `RawTransaction` :
|
||||||
|
|
||||||
|
- identité Worker stable `raw_transaction_ingest` et settings bornés ;
|
||||||
|
- lifecycle `start` / stop idempotent / terminal future sur runtime Tokio fourni par le caller ;
|
||||||
|
- supervisor privé avec ownership explicite des tâches source et persistence ;
|
||||||
|
- admission centrale `mpsc` bornée, backpressure observable et absence de drop silencieux ;
|
||||||
|
- canonicalisation/assembly via `ksp-raw-transaction-lib` et observation key déterministe ;
|
||||||
|
- persistence atomique via la seule façade `ksp-store-lib`, en mode `Normal`, avec insert/idempotence/tombstone et `content_conflict` explicite ;
|
||||||
|
- snapshots concrets latest-value projetés sur `WorkerSnapshotSource`, avec séquences et compteurs checked ;
|
||||||
|
- classification stable des faults Store/source/counter/drain ;
|
||||||
|
- hardening des races stop/fault, Store lent/fautif, saturation et shutdown borné ;
|
||||||
|
- expiration de `shutdown_drain_timeout` suivie d'un abort + join de toutes les tâches privées avant publication terminale ;
|
||||||
|
- canaris externes de dependency firewall, redaction, inventaire des modules et exports publics.
|
||||||
|
|
||||||
|
La release ferme volontairement **sans source réseau productive** :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Worker -X-> ksp-onchain-transport-lib
|
||||||
|
Worker -X-> ksp-config-lib
|
||||||
|
Worker -X-> ksp-job-backfill-lib
|
||||||
|
Worker -X-> backend Store physique
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Job Backfill historique paramétré et le Worker Ingest continu restent deux producteurs indépendants du même Store. `0.3.11` ne fournit ni adapter HTTP/WS/Yellowstone productif, ni API publique d'enqueue/source registration, ni replay/gap-repair live.
|
||||||
|
|
||||||
|
La trajectoire live stabilisée est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12 Yellowstone + hydration HTTP + continuité propre au run
|
||||||
|
0.3.13 WS standard / Helius / HTTP live + convergence multi-source
|
||||||
|
0.3.14 gap repair / hardening multi-source
|
||||||
|
0.3.15 ksp-app-raw-transaction-ingest-desk
|
||||||
|
0.3.16 Backfill multi-source / multi-stratégie
|
||||||
|
```
|
||||||
|
|
||||||
|
## Suite préparée
|
||||||
|
|
||||||
|
La session suivante part exclusivement de la base stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
et exécute :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/031-V0_3_12_START_PROMPT.md
|
||||||
|
```
|
||||||
|
|
||||||
|
pour ouvrir :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12 — Yellowstone + hydration HTTP + continuité de run du Worker RawTransaction
|
||||||
|
```
|
||||||
|
|
||||||
|
`0.3.12-pre.001` doit commencer par lecture, audit interne/externe, brainstorming, sizing et planification avant toute implémentation lourde. Il doit notamment réauditer Yellowstone/Solana/providers, l'injection Transport, l'hydration HTTP, le frontier, `from_slot`/ReplayInfo, les smokes réellement accessibles et le graphe Cargo cible avant d'ajouter un edge productif au Worker.
|
||||||
|
|
||||||
|
## Gate demandé
|
||||||
|
|
||||||
|
La tranche est purement mécanique. Aucune dépendance, feature, source runtime, configuration ou documentation durable n'est modifiée par `rel.001`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.11
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Après gate propre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
commit : v0.3.11-rel.001
|
||||||
|
tag : v0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun tag de prerelease ou `rel.001` n'est requis.
|
||||||
270
deltas/0.3.12/pre.001.md
Normal file
270
deltas/0.3.12/pre.001.md
Normal file
@@ -0,0 +1,270 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.12-pre.001 — audit Yellowstone, hydration HTTP et continuité Worker
|
||||||
|
|
||||||
|
## Base
|
||||||
|
|
||||||
|
```text
|
||||||
|
archive : khadhroony-solana-project-v0.3.11.zip
|
||||||
|
SHA-256 : a6bb59935650a7f48630d12ef80fbd1a44a84819dd129b085ff7f467369535fc
|
||||||
|
ZIP bytes : 8125100
|
||||||
|
ZIP entries : 1950
|
||||||
|
unzip -t : PASS
|
||||||
|
workspace members : 21
|
||||||
|
workspace.package.version base : 0.3.11
|
||||||
|
stable delta : deltas/0.3.11/rel.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Le ZIP a une racine unique et a été contrôlé sans entrée absolue, path traversal, symlink, `.git/`, `target/`, `node_modules/`, `Cargo.lock` ou `.env`.
|
||||||
|
|
||||||
|
Préconditions du prompt `031` confirmées :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Worker RawTransaction présent et documenté
|
||||||
|
aucune source réseau productive Worker stable
|
||||||
|
Common RAW présente
|
||||||
|
Transport Yellowstone générique présent
|
||||||
|
Store facade présente
|
||||||
|
Backfill indépendant du Worker
|
||||||
|
```
|
||||||
|
|
||||||
|
## Type de livraison
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-general-0.3.12-pre.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le type `general` est requis par le bump `Cargo.toml` de toute prerelease non-fix (`VER-ID-009`). L'archive d'échange contient uniquement les fichiers ajoutés/modifiés par cette tranche et son delta.
|
||||||
|
|
||||||
|
## Baseline exécutée avant modification
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 798 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Exécuter le gate `pre.001` de `prompts/031-V0_3_12_START_PROMPT.md` : audit complet de la stable et des règles, inventaire Worker/Transport/Common RAW/Store/Config, fraîcheur Yellowstone/Solana/providers/crates, matrice des signaux, architecture d'injection, hydration/provenance, continuité/frontier, threat model, smokes, graphe Cargo et sizing, sans implémentation réseau productive.
|
||||||
|
|
||||||
|
## Audit des règles
|
||||||
|
|
||||||
|
Aucune contradiction normative active n'a été trouvée dans les règles courantes. Aucun fichier de règles n'est modifié.
|
||||||
|
|
||||||
|
Contraintes fermées :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Worker/Job séparés
|
||||||
|
Config seul propriétaire des endpoints/secrets/composition
|
||||||
|
Store facade seulement
|
||||||
|
pas de public enqueue
|
||||||
|
pas de second moteur Yellowstone
|
||||||
|
nouvelle prerelease => Cargo 0.3.12-pre.1
|
||||||
|
tranches <= 15–20 minutes
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions fermées
|
||||||
|
|
||||||
|
### Edge et injection
|
||||||
|
|
||||||
|
```text
|
||||||
|
seul nouvel edge autorisé : Worker -> ksp-onchain-transport-lib
|
||||||
|
Worker -> Config reste interdit
|
||||||
|
Worker -> Backfill reste interdit
|
||||||
|
caller supérieur construit channel/request Yellowstone + pool/role HTTP
|
||||||
|
RawTransactionIngestYellowstoneSource + RawTransactionIngestRuntimeResources retenus
|
||||||
|
start stable source-neutral conservé
|
||||||
|
start_with_runtime_resources(settings, Arc<Store>, resources) retenu
|
||||||
|
```
|
||||||
|
|
||||||
|
### Matrice Yellowstone
|
||||||
|
|
||||||
|
```text
|
||||||
|
Transaction -> candidat hydration
|
||||||
|
TransactionStatus -> candidat hydration
|
||||||
|
Block transactions-> candidats hydration
|
||||||
|
BlockMeta -> continuity/health seulement
|
||||||
|
Slot -> continuity/health seulement
|
||||||
|
from_slot -> reconnect/replay Transport, pas checkpoint historique
|
||||||
|
ReplayInfo -> borne de rétention seulement
|
||||||
|
```
|
||||||
|
|
||||||
|
### RAW/hydration
|
||||||
|
|
||||||
|
```text
|
||||||
|
RAW-direct Yellowstone non prouvé et interdit en P0
|
||||||
|
get_transaction_observed = hydration P0
|
||||||
|
Base64 + confirmed/finalized + maxSupportedTransactionVersion=0
|
||||||
|
processed exclu du chemin hydraté P0
|
||||||
|
get_block_observed non requis au P0
|
||||||
|
```
|
||||||
|
|
||||||
|
### Provenance
|
||||||
|
|
||||||
|
Le modèle Store existant est conservé sans migration. La route multi-hop est représentée par des codes sûrs composites `yellowstone_http` contenant les identités logiques Yellowstone et le winner HTTP, toutes bornées par `RawProvenanceCode`. Aucune URL/token/header n'est conservé.
|
||||||
|
|
||||||
|
### Continuité
|
||||||
|
|
||||||
|
```text
|
||||||
|
Transport high-watermark != Worker processing frontier
|
||||||
|
processing frontier != blockchain completeness
|
||||||
|
reconnect != replay != repair
|
||||||
|
replay attempt != preuve de couverture
|
||||||
|
continuity gap prouvé par ReplayInfo => fault sûr
|
||||||
|
aucun Backfill/repair automatique en 0.3.12
|
||||||
|
frontier run-local, non persistante
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fraîcheur externe du 8 septembre 2026
|
||||||
|
|
||||||
|
```text
|
||||||
|
yellowstone-grpc-proto latest = 12.7.0 (2026-08-29)
|
||||||
|
workspace = ^12.7 => aucun bump requis
|
||||||
|
Solana getTransaction fournit blockTime/meta/slot/transaction/version
|
||||||
|
Solana getBlock reste disponible mais non requis au P0
|
||||||
|
OrbitFlare documente Devnet gRPC et free Devnet-only
|
||||||
|
Helius LaserStream documente Yellowstone compatibility et replay provider-specific ~216k slots/~24h
|
||||||
|
```
|
||||||
|
|
||||||
|
Le changelog Yellowstone du 22 juillet 2026 corrige un défaut `from_slot` + `blocks` pouvant reprendre au live head sans rejouer les Blocks. KSP ne considérera donc jamais l'acceptation de `from_slot` comme une preuve de continuité.
|
||||||
|
|
||||||
|
## Smokes
|
||||||
|
|
||||||
|
Smokes Transport existants :
|
||||||
|
|
||||||
|
```text
|
||||||
|
OrbitFlare Devnet : opt-in, token opérateur
|
||||||
|
PublicNode Mainnet/Testnet : opt-in, tokens opérateur
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun secret live n'est fourni dans l'archive. Aucun smoke Worker n'est exécuté en `pre.001` et aucun faux PASS live n'est déclaré.
|
||||||
|
|
||||||
|
## Graphe Cargo cible
|
||||||
|
|
||||||
|
Après `pre.002` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
-> ksp-core-lib
|
||||||
|
-> ksp-logging-lib
|
||||||
|
-> ksp-onchain-transport-lib
|
||||||
|
-> ksp-raw-transaction-lib
|
||||||
|
-> ksp-store-lib default-features=false
|
||||||
|
-> ksp-worker-api
|
||||||
|
-> sha2
|
||||||
|
-> tokio macros,rt,sync,time
|
||||||
|
```
|
||||||
|
|
||||||
|
Pas d'edge Worker direct vers Config, Backfill, Store backend, Tonic, Reqwest, Yellowstone proto ou SDK provider.
|
||||||
|
|
||||||
|
## Sizing recalibré
|
||||||
|
|
||||||
|
```text
|
||||||
|
pre.002 Transport resource/source contract
|
||||||
|
pre.003 Transaction + TransactionStatus adapters
|
||||||
|
pre.004 HTTP hydration + provenance
|
||||||
|
pre.005 Block + BlockMeta + Slot
|
||||||
|
pre.006 source task + supervisor/coalescence
|
||||||
|
pre.007 processing frontier
|
||||||
|
pre.008 reconnect/from_slot/ReplayInfo
|
||||||
|
pre.009 race/retry/backpressure hardening
|
||||||
|
pre.010 cross-layer completeness/security
|
||||||
|
pre.011 technical/live gate
|
||||||
|
pre.012 documentation reconciliation
|
||||||
|
pre.013 publication preparation
|
||||||
|
rel.001
|
||||||
|
```
|
||||||
|
|
||||||
|
Décision : maintenir `0.3.12`. Le split frontier/replay évite une tranche de continuité trop large ; toute tranche dépassant réellement 15–20 minutes sera encore scindée avant exécution.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/plans/033-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY_PLAN.md
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
deltas/0.3.12/pre.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
Versions :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml : 518 -> 519
|
||||||
|
workspace.package.version : 0.3.11 -> 0.3.12-pre.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le bump Cargo est requis par `VER-ID-009` pour une prerelease non-fix, même si la tranche ne modifie aucun Rust.
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation après modification
|
||||||
|
|
||||||
|
Audits statiques réellement exécutés sur le delta complet :
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 801 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
Résultat : PASS.
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
`cargo` n'est pas installé dans l'environnement d'assemblage. Ne sont donc pas déclarés PASS localement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test --workspace --all-targets --all-features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
Le journal opérateur fourni avec la stable montre un gate Cargo complet vert sur `0.3.11`; cette preuve reste distincte du delta `0.3.12-pre.001`.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Après application :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions reportées
|
||||||
|
|
||||||
|
```text
|
||||||
|
RAW-direct Yellowstone après preuve byte-exact
|
||||||
|
retry Worker borné spécifique au null getTransaction
|
||||||
|
multi-source redundancy/failover
|
||||||
|
hot listener reconfiguration complète
|
||||||
|
repair automatique 0.3.14
|
||||||
|
checkpoint durable inter-process
|
||||||
|
provider-specific capabilities permanentes
|
||||||
|
```
|
||||||
261
deltas/0.3.12/pre.002.md
Normal file
261
deltas/0.3.12/pre.002.md
Normal file
@@ -0,0 +1,261 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.002` — edge Transport + contrats runtime/source Yellowstone
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.001
|
||||||
|
workspace.package.version = 0.3.12-pre.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour `pre.001` est vert sur :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
Rust rule audit
|
||||||
|
Markdown table audit
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
```
|
||||||
|
|
||||||
|
Le journal ne contient pas de `cargo test` ni de `cargo tree` ; ces commandes ne sont pas inventées comme PASS.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Ouvrir uniquement le nouvel edge de production décidé en `pre.001` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib -> ksp-onchain-transport-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
et matérialiser les contrats Worker-owned nécessaires à la future source Yellowstone + hydration HTTP, sans ouvrir de stream ni exécuter de requête réseau.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Cette tranche est une prerelease non-fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Graphe Worker normal attendu
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-core-lib
|
||||||
|
ksp-logging-lib
|
||||||
|
ksp-onchain-transport-lib
|
||||||
|
ksp-raw-transaction-lib
|
||||||
|
ksp-store-lib default-features=false
|
||||||
|
ksp-worker-api
|
||||||
|
sha2
|
||||||
|
tokio macros,rt,sync,time
|
||||||
|
```
|
||||||
|
|
||||||
|
Toujours interdits directement au Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-config-lib
|
||||||
|
ksp-job-api
|
||||||
|
ksp-job-backfill-lib
|
||||||
|
ksp-store-api
|
||||||
|
ksp-store-postgres-lib
|
||||||
|
reqwest
|
||||||
|
tokio-tungstenite
|
||||||
|
tonic
|
||||||
|
yellowstone-grpc-proto
|
||||||
|
SDK provider
|
||||||
|
```
|
||||||
|
|
||||||
|
## Nouveau contrat `RawTransactionIngestYellowstoneSource`
|
||||||
|
|
||||||
|
Le type public à champs privés possède :
|
||||||
|
|
||||||
|
```text
|
||||||
|
YellowstoneGrpcChannel
|
||||||
|
YellowstoneSubscribeRequest
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName d'hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
`RawTransactionIngestYellowstoneSource::new(...)` valide sans I/O :
|
||||||
|
|
||||||
|
```text
|
||||||
|
request Yellowstone valide
|
||||||
|
au moins une famille transactions / transactions_status / blocks
|
||||||
|
commitment explicite confirmed ou finalized
|
||||||
|
présence d'une route HTTP configurée compatible get_transaction
|
||||||
|
cohérence cluster entre Yellowstone et toutes les routes HTTP compatibles
|
||||||
|
```
|
||||||
|
|
||||||
|
Les erreurs de composition sont projetées sur `worker_raw_transaction_ingest.runtime_invalid` avec uniquement une condition stable et sûre.
|
||||||
|
|
||||||
|
## Nouveau contrat `RawTransactionIngestRuntimeResources`
|
||||||
|
|
||||||
|
`RawTransactionIngestRuntimeResources::new(yellowstone_source)` possède exactement cette première famille runtime.
|
||||||
|
|
||||||
|
Aucune collection multi-source, enum provider, closure source, callback public, `enqueue`, `send`, getter de client interne ou backend n'est exposé.
|
||||||
|
|
||||||
|
La ressource valide également la cohérence entre le réseau du source composé et `RawTransactionIngestSettings.network` avant le démarrage.
|
||||||
|
|
||||||
|
## Nouveau start de composition
|
||||||
|
|
||||||
|
Ajout :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestWorker::start_with_runtime_resources(
|
||||||
|
settings,
|
||||||
|
Arc<Store>,
|
||||||
|
RawTransactionIngestRuntimeResources,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
La méthode conserve :
|
||||||
|
|
||||||
|
```text
|
||||||
|
runtime Tokio caller-owned
|
||||||
|
validation réseau Store/settings
|
||||||
|
validation réseau Worker/source
|
||||||
|
handle/snapshot/shutdown existants
|
||||||
|
```
|
||||||
|
|
||||||
|
En `pre.002`, les ressources sont volontairement validées mais ne sont pas activées avant délégation au runtime source-neutral existant. Cela ferme l'API de composition sans anticiper `pre.006`.
|
||||||
|
|
||||||
|
## Absence volontaire de comportement live
|
||||||
|
|
||||||
|
Les canaris interdisent dans cette tranche :
|
||||||
|
|
||||||
|
```text
|
||||||
|
open_standard_subscribe
|
||||||
|
next_update
|
||||||
|
get_transaction_observed
|
||||||
|
get_block_observed
|
||||||
|
source task spawn spécifique
|
||||||
|
nouveau JoinSet
|
||||||
|
HTTP hydration
|
||||||
|
persistence issue du réseau
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.003` reste donc libre d'introduire uniquement les adapters déterministes `Transaction`/`TransactionStatus`, tandis que l'activation productive de la source reste réservée à `pre.006`.
|
||||||
|
|
||||||
|
## Sécurité et redaction
|
||||||
|
|
||||||
|
Le `Debug` des ressources expose uniquement des métadonnées sûres :
|
||||||
|
|
||||||
|
```text
|
||||||
|
logical Yellowstone endpoint name
|
||||||
|
provider
|
||||||
|
network
|
||||||
|
filter counts
|
||||||
|
commitment
|
||||||
|
presence from_slot
|
||||||
|
hydration role
|
||||||
|
HTTP endpoint count
|
||||||
|
```
|
||||||
|
|
||||||
|
Il n'expose ni URL, token, header, filter name ni client interne.
|
||||||
|
|
||||||
|
Tests déterministes ajoutés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
acceptation confirmed + transaction + get_transaction
|
||||||
|
rejet processed
|
||||||
|
rejet commitment implicite
|
||||||
|
rejet absence de famille ingestion-bearing
|
||||||
|
rejet rôle HTTP non compatible
|
||||||
|
rejet mismatch Yellowstone/HTTP cluster
|
||||||
|
rejet mismatch Worker/source network
|
||||||
|
Debug redaction
|
||||||
|
public API/root contract
|
||||||
|
manifest/dependency firewall
|
||||||
|
Transport confiné à runtime_resources.rs
|
||||||
|
absence de live I/O en pre.002
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
un seul nouvel edge productif Worker -> Onchain Transport
|
||||||
|
composition fournie par l'appelant, sans dépendance Worker -> Config
|
||||||
|
ressources Transport encapsulées par des types Worker-owned à champs privés
|
||||||
|
commitment de la future hydration limité à confirmed/finalized dès la composition
|
||||||
|
aucune activation réseau avant la tranche productive prévue
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ouverte n'est bloquante pour `pre.002`. Les choix volontairement reportés restent ceux du plan actif : forme exacte du signal privé `Transaction`/`TransactionStatus` en `pre.003`, provenance Yellowstone + hydration en `pre.004`, puis continuité/replay dans les tranches dédiées.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
deltas/0.3.12/pre.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 802 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas installés dans l'environnement d'assemblage. Aucun gate Cargo local n'est déclaré PASS.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
Cette tranche ne revendique pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
session Yellowstone ouverte
|
||||||
|
signal Transaction/TransactionStatus produit
|
||||||
|
hydration HTTP exécutée
|
||||||
|
provenance composite assemblée
|
||||||
|
coalescence d'hydration
|
||||||
|
frontier/replay/repair
|
||||||
|
smoke live Worker
|
||||||
|
réconciliation documentaire finale README/USAGE/architecture
|
||||||
|
```
|
||||||
|
|
||||||
|
## Suite
|
||||||
|
|
||||||
|
Après gate opérateur vert, `pre.003` adapte uniquement `YellowstoneTransactionUpdate` et `YellowstoneTransactionStatusUpdate` vers le signal Worker privé prévu par le plan, sans HTTP ni persistence réseau.
|
||||||
136
deltas/0.3.12/pre.003-fix.001.md
Normal file
136
deltas/0.3.12/pre.003-fix.001.md
Normal file
@@ -0,0 +1,136 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.003-fix.001.md -->
|
||||||
|
<!-- version: 2 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.003-fix.001` — suppression du dead code préparatoire
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.003
|
||||||
|
workspace.package.version = 0.3.12-pre.3
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur reçu pour `pre.003` montre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
audit Rust : clean
|
||||||
|
audit Markdown : clean (340 tables, 803 fichiers)
|
||||||
|
cargo check --workspace : PASS visible
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib : 45 PASS, 0 fail
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings : FAIL
|
||||||
|
```
|
||||||
|
|
||||||
|
L'échec Clippy est limité à huit diagnostics `dead_code` dans `runtime_resources.rs` pour les adapters privés préparatoires de `pre.003` et le champ privé `route`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger exclusivement le gate Clippy de `pre.003` sans avancer le périmètre fonctionnel vers `pre.004`.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
La correction touche du code Rust ; conformément à `VER-ID-007` et `VER-ID-010` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
delivery id = 0.3.12-pre.003-fix.001
|
||||||
|
workspace.package.version = 0.3.12-pre.3.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Correction
|
||||||
|
|
||||||
|
`RUST-API-008` exige qu'un helper strictement privé, sans consommateur de production et présent uniquement pour une préparation testée, soit compilé sous `#[cfg(test)]` jusqu'à sa première consommation réelle.
|
||||||
|
|
||||||
|
Sont donc passés sous `#[cfg(test)]` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
import sha2::Digest utilisé uniquement par le fingerprint préparatoire
|
||||||
|
RAW_TRANSACTION_INGEST_YELLOWSTONE_FILTER_FINGERPRINT_DOMAIN
|
||||||
|
RawTransactionIngestSourceFamily
|
||||||
|
RawTransactionIngestSourceTimestamp + Debug
|
||||||
|
RawTransactionIngestSourceSignal + Debug
|
||||||
|
RawTransactionIngestYellowstoneSignalView
|
||||||
|
impls Transaction et TransactionStatus
|
||||||
|
impls From vers le signal privé
|
||||||
|
matched_filter_fingerprint
|
||||||
|
project_yellowstone_signal
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces éléments restent présents dans la source et restent exercés par les unit tests `pre.003`; ils réentreront dans le build productif à leur première consommation réelle en `pre.004`.
|
||||||
|
|
||||||
|
Aucun `#[allow(dead_code)]` ni `#[expect(dead_code)]` n'est utilisé.
|
||||||
|
|
||||||
|
Le champ `RawTransactionIngestYellowstoneSource::route` n'est pas test-only : sa validation provider/endpoint fait déjà partie du contrat productif `pre.003`. Le `Debug` sûr de la source lit maintenant ce champ, ce qui supprime son warning sans masquer le lint.
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
respect strict de RUST-API-008
|
||||||
|
pas de suppression des adapters testés de pre.003
|
||||||
|
pas d'élargissement de visibilité
|
||||||
|
pas d'allow/expect dead_code
|
||||||
|
route validée conservée dans la source productive
|
||||||
|
aucune activation live anticipée
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune. La première consommation productive des adapters reste `pre.004`.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.003-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 804 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas installés dans cet environnement. Aucun gate Cargo local n'est déclaré PASS.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
Ce correctif ne revendique toujours pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
stream Yellowstone ouvert
|
||||||
|
update réseau consommée
|
||||||
|
hydration HTTP
|
||||||
|
Common RAW issu du réseau
|
||||||
|
provenance composite finale
|
||||||
|
persistence réseau
|
||||||
|
Block / BlockMeta / Slot
|
||||||
|
coalescence
|
||||||
|
frontier / from_slot / ReplayInfo
|
||||||
|
```
|
||||||
130
deltas/0.3.12/pre.003-fix.002.md
Normal file
130
deltas/0.3.12/pre.003-fix.002.md
Normal file
@@ -0,0 +1,130 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.003-fix.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.003-fix.002` — correction du canari source sous `#[cfg(test)]`
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.003-fix.001
|
||||||
|
workspace.package.version = 0.3.12-pre.3.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur reçu pour `pre.003-fix.001` montre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
audit Rust : clean
|
||||||
|
audit Markdown : clean (340 tables, 804 fichiers)
|
||||||
|
cargo check --workspace : PASS
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings : PASS
|
||||||
|
unit tests Worker : 45 PASS
|
||||||
|
integration dependency_boundary : 4 PASS
|
||||||
|
integration public_api : 8 PASS
|
||||||
|
integration release_completeness : 3 PASS
|
||||||
|
integration hardening : 9 PASS, 1 FAIL
|
||||||
|
```
|
||||||
|
|
||||||
|
L'unique échec est `v0_3_12_pre_003_private_signal_debug_and_shape_do_not_expose_signature_filters_or_payload`, qui rapporte à tort `required private signal field missing: created_at:`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger exclusivement le canari source devenu incompatible avec l'attribut `#[cfg(test)]` introduit par `pre.003-fix.001`, sans modifier le contrat runtime ni avancer `pre.004`.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
La correction modifie un test Rust ; conformément aux règles de version du workspace :
|
||||||
|
|
||||||
|
```text
|
||||||
|
delivery id = 0.3.12-pre.003-fix.002
|
||||||
|
workspace.package.version = 0.3.12-pre.3.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Correction
|
||||||
|
|
||||||
|
Le scanner du test isolait `RawTransactionIngestSourceSignal` avec une terminaison littérale qui supposait que son `impl Debug` suivait immédiatement la structure :
|
||||||
|
|
||||||
|
```text
|
||||||
|
}\n\nimpl std::fmt::Debug for RawTransactionIngestSourceSignal
|
||||||
|
```
|
||||||
|
|
||||||
|
Après `pre.003-fix.001`, la forme correcte est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
impl std::fmt::Debug for RawTransactionIngestSourceSignal
|
||||||
|
```
|
||||||
|
|
||||||
|
Le scanner est donc rendu indépendant de cet attribut : après avoir trouvé la déclaration de la structure, il borne désormais le segment sur le début de `impl std::fmt::Debug for RawTransactionIngestSourceSignal` lui-même.
|
||||||
|
|
||||||
|
Cette forme fonctionne avec les adapters actuellement test-only et restera compatible lorsque `pre.004` remettra ces helpers dans le build productif.
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune modification du signal privé
|
||||||
|
aucune modification de runtime_resources.rs
|
||||||
|
aucun allow/expect lint
|
||||||
|
aucune activation Yellowstone/HTTP/Store
|
||||||
|
aucune avance fonctionnelle vers pre.004
|
||||||
|
correction limitée au canari qui inspecte la source
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.003-fix.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
```
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas installés dans cet environnement ; aucun gate Cargo local n'est déclaré PASS.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
Ce fix ne revendique toujours pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
stream Yellowstone ouvert
|
||||||
|
update réseau consommée
|
||||||
|
hydration HTTP
|
||||||
|
Common RAW issu du réseau
|
||||||
|
provenance composite finale
|
||||||
|
persistence réseau
|
||||||
|
Block / BlockMeta / Slot
|
||||||
|
coalescence
|
||||||
|
frontier / from_slot / ReplayInfo
|
||||||
|
```
|
||||||
255
deltas/0.3.12/pre.003.md
Normal file
255
deltas/0.3.12/pre.003.md
Normal file
@@ -0,0 +1,255 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.003.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.003` — signaux privés Yellowstone Transaction + TransactionStatus
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.002
|
||||||
|
workspace.package.version = 0.3.12-pre.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour `pre.002` est vert sur :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
Rust rule audit
|
||||||
|
Markdown table audit
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib : 41 PASS
|
||||||
|
cargo tree Worker normal/features
|
||||||
|
cargo tree --duplicates exécuté pour inspection
|
||||||
|
```
|
||||||
|
|
||||||
|
Le graphe confirme que `reqwest`, `tonic` et `yellowstone-grpc-proto` restent transitifs via `ksp-onchain-transport-lib`, jamais directs dans le Worker.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement les deux adapters déterministes prévus :
|
||||||
|
|
||||||
|
```text
|
||||||
|
YellowstoneTransactionUpdate -> signal Worker privé
|
||||||
|
YellowstoneTransactionStatusUpdate -> signal Worker privé
|
||||||
|
```
|
||||||
|
|
||||||
|
sans ouvrir de session Yellowstone, sans HTTP et sans persistence réseau.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.3
|
||||||
|
```
|
||||||
|
|
||||||
|
## Signal Worker privé
|
||||||
|
|
||||||
|
Le signal est strictement privé à `runtime_resources.rs`. Sa forme contient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawNetworkId
|
||||||
|
RawTransactionSignature
|
||||||
|
slot u64
|
||||||
|
transaction_index Option<u64>
|
||||||
|
family Transaction | TransactionStatus
|
||||||
|
route sûre provider + endpoint_id
|
||||||
|
matched_filter_count
|
||||||
|
matched_filter_fingerprint [u8; 32]
|
||||||
|
created_at Option<seconds+nanos>
|
||||||
|
```
|
||||||
|
|
||||||
|
Il ne contient volontairement pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
transaction body Yellowstone
|
||||||
|
transaction meta Yellowstone
|
||||||
|
is_vote
|
||||||
|
TransactionStatus.error
|
||||||
|
raw/provider payload
|
||||||
|
HTTP material
|
||||||
|
Store acquisition
|
||||||
|
```
|
||||||
|
|
||||||
|
La signature Yellowstone `[u8; 64]` est convertie directement vers `RawTransactionSignature`; aucune Base58 intermédiaire n'est introduite.
|
||||||
|
|
||||||
|
## Adapter `Transaction`
|
||||||
|
|
||||||
|
Le mapping utilise exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
update.slot()
|
||||||
|
update.transaction().signature()
|
||||||
|
update.transaction().index()
|
||||||
|
update.filters()
|
||||||
|
update.created_at()
|
||||||
|
```
|
||||||
|
|
||||||
|
Le body et la meta présents dans `YellowstoneTransactionInfo` restent ignorés jusqu'à l'hydration HTTP qualifiée de `pre.004`.
|
||||||
|
|
||||||
|
## Adapter `TransactionStatus`
|
||||||
|
|
||||||
|
Le mapping utilise exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
update.slot()
|
||||||
|
update.signature()
|
||||||
|
update.index()
|
||||||
|
update.filters()
|
||||||
|
update.created_at()
|
||||||
|
```
|
||||||
|
|
||||||
|
`update.error()` n'est pas consulté. Le remote error opaque n'est donc ni copié dans le signal, ni transformé en contexte Worker, ni loggé.
|
||||||
|
|
||||||
|
## Fingerprint des filtres
|
||||||
|
|
||||||
|
Le fingerprint est Worker-owned, SHA-256 et domain-separated :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp.raw_transaction_ingest.yellowstone.filters.v1\0
|
||||||
|
```
|
||||||
|
|
||||||
|
Les noms sont triés puis dédupliqués avant hashing ; chaque élément est length-prefixed par un `u64` big-endian. Le résultat est donc déterministe pour le même ensemble de filtres, même si l'ordre d'enveloppe diffère.
|
||||||
|
|
||||||
|
Le golden de fixture `filter-a + filter-b` est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
20831dc8378b6070f91d9de6d3a1d2b3d77298e3ada76c01faa30826ffabcd84
|
||||||
|
```
|
||||||
|
|
||||||
|
Les noms eux-mêmes ne sont pas conservés dans le signal et n'apparaissent pas dans son `Debug`.
|
||||||
|
|
||||||
|
## Route source sûre
|
||||||
|
|
||||||
|
La source Yellowstone convertit dès construction :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cluster -> RawNetworkId
|
||||||
|
provider -> RawProvenanceCode
|
||||||
|
logical endpoint name -> RawProvenanceCode
|
||||||
|
```
|
||||||
|
|
||||||
|
Une valeur impossible à représenter selon les bornes Store est rejetée par `worker_raw_transaction_ingest.runtime_invalid` avec une condition statique. Il n'y a ni troncature ni fuite de valeur.
|
||||||
|
|
||||||
|
Cela prépare `pre.004`, où le provider/endpoint HTTP gagnant sera combiné avec cette identité Yellowstone dans la provenance composite retenue par le plan.
|
||||||
|
|
||||||
|
## Absence volontaire de comportement live
|
||||||
|
|
||||||
|
Toujours absents en `pre.003` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
open_standard_subscribe
|
||||||
|
next_update
|
||||||
|
get_transaction_observed
|
||||||
|
get_block_observed
|
||||||
|
source task Yellowstone
|
||||||
|
HTTP hydration
|
||||||
|
RawTransactionIngress issue du réseau
|
||||||
|
Store write issue du réseau
|
||||||
|
coalescence hydration
|
||||||
|
frontier / reconnect / replay
|
||||||
|
```
|
||||||
|
|
||||||
|
L'activation productive de la session reste `pre.006`.
|
||||||
|
|
||||||
|
## Tests et hardening ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Transaction fixture -> identité source-neutral exacte
|
||||||
|
TransactionStatus fixture -> même forme sans payload/error
|
||||||
|
fingerprint golden exact
|
||||||
|
fingerprint order/duplicate insensitive
|
||||||
|
signal Debug redacted
|
||||||
|
route provider/endpoint non représentable rejetée
|
||||||
|
source-scan des deux adapters Transport
|
||||||
|
TransactionStatus.error absent du mapping
|
||||||
|
signal non public
|
||||||
|
aucun I/O live prématuré
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
un seul signal privé commun aux deux familles
|
||||||
|
signature canonique conservée en bytes fixes
|
||||||
|
transaction_index toujours Some pour ces deux DTOs mais champ du signal reste optionnel pour les familles suivantes
|
||||||
|
filter fingerprint canonique indépendant de l'ordre
|
||||||
|
gRPC route identity validée tôt pour la future provenance
|
||||||
|
remote TransactionStatus.error ignoré au boundary Worker
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ouverte n'est bloquante pour `pre.003`. Les choix HTTP missing/mismatch, provenance composite finale et fan-out/coalescence restent volontairement dans `pre.004`, conformément au plan.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.003.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
```
|
||||||
|
|
||||||
|
L'audit Markdown final est exécuté après création de ce delta.
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas installés dans l'environnement d'assemblage. Aucun gate Cargo local n'est déclaré PASS.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
Cette tranche ne revendique pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
session Yellowstone ouverte
|
||||||
|
update réseau réellement consommée
|
||||||
|
hydration HTTP
|
||||||
|
conversion réseau vers Common RAW
|
||||||
|
provenance composite finale
|
||||||
|
persistence réseau
|
||||||
|
Block / BlockMeta / Slot
|
||||||
|
coalescence
|
||||||
|
processing frontier
|
||||||
|
from_slot / ReplayInfo
|
||||||
|
smoke live Worker
|
||||||
|
réconciliation README/USAGE/architecture finale
|
||||||
|
```
|
||||||
|
|
||||||
|
## Suite
|
||||||
|
|
||||||
|
Après gate opérateur vert, `pre.004` ferme `signal -> get_transaction_observed -> Common RAW -> RawTransactionIngress`, avec `missing`, mismatches et provenance Yellowstone + HTTP observée. Block et la continuité/replay restent hors scope.
|
||||||
97
deltas/0.3.12/pre.004-fix.001.md
Normal file
97
deltas/0.3.12/pre.004-fix.001.md
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.004-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.004-fix.001` — correction du gate Clippy strict
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.004
|
||||||
|
workspace.package.version = 0.3.12-pre.4
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Rust rule audit : clean
|
||||||
|
Markdown table audit : clean (340 tables, 806 fichiers)
|
||||||
|
cargo check --workspace : PASS
|
||||||
|
cargo test -p ksp-raw-transaction-lib : PASS
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib : PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
Le seul gate rouge est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
```
|
||||||
|
|
||||||
|
avec `clippy::large_enum_variant` et deux occurrences de `clippy::collapsible_if` dans `runtime_resources.rs`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger exclusivement ces trois diagnostics sans `#[allow(...)]`, sans `#[expect(...)]`, sans changer le comportement de hydration et sans avancer le scope de `pre.005`.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.4.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Corrections
|
||||||
|
|
||||||
|
`RawTransactionIngestHydrationOutcome::Available` devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Available(Box<RawTransactionIngress>)
|
||||||
|
```
|
||||||
|
|
||||||
|
La box ne concerne que ce résultat privé `#[cfg(test)]`; elle supprime la disproportion de taille entre `Available` et `Missing`. Les unit tests déplacent explicitement l'ingress hors de la box avant admission.
|
||||||
|
|
||||||
|
Le contrôle `transaction_index` utilise désormais une chaîne `if let` Rust 2024 unique. Il conserve exactement la règle précédente : un mismatch n'est rejeté que si le signal possède un index et si la réponse HTTP fournit un `SolanaWireField::Value` différent. `Omitted` et `Null` restent acceptés comme absence d'index HTTP comparable.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.004-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Non-changements
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune API publique
|
||||||
|
aucune dépendance
|
||||||
|
aucun changement de provenance
|
||||||
|
aucune politique retry Worker
|
||||||
|
aucun Block / BlockMeta / Slot
|
||||||
|
aucun source task Yellowstone productif
|
||||||
|
aucune continuité / replay
|
||||||
|
aucun lint neutralisé
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate opérateur requis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Après gate vert, la suite reste `pre.005`.
|
||||||
292
deltas/0.3.12/pre.004.md
Normal file
292
deltas/0.3.12/pre.004.md
Normal file
@@ -0,0 +1,292 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.004.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.004` — hydration `getTransaction` et provenance composite Yellowstone + HTTP
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.003-fix.002
|
||||||
|
workspace.package.version = 0.3.12-pre.3.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour cette base est entièrement vert :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
Rust rule audit : clean
|
||||||
|
Markdown table audit : clean (340 tables, 805 fichiers)
|
||||||
|
cargo check --workspace : PASS
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings : PASS
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib : PASS
|
||||||
|
unit tests Worker : 45 PASS
|
||||||
|
integration dependency_boundary : 4 PASS
|
||||||
|
integration hardening : 10 PASS
|
||||||
|
integration public_api : 8 PASS
|
||||||
|
integration release_completeness : 3 PASS
|
||||||
|
doc-tests : 0 fail
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Fermer et qualifier déterministiquement, sans source task productive, la chaîne :
|
||||||
|
|
||||||
|
```text
|
||||||
|
signal Yellowstone Transaction / TransactionStatus
|
||||||
|
-> signature canonique 64 bytes -> Base58 commun
|
||||||
|
-> get_transaction_observed
|
||||||
|
Base64
|
||||||
|
confirmed | finalized
|
||||||
|
maxSupportedTransactionVersion = 0
|
||||||
|
-> contrôles source/hydration
|
||||||
|
-> RawTransactionMaterial commun
|
||||||
|
-> RawTransactionIngress existant
|
||||||
|
```
|
||||||
|
|
||||||
|
La tranche ne branche ni Block, ni continuité/replay, ni session Yellowstone productive.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.4
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common RAW — formatage Base58
|
||||||
|
|
||||||
|
`ksp-raw-transaction-lib` expose désormais `format_raw_transaction_signature` depuis la racine de crate.
|
||||||
|
|
||||||
|
La fonction encode exactement les 64 bytes de `RawTransactionSignature` vers le texte Base58 canonique attendu par les RPC Solana. Elle ne dépend d'aucune crate Transport ni d'un second codec externe. Les bornes restent celles déjà publiques :
|
||||||
|
|
||||||
|
```text
|
||||||
|
MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES = 64
|
||||||
|
MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES = 88
|
||||||
|
```
|
||||||
|
|
||||||
|
Le round-trip `bytes -> format -> parse -> bytes` est qualifié sur plusieurs signatures, dont la signature nulle représentée par exactement 64 caractères `1`.
|
||||||
|
|
||||||
|
## Hydration HTTP qualifiée
|
||||||
|
|
||||||
|
Le helper privé de qualification construit `SolanaGetTransactionConfig` avec :
|
||||||
|
|
||||||
|
```text
|
||||||
|
encoding = base64
|
||||||
|
commitment = confirmed | finalized
|
||||||
|
maxSupportedTransactionVersion = 0
|
||||||
|
```
|
||||||
|
|
||||||
|
La signature RPC est obtenue exclusivement via `ksp_raw_transaction_lib::format_raw_transaction_signature`.
|
||||||
|
|
||||||
|
`HttpTransportPool::get_transaction_observed` reste propriétaire du routage, retry/backoff et rate-limit Transport. Le Worker n'ajoute aucune boucle retry autour d'une erreur Transport.
|
||||||
|
|
||||||
|
## Contrôles avant admission
|
||||||
|
|
||||||
|
Avant de créer un ingress, la qualification impose :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network signal == network Worker == network source
|
||||||
|
route signal == route source
|
||||||
|
slot HTTP == slot signal
|
||||||
|
transaction_index HTTP == index signal lorsque les deux côtés le fournissent
|
||||||
|
transaction HTTP réellement encodée en Base64
|
||||||
|
signature extraite du wire Base64 HTTP == signature signal
|
||||||
|
```
|
||||||
|
|
||||||
|
Un mismatch devient `worker_raw_transaction_ingest.runtime_invalid` avec une condition `hydration.*` stable. Il n'est jamais converti en content conflict Store.
|
||||||
|
|
||||||
|
Les erreurs Transport sont reclassées en `worker_raw_transaction_ingest.source_failed` en ne conservant que le domain/code lower-layer déjà sûrs.
|
||||||
|
|
||||||
|
## Missing
|
||||||
|
|
||||||
|
`getTransaction -> null` retourne :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Missing(RawTransactionReference)
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun `RawTransactionMaterial`, aucune provenance et aucun ingress ne sont inventés. Aucun retry Worker supplémentaire n'est ajouté dans cette tranche.
|
||||||
|
|
||||||
|
## Provenance composite
|
||||||
|
|
||||||
|
La provenance réutilise sans migration `RawAcquisitionProvenance` et `RawProvenanceCode` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
protocol = yellowstone_http
|
||||||
|
provider = ys.<grpc_provider>:http.<http_provider_observé>
|
||||||
|
endpoint_id = ys.<grpc_endpoint>:http.<http_endpoint_observé>
|
||||||
|
acquisition_method = transaction_get_transaction | status_get_transaction
|
||||||
|
origin = Live
|
||||||
|
commitment = confirmed | finalized
|
||||||
|
capture_session_id = WorkerId sûr
|
||||||
|
filter_id = nom direct unique représentable | sha256.<fingerprint>
|
||||||
|
observed_at = created_at Yellowstone seulement si représentable et <= received_at
|
||||||
|
```
|
||||||
|
|
||||||
|
Le provider et l'endpoint HTTP proviennent exclusivement du `HttpObservedValue` gagnant. Les URL, headers, tokens, bodies HTTP bruts et remote errors ne sont pas recopiés.
|
||||||
|
|
||||||
|
Le constructeur `RawTransactionIngestYellowstoneSource::new` valide maintenant que chaque route HTTP compatible peut former les codes composites provider/endpoint sans dépasser `RawProvenanceCode`. Aucune troncature n'est admise.
|
||||||
|
|
||||||
|
## Common RAW et wire
|
||||||
|
|
||||||
|
Après contrôles, la réponse HTTP est projetée vers `RawTransactionMaterial::binary_base64` en préservant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
blockTime -> Option<i64> puis RawTimestamp commun en millisecondes
|
||||||
|
meta omitted | null | value
|
||||||
|
version omitted | null | legacy | numeric
|
||||||
|
transactionIndex omitted | null | value
|
||||||
|
```
|
||||||
|
|
||||||
|
La signature réellement embarquée dans le wire Base64 est extraite via `extract_raw_transaction_signature_from_binary_base64` avant admission.
|
||||||
|
|
||||||
|
Les fixtures couvrent notamment :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Legacy + meta valeur + block_time valeur + index présent
|
||||||
|
V0 + meta null + block_time null + index omis
|
||||||
|
```
|
||||||
|
|
||||||
|
## `RUST-API-008`
|
||||||
|
|
||||||
|
Le source task productif n'est pas créé avant `pre.006`. Les adapters Transaction/TransactionStatus et la chaîne d'hydration, encore sans consumer productif, restent donc sous `#[cfg(test)]` conformément à `RUST-API-008`.
|
||||||
|
|
||||||
|
Aucun `#[allow(dead_code)]` ni `#[expect(dead_code)]` n'est introduit et aucune surface publique artificielle n'est créée.
|
||||||
|
|
||||||
|
Les éléments immédiatement productifs de `pre.004` sont limités à :
|
||||||
|
|
||||||
|
```text
|
||||||
|
format_raw_transaction_signature dans la common RAW
|
||||||
|
validation de représentabilité de provenance composite au constructeur de source
|
||||||
|
```
|
||||||
|
|
||||||
|
## Canaris déterministes ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
Base58 formatter round-trip et crate-root API
|
||||||
|
getTransaction confirmed/base64/max version 0
|
||||||
|
getTransaction finalized/base64/max version 0
|
||||||
|
commitment finalized conservé dans la provenance
|
||||||
|
winner HTTP observé dans provider/endpoint composites
|
||||||
|
provenance Transaction et TransactionStatus distincte
|
||||||
|
filter unique direct
|
||||||
|
multi-filter -> fingerprint sha256 borné
|
||||||
|
created_at futur ignoré pour observed_at
|
||||||
|
getTransaction null -> Missing sans ingress/retry Worker
|
||||||
|
slot mismatch
|
||||||
|
transaction index mismatch
|
||||||
|
signature wire mismatch
|
||||||
|
network mismatch avant I/O
|
||||||
|
composite provider overflow rejeté sans troncature
|
||||||
|
absence Block/getBlock/stream/source spawn
|
||||||
|
absence Config/Backfill/backend/reqwest/tonic/proto direct dans le Worker
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune migration Store pour une chaîne de provenance multi-hop
|
||||||
|
Base58 appartient à la common RAW, pas au Worker
|
||||||
|
HTTP observed est la source de vérité du provider/endpoint gagnant
|
||||||
|
processed reste exclu du P0 hydration
|
||||||
|
missing n'ajoute pas de retry Worker par défaut
|
||||||
|
mismatch source/hydration est un fault Worker avant Store
|
||||||
|
pas de RAW direct Yellowstone avant qualification byte-exacte dédiée
|
||||||
|
adapters/hydration restent test-only jusqu'au source task productif de pre.006
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ouverte n'est bloquante pour `pre.004`.
|
||||||
|
|
||||||
|
Les sujets suivants restent volontairement réservés aux tranches ultérieures :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Block / BlockMeta / Slot : pre.005
|
||||||
|
source task productive et coalescence : pre.006
|
||||||
|
processing frontier : pre.007
|
||||||
|
reconnect/from_slot/ReplayInfo : pre.008
|
||||||
|
races/retry/backpressure final : pre.009
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.004.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-raw-transaction-lib/README.md
|
||||||
|
crates/ksp-raw-transaction-lib/USAGE.md
|
||||||
|
crates/ksp-raw-transaction-lib/src/lib.rs
|
||||||
|
crates/ksp-raw-transaction-lib/src/signature.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/public_api.rs
|
||||||
|
crates/ksp-raw-transaction-lib/unit_tests/signature.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/plans/033-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY_PLAN.md
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 806 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas installés dans l'environnement d'assemblage. Aucun résultat Cargo local n'est déclaré PASS.
|
||||||
|
|
||||||
|
Le gate opérateur requis reste :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
Cette tranche ne revendique pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
open_standard_subscribe
|
||||||
|
consommation d'une update réseau live
|
||||||
|
source task Yellowstone productif
|
||||||
|
persistence d'une acquisition issue d'un stream réel
|
||||||
|
Block / BlockMeta / Slot
|
||||||
|
getBlock hydration
|
||||||
|
coalescence productive
|
||||||
|
processing frontier
|
||||||
|
from_slot / ReplayInfo
|
||||||
|
repair historique
|
||||||
|
smoke live Worker
|
||||||
|
réconciliation documentaire finale de pre.012
|
||||||
|
```
|
||||||
|
|
||||||
|
## Suite
|
||||||
|
|
||||||
|
Après gate opérateur vert, `pre.005` adapte `Block` en signaux transactionnels et `BlockMeta`/`Slot` en signaux continuity-only, sans encore introduire la politique reconnect Worker.
|
||||||
105
deltas/0.3.12/pre.005-fix.001.md
Normal file
105
deltas/0.3.12/pre.005-fix.001.md
Normal file
@@ -0,0 +1,105 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.005-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.005-fix.001` — correction de la conversion de signature Block
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.005
|
||||||
|
workspace.package.version = 0.3.12-pre.5
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Rust rule audit : clean
|
||||||
|
Markdown table audit : clean (340 tables, 808 fichiers)
|
||||||
|
cargo check --workspace : PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
La compilation des tests Worker échoue ensuite sur un unique diagnostic `E0599` : l'adapter privé `Block` appelle `YellowstoneTransactionSignature::into_bytes()`, méthode absente de la façade Transport.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement cette incompatibilité d'API sans changer le comportement `Block`, sans export supplémentaire et sans avancer le scope productif de `pre.006`.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.5.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Correction
|
||||||
|
|
||||||
|
La construction de la signature source-neutral passe de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionSignature::new(signature.into_bytes())
|
||||||
|
```
|
||||||
|
|
||||||
|
à :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionSignature::new(*signature.as_bytes())
|
||||||
|
```
|
||||||
|
|
||||||
|
`YellowstoneTransactionSignature::as_bytes()` est le contrat réel exposé par Transport. Le dereference copie le tableau canonique `[u8; 64]` attendu par `RawTransactionSignature::new`. Cette forme est déjà utilisée dans le même module pour `TransactionStatus`.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.005-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Non-changements
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune API publique
|
||||||
|
aucune dépendance
|
||||||
|
aucun changement de provenance
|
||||||
|
aucun changement d'ordre Block
|
||||||
|
aucun RAW depuis BlockMeta/Slot
|
||||||
|
aucun source task Yellowstone productif
|
||||||
|
aucune coalescence productive
|
||||||
|
aucun reconnect/from_slot/ReplayInfo
|
||||||
|
aucun lint neutralisé
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
```
|
||||||
|
|
||||||
|
Les contrôles statiques vérifient également qu'il ne reste aucun appel `YellowstoneTransactionSignature::into_bytes()` dans `runtime_resources.rs` et que la conversion `*signature.as_bytes()` correspond à la forme déjà utilisée par l'adapter `TransactionStatus`.
|
||||||
|
|
||||||
|
## Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas disponibles dans l'environnement d'assemblage. Aucun gate Cargo local n'est revendiqué.
|
||||||
|
|
||||||
|
## Gate opérateur requis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Après gate vert, la suite reste `pre.006`.
|
||||||
188
deltas/0.3.12/pre.005.md
Normal file
188
deltas/0.3.12/pre.005.md
Normal file
@@ -0,0 +1,188 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.005.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.005` — Block transactionnel + BlockMeta/Slot continuity-only
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.004-fix.001
|
||||||
|
workspace.package.version = 0.3.12-pre.4.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour cette base est entièrement vert : audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 17 tests Common RAW et 52 tests Worker avec toutes les suites d'intégration associées.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Fermer les adapters privés Yellowstone restants nécessaires avant le source task productif :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Block -> N signaux transactionnels d'hydration dans l'ordre source
|
||||||
|
BlockMeta -> signal continuity-only
|
||||||
|
Slot -> signal continuity-only
|
||||||
|
```
|
||||||
|
|
||||||
|
La tranche ne branche ni session live, ni reconnect policy Worker, ni `from_slot`, ni `ReplayInfo`, ni processing frontier.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.5
|
||||||
|
```
|
||||||
|
|
||||||
|
## Block
|
||||||
|
|
||||||
|
`YellowstoneBlockUpdate` implémente un adapter privé borné qui lit uniquement les identités nécessaires au P0 :
|
||||||
|
|
||||||
|
```text
|
||||||
|
filters
|
||||||
|
created_at
|
||||||
|
slot
|
||||||
|
transactions[].signature
|
||||||
|
transactions[].index
|
||||||
|
```
|
||||||
|
|
||||||
|
Chaque transaction incluse produit un `RawTransactionIngestSourceSignal` avec `family = Block`. L'ordre du `Vec` de sortie suit strictement l'ordre `transactions[]` du DTO Transport. Aucun body/meta Yellowstone n'entre dans Common RAW directement.
|
||||||
|
|
||||||
|
La future hydration réutilise `getTransaction` et la provenance composite `yellowstone_http`; la méthode d'acquisition devient `block_get_transaction`, distincte de `transaction_get_transaction` et `status_get_transaction`.
|
||||||
|
|
||||||
|
`get_block_observed` reste hors P0 et n'est pas introduit.
|
||||||
|
|
||||||
|
## BlockMeta et Slot
|
||||||
|
|
||||||
|
Les deux DTOs convergent vers un signal Worker privé séparé, `RawTransactionIngestContinuitySignal`. Sa forme est strictement bornée à :
|
||||||
|
|
||||||
|
```text
|
||||||
|
created_at
|
||||||
|
family
|
||||||
|
matched_filter_count
|
||||||
|
matched_filter_fingerprint
|
||||||
|
matched_filter_id
|
||||||
|
network
|
||||||
|
parent_slot
|
||||||
|
route
|
||||||
|
slot
|
||||||
|
status
|
||||||
|
```
|
||||||
|
|
||||||
|
`BlockMeta` fournit slot/parent et aucun status. Ses blockhash, rewards, block_time, block_height et compteurs ne sont pas utilisés comme preuve de complétude transactionnelle dans cette tranche.
|
||||||
|
|
||||||
|
`Slot` fournit slot/parent/status. Les sept états Yellowstone sont mappés vers un enum Worker privé source-neutral. Le diagnostic `dead_error` n'est jamais lu.
|
||||||
|
|
||||||
|
Aucun signal continuity-only ne peut produire un `RawTransactionIngress`.
|
||||||
|
|
||||||
|
## `RUST-API-008`
|
||||||
|
|
||||||
|
Le consumer productif n'arrive qu'en `pre.006`. Les adapters et projecteurs privés `Block`/`BlockMeta`/`Slot` restent donc sous `#[cfg(test)]`, comme la chaîne Transaction/Status/hydration déjà qualifiée.
|
||||||
|
|
||||||
|
Aucun `#[allow(dead_code)]`, `#[expect(dead_code)]` ou export public artificiel n'est ajouté.
|
||||||
|
|
||||||
|
## Canaris
|
||||||
|
|
||||||
|
```text
|
||||||
|
Block -> 3 signatures exactes dans l'ordre source, avec indexes non triés conservés
|
||||||
|
Block -> slot/network/filter context commun
|
||||||
|
Block -> méthode de provenance block_get_transaction
|
||||||
|
BlockMeta -> continuity-only slot/parent sans status
|
||||||
|
Slot -> continuity-only slot/parent/status
|
||||||
|
mapping exact Processed/Confirmed/Finalized/FirstShredReceived/Completed/CreatedBank/Dead
|
||||||
|
Debug continuity sans filter names ni RAW material
|
||||||
|
scanner continuity sans signature/transaction/meta/payload/body/error/dead_error
|
||||||
|
impls réels YellowstoneBlockUpdate/YellowstoneBlockMetaUpdate/YellowstoneSlotUpdate présents
|
||||||
|
aucun open_standard_subscribe/next_update/get_block_observed
|
||||||
|
aucune dépendance directe Config/Backfill/backend/reqwest/tonic/proto
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
Block est une troisième famille de discovery/hydration, pas un second pipeline RAW
|
||||||
|
BlockMeta et Slot sont continuity-only
|
||||||
|
les compteurs BlockMeta ne prouvent jamais la complétude
|
||||||
|
dead_error ne traverse jamais la frontière Worker
|
||||||
|
getBlock reste hors P0
|
||||||
|
reconnect/from_slot/ReplayInfo restent pre.008
|
||||||
|
activation source productive et coalescence restent pre.006
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ouverte ne bloque `pre.005`.
|
||||||
|
|
||||||
|
Restent réservés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
source task productive + coalescence : pre.006
|
||||||
|
processing frontier run-local : pre.007
|
||||||
|
reconnect/from_slot/ReplayInfo : pre.008
|
||||||
|
races/retry/backpressure : pre.009
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.005.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean
|
||||||
|
```
|
||||||
|
|
||||||
|
Les scanners source ciblés vérifient également l'absence d'I/O live prématurée, de `get_block_observed`, de `dead_error` dans l'adapter Slot et de dépendances d'implémentation directes interdites.
|
||||||
|
|
||||||
|
## Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas installés dans l'environnement d'assemblage. Aucun résultat Cargo local n'est déclaré PASS.
|
||||||
|
|
||||||
|
Gate opérateur requis :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
```text
|
||||||
|
pas de source Yellowstone productive
|
||||||
|
pas de stream ouvert
|
||||||
|
pas de getBlock
|
||||||
|
pas de RAW direct depuis Block
|
||||||
|
pas de RAW depuis BlockMeta/Slot
|
||||||
|
pas de coalescence productive
|
||||||
|
pas de frontier
|
||||||
|
pas de reconnect/replay/repair Worker
|
||||||
|
pas de smoke live Worker
|
||||||
|
```
|
||||||
96
deltas/0.3.12/pre.006-fix.001.md
Normal file
96
deltas/0.3.12/pre.006-fix.001.md
Normal file
@@ -0,0 +1,96 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.006-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.006-fix.001` — retours explicites et guard réseau pré-hydration
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.006
|
||||||
|
workspace.package.version = 0.3.12-pre.6
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué confirme que les audits et `cargo check --workspace` passent. Les 58 unit tests s'exécutent, avec 57 PASS et un unique échec sur le contexte du guard réseau. Le gate Clippy strict échoue sur trois `clippy::implicit-return`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger exclusivement ces diagnostics de `pre.006`, sans neutraliser de lint et sans avancer les responsabilités `pre.007`/`pre.008`.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.6.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Corrections
|
||||||
|
|
||||||
|
La branche neutre `Account/Ping/Pong/Entry` du routage Yellowstone retourne maintenant explicitement `Ok(())`.
|
||||||
|
|
||||||
|
La closure de sélection de la prochaine hydration pending utilise des `return` explicites dans ses deux branches, conformément au lint workspace `clippy::implicit-return`.
|
||||||
|
|
||||||
|
`hydration_key` rejette désormais avant coalescence et avant HTTP :
|
||||||
|
|
||||||
|
```text
|
||||||
|
signal.network != source.network -> hydration.network_mismatch
|
||||||
|
```
|
||||||
|
|
||||||
|
Ce guard restaure le contrat de qualification `pre.004`. Le contrôle équivalent de `finalize_yellowstone_hydration` reste volontairement présent comme défense secondaire. Aucun test n'est affaibli pour accepter `hydration.network_key_mismatch`.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.006-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Non-changements
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune API publique
|
||||||
|
aucune dépendance
|
||||||
|
aucun changement de clé de coalescence
|
||||||
|
aucun changement de provenance
|
||||||
|
aucune modification des bornes pending/in-flight
|
||||||
|
aucune modification de la supervision JoinSet
|
||||||
|
aucun processing frontier
|
||||||
|
aucun from_slot / ReplayInfo / reconnect policy
|
||||||
|
aucun repair/backfill implicite
|
||||||
|
aucun lint neutralisé
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
```
|
||||||
|
|
||||||
|
Contrôles statiques complémentaires : absence des trois retours implicites signalés par le gate, guard `hydration.network_mismatch` présent dans `hydration_key`, version workspace exacte et reproduction du delta sur la base `pre.006`.
|
||||||
|
|
||||||
|
## Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas disponibles dans l'environnement d'assemblage. Aucun gate Cargo local n'est revendiqué.
|
||||||
|
|
||||||
|
## Gate opérateur requis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Après gate vert, la suite reste `pre.007`.
|
||||||
219
deltas/0.3.12/pre.006.md
Normal file
219
deltas/0.3.12/pre.006.md
Normal file
@@ -0,0 +1,219 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.006.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.006` — source Yellowstone productive + coalescence bornée
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.005-fix.001
|
||||||
|
workspace.package.version = 0.3.12-pre.5.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour cette base est entièrement vert : audits Rust/Markdown, `cargo check --workspace`, Clippy strict et 56 tests Worker avec toutes les suites d'intégration associées.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Ouvrir la première source Yellowstone réellement productive en réutilisant exclusivement les moteurs déjà qualifiés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Transport Yellowstone session
|
||||||
|
adapters Transaction/Status/Block
|
||||||
|
hydration HTTP getTransaction observed
|
||||||
|
coalescence bornée
|
||||||
|
RawTransactionIngress existant
|
||||||
|
SourceTasks JoinSet existant
|
||||||
|
supervisor/admission/persistence existants
|
||||||
|
```
|
||||||
|
|
||||||
|
La tranche ne crée ni processing frontier, ni politique Worker de reconnect/from_slot/ReplayInfo, ni repair historique.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.6
|
||||||
|
```
|
||||||
|
|
||||||
|
## Source task productive
|
||||||
|
|
||||||
|
`RawTransactionIngestWorker::start_with_runtime_resources` consomme maintenant les ressources validées et ajoute exactement un child source au `JoinSet` privé existant.
|
||||||
|
|
||||||
|
Le child appelle uniquement la façade Transport :
|
||||||
|
|
||||||
|
```text
|
||||||
|
YellowstoneGrpcChannel::open_standard_subscribe
|
||||||
|
SolanaYellowstoneGrpcSubscribeSession::next_update
|
||||||
|
SolanaYellowstoneGrpcSubscribeSession::close
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun client `tonic`, `reqwest`, proto Yellowstone ou SDK provider n'est instancié par le Worker.
|
||||||
|
|
||||||
|
L'ouverture est stop-preemptible. Après stop/fault, les hydrations privées sont abort/join et la session est fermée via Transport. Toute erreur distante est réduite à un `ErrorCode` KSP sûr avant de remonter au supervisor, qui conserve son mapping stable `source_failed`.
|
||||||
|
|
||||||
|
## Activation des adapters
|
||||||
|
|
||||||
|
Les adapters `Transaction`, `TransactionStatus`, `Block`, `BlockMeta` et `Slot` sortent de `#[cfg(test)]` car ils possèdent désormais un consumer productif réel, conformément à `RUST-API-008`.
|
||||||
|
|
||||||
|
Le routage est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Transaction/TransactionStatus/Block -> candidat hydration
|
||||||
|
BlockMeta/Slot -> continuity-only, aucune admission RAW
|
||||||
|
Account/Entry/Ping/Pong -> ignorés par le vertical RAW
|
||||||
|
```
|
||||||
|
|
||||||
|
Le wrapper fixture `hydrate_yellowstone_signal` reste test-only ; le chemin productif sépare fetch HTTP et finalisation afin de permettre le fan-out coalescé.
|
||||||
|
|
||||||
|
## Coalescence
|
||||||
|
|
||||||
|
La clé est exactement `(network, signature, commitment)`.
|
||||||
|
|
||||||
|
Un `RawTransactionIngestHydrationCoordinator` privé possède :
|
||||||
|
|
||||||
|
```text
|
||||||
|
BTreeMap pending
|
||||||
|
JoinSet hydration
|
||||||
|
max_in_flight = settings.persistence_concurrency()
|
||||||
|
max_pending_signals = MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY
|
||||||
|
```
|
||||||
|
|
||||||
|
Des signaux Transaction/Status/Block identiques partagent donc un seul `getTransaction` in-flight. À la complétion, la réponse HTTP observée est finalisée séparément pour chaque signal afin de conserver family/filter/observed_at et donc des provenances distinctes.
|
||||||
|
|
||||||
|
Aucun cache terminé n'est conservé. La saturation applique de la backpressure au stream et ne provoque aucun drop silencieux.
|
||||||
|
|
||||||
|
## Hydration/admission
|
||||||
|
|
||||||
|
Le fetch reste strictement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
get_transaction_observed
|
||||||
|
base64
|
||||||
|
confirmed | finalized
|
||||||
|
maxSupportedTransactionVersion = 0
|
||||||
|
```
|
||||||
|
|
||||||
|
La finalisation conserve tous les guards `pre.004` : network, source route, slot, transaction index, signature wire et provenance composite depuis le winner HTTP réel. `null` ne produit aucun ingress.
|
||||||
|
|
||||||
|
Les ingresses disponibles passent uniquement par le sender central créé par `RawTransactionAdmission::new`; la persistence Normal reste inchangée.
|
||||||
|
|
||||||
|
## Continuité
|
||||||
|
|
||||||
|
`BlockMeta` et `Slot` sont réellement reçus/projetés dans le source task, mais restent continuity-only. Aucune frontier n'est maintenue et aucun gap n'est interprété dans cette tranche.
|
||||||
|
|
||||||
|
Le snapshot reconnect/replay de Transport n'est pas encore consommé par le Worker ; cette responsabilité reste `pre.008`.
|
||||||
|
|
||||||
|
## API / dépendances
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun nouvel export public
|
||||||
|
aucun nouveau module
|
||||||
|
aucune nouvelle dépendance Cargo
|
||||||
|
Worker -> Transport reste l'unique edge live
|
||||||
|
Worker -> Config/Backfill/Store backend reste interdit
|
||||||
|
```
|
||||||
|
|
||||||
|
## Canaris ajoutés/ajustés
|
||||||
|
|
||||||
|
```text
|
||||||
|
source runtime réellement supervisée par SourceTasks
|
||||||
|
open_standard_subscribe/next_update/close présents
|
||||||
|
coalescence key network/signature/commitment
|
||||||
|
Transaction + Status identiques -> une pending key / une hydration task
|
||||||
|
bounded pending/in-flight
|
||||||
|
stop preemptible
|
||||||
|
abort/join hydration
|
||||||
|
aucun getBlock/get_block_observed
|
||||||
|
aucun remote error/dead_error/secret
|
||||||
|
public API et module inventory inchangés
|
||||||
|
anciens canaris pre.003-pre.005 ajustés de offline -> private/Transport-only
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
coalescence productive dans le Worker, retry réseau reste Transport
|
||||||
|
concurrence hydration bornée par le budget de concurrence runtime déjà validé
|
||||||
|
pending total borné par la borne maximale d'admission publiée
|
||||||
|
fan-out après fetch partagé, avant admission
|
||||||
|
source close via Transport uniquement
|
||||||
|
frontier/reconnect/replay non anticipés
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ouverte ne bloque `pre.006`.
|
||||||
|
|
||||||
|
Restent réservés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
processing frontier run-local : pre.007
|
||||||
|
reconnect/from_slot/ReplayInfo : pre.008
|
||||||
|
races/retry/backpressure final : pre.009
|
||||||
|
cross-layer completeness/security : pre.010
|
||||||
|
live opt-in : pre.011
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.006.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
```
|
||||||
|
|
||||||
|
Des scanners statiques ciblés vérifient également le confinement Transport, l'absence de `get_block_observed`, l'absence de dépendances directes interdites, la clé de coalescence et l'inventaire public/module inchangé.
|
||||||
|
|
||||||
|
## Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas disponibles dans l'environnement d'assemblage. Aucun gate Cargo local n'est revendiqué.
|
||||||
|
|
||||||
|
## Gate opérateur requis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
```text
|
||||||
|
pas de processing frontier
|
||||||
|
pas d'interprétation reconnect/replay Worker
|
||||||
|
pas de from_slot Worker
|
||||||
|
pas de ReplayInfo Worker
|
||||||
|
pas de détection de gap de rétention
|
||||||
|
pas de repair/backfill implicite
|
||||||
|
pas de getBlock
|
||||||
|
pas de nouveau backend/Config edge
|
||||||
|
pas de smoke live revendiqué
|
||||||
|
```
|
||||||
107
deltas/0.3.12/pre.007-fix.001.md
Normal file
107
deltas/0.3.12/pre.007-fix.001.md
Normal file
@@ -0,0 +1,107 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.007-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.007-fix.001` — correction de propagation de la processing frontier
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.007
|
||||||
|
workspace.package.version = 0.3.12-pre.7
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.007` a validé les audits Rust/Markdown puis a échoué pendant `cargo check`/Clippy sur la propagation interne du receiver frontier et sur deux callsites test-only encore sur l'ancienne arité de `queue_signal`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement ces défauts d'intégration sans modifier le contrat fonctionnel de la processing frontier `pre.007`.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.7.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modifications
|
||||||
|
|
||||||
|
### Runtime supervisor
|
||||||
|
|
||||||
|
`processing_frontier_receiver` est désormais passé à `supervise_until_stop`, qui est le seul chemin qui le consomme via `wait_processing_frontier` et `record_processing_frontier`.
|
||||||
|
|
||||||
|
Le paramètre ajouté par erreur à `drain_admission_and_persistence` est retiré ; le drain reste inchangé fonctionnellement.
|
||||||
|
|
||||||
|
### Canari coalescence
|
||||||
|
|
||||||
|
Le test `pre_006_coalescence_key_merges_transaction_and_status_before_http_fanout` construit désormais un `RawTransactionIngestProcessingFrontierReporter` privé sur un `watch` local et le passe aux deux appels `queue_signal`.
|
||||||
|
|
||||||
|
Le comportement testé reste identique : les signaux Transaction et TransactionStatus identiques coalescent vers une seule hydration en vol.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichier ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.007-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Invariants préservés
|
||||||
|
|
||||||
|
Le correctif ne modifie pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
définition pending / settled
|
||||||
|
compaction des slots
|
||||||
|
processing_frontier_slot
|
||||||
|
oldest_pending_slot
|
||||||
|
hydration_pending
|
||||||
|
source Yellowstone productive
|
||||||
|
coalescence key
|
||||||
|
hydration HTTP
|
||||||
|
admission / persistence
|
||||||
|
API publique
|
||||||
|
reconnect / from_slot / ReplayInfo / repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
contrôle statique des callsites drain_admission_and_persistence
|
||||||
|
contrôle statique des callsites supervise_until_stop
|
||||||
|
contrôle statique des callsites queue_signal du canari de coalescence
|
||||||
|
reconstruction du delta sur base pre.007
|
||||||
|
unzip -t de l'archive finale
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit pas Cargo/rustc/rustfmt. Les gates Cargo doivent être rejoués par l'opérateur :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions
|
||||||
|
|
||||||
|
Aucune nouvelle décision fonctionnelle.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question ouverte introduite par ce fix.
|
||||||
108
deltas/0.3.12/pre.007-fix.002.md
Normal file
108
deltas/0.3.12/pre.007-fix.002.md
Normal file
@@ -0,0 +1,108 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.007-fix.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.007-fix.002` — correction du canari d'inventaire des watches
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.007-fix.001
|
||||||
|
workspace.package.version = 0.3.12-pre.7.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.007-fix.001` a validé les audits Rust/Markdown, `cargo check`, Clippy strict et les 61 tests unitaires Worker. L'unique échec restant provient du test d'intégration `dependency_boundary`, qui comptait encore exactement deux `tokio::sync::watch::channel` dans `runtime.rs`.
|
||||||
|
|
||||||
|
Depuis `pre.007`, une troisième watch run-local transporte légitimement la projection latest-value de processing frontier.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement ce canari structurel obsolète, sans modifier le runtime ni le contrat fonctionnel de la processing frontier.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.7.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modifications
|
||||||
|
|
||||||
|
### Canari `dependency_boundary`
|
||||||
|
|
||||||
|
Le test distingue désormais explicitement les usages de watch dans `runtime.rs` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
2 x watch::channel(false) : stop external-control + stop private-source
|
||||||
|
1 x watch::channel(RawTransactionIngestProcessingFrontierProjection::empty()) : frontier run-local latest-value
|
||||||
|
3 x watch::channel au total dans runtime.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
Le canari conserve aussi l'invariant d'une seule watch de snapshot concret dans `snapshot.rs`.
|
||||||
|
|
||||||
|
Aucun test n'est affaibli : le contrôle devient plus précis qu'un simple comptage global.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichier ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.007-fix.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Invariants préservés
|
||||||
|
|
||||||
|
Le correctif ne modifie pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
runtime.rs
|
||||||
|
pending / settled
|
||||||
|
compaction des slots
|
||||||
|
processing_frontier_slot
|
||||||
|
oldest_pending_slot
|
||||||
|
hydration_pending
|
||||||
|
source Yellowstone productive
|
||||||
|
coalescence / hydration HTTP
|
||||||
|
admission / persistence
|
||||||
|
API publique
|
||||||
|
reconnect / from_slot / ReplayInfo / repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
contrôle statique de l'inventaire des watches runtime/snapshot
|
||||||
|
reconstruction du delta sur base pre.007-fix.001
|
||||||
|
unzip -t de l'archive finale
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit pas Cargo/rustc/rustfmt. Les gates Cargo doivent être rejoués par l'opérateur :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions
|
||||||
|
|
||||||
|
Aucune nouvelle décision fonctionnelle.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question ouverte introduite par ce fix.
|
||||||
216
deltas/0.3.12/pre.007.md
Normal file
216
deltas/0.3.12/pre.007.md
Normal file
@@ -0,0 +1,216 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.007.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.007` — processing frontier run-local bornée
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.006-fix.001
|
||||||
|
workspace.package.version = 0.3.12-pre.6.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour cette base est entièrement vert : audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 58 tests Worker et toutes les suites d'intégration associées.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Ajouter la processing frontier run-local prévue au plan sans anticiper reconnect/replay/repair :
|
||||||
|
|
||||||
|
```text
|
||||||
|
source work observed
|
||||||
|
pending / settled par slot
|
||||||
|
oldest pending slot
|
||||||
|
highest unblocked actually-observed settled slot
|
||||||
|
latest-value snapshot projection
|
||||||
|
```
|
||||||
|
|
||||||
|
La frontier reste strictement processing-only.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.7
|
||||||
|
```
|
||||||
|
|
||||||
|
## Définition pending / settled
|
||||||
|
|
||||||
|
Un signal transactionnel devient pending seulement après validation de sa clé d'hydration et insertion dans le coordinator borné.
|
||||||
|
|
||||||
|
Il devient settled lorsque :
|
||||||
|
|
||||||
|
```text
|
||||||
|
hydration HTTP -> Missing
|
||||||
|
ou
|
||||||
|
hydration HTTP -> Available puis ingress accepté par la queue centrale
|
||||||
|
```
|
||||||
|
|
||||||
|
`BlockMeta` et `Slot` sont continuity-only et deviennent settled dès leur projection locale, sans créer de RAW.
|
||||||
|
|
||||||
|
Un signal non admis à cause d'un stop ou d'un fault reste pending dans la dernière projection de run. `settled` ne signifie jamais Store persisted/durable.
|
||||||
|
|
||||||
|
## Frontier
|
||||||
|
|
||||||
|
Le tracker privé maintient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
pending_total
|
||||||
|
pending par slot
|
||||||
|
settled par slot retenu
|
||||||
|
oldest_pending_slot
|
||||||
|
processing_frontier_slot
|
||||||
|
```
|
||||||
|
|
||||||
|
La frontier est la plus haute slot réellement observée/settled située avant la plus ancienne slot encore pending. Sans pending, elle devient la plus haute slot settled effectivement observée.
|
||||||
|
|
||||||
|
Ainsi, une slot pending empêche explicitement toute progression à travers elle.
|
||||||
|
|
||||||
|
## Borne mémoire
|
||||||
|
|
||||||
|
Les slots pending sont conservées exactement. Les slots settled-only sont compactées par intervalles autour des slots pending : un seul maximum settled est conservé par intervalle.
|
||||||
|
|
||||||
|
La représentation est donc bornée linéairement par le nombre de slots pending distinctes, lui-même borné par le coordinator d'hydration. Un flux continuity-only sans pending se compacte à une seule candidate.
|
||||||
|
|
||||||
|
Aucun historique de slots non borné n'est conservé.
|
||||||
|
|
||||||
|
## Projection snapshot
|
||||||
|
|
||||||
|
Le source task émet une projection privée latest-value par `tokio::sync::watch` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
hydration_pending
|
||||||
|
processing_frontier_slot
|
||||||
|
oldest_pending_slot
|
||||||
|
```
|
||||||
|
|
||||||
|
Le supervisor reste l'unique owner/mutateur du snapshot concret et applique ces valeurs via `RawTransactionIngestSnapshotPublisher`.
|
||||||
|
|
||||||
|
Les trois getters deviennent publics sur `RawTransactionIngestSnapshot`. Aucun nouveau type public ni module public n'est ajouté.
|
||||||
|
|
||||||
|
`WorkerActivity` considère désormais `hydration_pending > 0` comme activité réelle même si la queue centrale et la persistence sont momentanément vides.
|
||||||
|
|
||||||
|
## Sécurité / redaction
|
||||||
|
|
||||||
|
La projection ne transporte jamais :
|
||||||
|
|
||||||
|
```text
|
||||||
|
signature
|
||||||
|
filter id/value
|
||||||
|
provider
|
||||||
|
endpoint
|
||||||
|
transaction/meta
|
||||||
|
URL/header/credential
|
||||||
|
remote error text
|
||||||
|
```
|
||||||
|
|
||||||
|
Les erreurs de compteur/frontier utilisent uniquement des codes/contextes internes sûrs.
|
||||||
|
|
||||||
|
## Canaris ajoutés/ajustés
|
||||||
|
|
||||||
|
```text
|
||||||
|
frontier ne traverse jamais une slot pending
|
||||||
|
oldest pending avance après settlement
|
||||||
|
frontier rejoint la plus haute slot settled lorsque tout pending est résolu
|
||||||
|
compaction bornée des settled-only
|
||||||
|
snapshot initial frontier vide
|
||||||
|
snapshot latest-value des trois getters
|
||||||
|
WorkerActivity active sur hydration pending
|
||||||
|
public getters exacts
|
||||||
|
source.run inclut le publisher frontier privé
|
||||||
|
aucun ReplayInfo/from_slot/continuity_gap/repair
|
||||||
|
aucun backend/direct protocol edge
|
||||||
|
release completeness inclut les nouveaux canaris
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
frontier = source-processing, pas Store durability
|
||||||
|
Missing = settled source-processing
|
||||||
|
Available = settled après send admission réussi
|
||||||
|
continuity-only = settled immédiat local
|
||||||
|
projection supervisor via watch latest-value
|
||||||
|
pas de checkpoint persistant
|
||||||
|
pas de claim monotone de blockchain completeness
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ne bloque `pre.007`.
|
||||||
|
|
||||||
|
Restent réservés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
reconnect / from_slot / ReplayInfo / retention gap : pre.008
|
||||||
|
races/retry/backpressure final : pre.009
|
||||||
|
cross-layer completeness/security : pre.010
|
||||||
|
live opt-in : pre.011
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.007.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/snapshot.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
```
|
||||||
|
|
||||||
|
Des scanners statiques ciblés vérifient également la borne du tracker, le blocage par oldest pending, les getters publics, le confinement Transport et l'absence de responsabilités `pre.008`.
|
||||||
|
|
||||||
|
## Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas disponibles dans l'environnement d'assemblage. Aucun gate Cargo local n'est revendiqué.
|
||||||
|
|
||||||
|
## Gate opérateur requis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
```text
|
||||||
|
pas de reconnect policy Worker
|
||||||
|
pas de from_slot Worker
|
||||||
|
pas d'interprétation ReplayInfo
|
||||||
|
pas de preuve replay
|
||||||
|
pas de continuity gap fault
|
||||||
|
pas de repair/backfill implicite
|
||||||
|
pas de checkpoint inter-process
|
||||||
|
pas de getBlock
|
||||||
|
pas de nouveau backend/Config edge
|
||||||
|
pas de smoke live revendiqué
|
||||||
|
```
|
||||||
107
deltas/0.3.12/pre.008-fix.001.md
Normal file
107
deltas/0.3.12/pre.008-fix.001.md
Normal file
@@ -0,0 +1,107 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.008-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.008-fix.001` — correction du faux positif hardening `historical`
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.008
|
||||||
|
workspace.package.version = 0.3.12-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.008` valide les audits Rust/Markdown, `cargo check`, Clippy strict, 388 tests unitaires Transport, 51 `public_api` Transport, 43 `release_completeness` Transport, 63 tests unitaires Worker et 9 `dependency_boundary` Worker.
|
||||||
|
|
||||||
|
L'unique échec est le canari `pre_010_production_surface_has_no_historical_backfill_or_retriever_contract`, car la documentation de crate Worker contient la phrase `historical repair remains outside this crate`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement ce faux positif documentaire sans modifier ni affaiblir le canari hardening.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.8.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Dans `ksp-worker-raw-transaction-ingest-lib/src/lib.rs` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
historical repair remains outside this crate
|
||||||
|
```
|
||||||
|
|
||||||
|
devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
history repair remains outside this crate
|
||||||
|
```
|
||||||
|
|
||||||
|
Le token `historical` reste interdit par le canari existant. Aucun code productif, contrat, dépendance, type, import ou comportement n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichier ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.008-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Invariants préservés
|
||||||
|
|
||||||
|
Le correctif ne modifie pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Transport reconnect/from_slot/ReplayInfo
|
||||||
|
projection source-neutral Worker
|
||||||
|
processing frontier run-local
|
||||||
|
coalescence/hydration HTTP
|
||||||
|
admission/persistence
|
||||||
|
politique proven continuity gap -> source fault
|
||||||
|
API publique
|
||||||
|
backfill/repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
canari statique des tokens historiques/backfill/retriever dans src/
|
||||||
|
contrôle diff exact contre pre.008
|
||||||
|
reconstruction du delta sur la base pre.008
|
||||||
|
unzip -t de l'archive finale
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit pas Cargo/rustc/rustfmt. Le gate opérateur doit être rejoué :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions
|
||||||
|
|
||||||
|
Le canari hardening reste strict ; une phrase documentaire ne doit pas utiliser un token réservé à l'absence de surface historique si cela provoque un faux positif structurel.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
105
deltas/0.3.12/pre.008-fix.002.md
Normal file
105
deltas/0.3.12/pre.008-fix.002.md
Normal file
@@ -0,0 +1,105 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.008-fix.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.008-fix.002` — correction du faux positif `public_api` sur `repair`
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.008-fix.001
|
||||||
|
workspace.package.version = 0.3.12-pre.8.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur valide les audits, `cargo check`, Clippy strict, l'ensemble Transport, 63 tests unitaires Worker, 9 `dependency_boundary` et 15 `hardening`. L'unique échec restant est le canari `v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processing_only`, car la documentation de crate Worker contient encore le token `repair`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement ce faux positif documentaire sans modifier ni affaiblir le canari `public_api` de `pre.007`.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.8.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Dans `ksp-worker-raw-transaction-ingest-lib/src/lib.rs` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
history repair remains outside this crate
|
||||||
|
```
|
||||||
|
|
||||||
|
devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
history remediation remains outside this crate
|
||||||
|
```
|
||||||
|
|
||||||
|
Le token `repair` reste interdit par le canari `pre.007` existant. Aucun code productif, contrat, dépendance, type, import ou comportement n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichier ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.008-fix.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Invariants préservés
|
||||||
|
|
||||||
|
Le correctif ne modifie pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Transport reconnect/from_slot/ReplayInfo
|
||||||
|
projection source-neutral Worker
|
||||||
|
processing frontier run-local
|
||||||
|
coalescence/hydration HTTP
|
||||||
|
admission/persistence
|
||||||
|
politique proven continuity gap -> source fault
|
||||||
|
API publique
|
||||||
|
backfill/repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
canari statique exact des tokens interdits dans src/lib.rs
|
||||||
|
contrôle diff exact contre pre.008-fix.001
|
||||||
|
reconstruction du delta sur la base pre.008-fix.001
|
||||||
|
unzip -t de l'archive finale
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit pas Cargo/rustc/rustfmt. Le gate opérateur doit être rejoué :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions
|
||||||
|
|
||||||
|
Le canari `pre.007` reste strict et inchangé. La documentation de racine publique évite les tokens réservés aux surfaces explicitement absentes.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
205
deltas/0.3.12/pre.008.md
Normal file
205
deltas/0.3.12/pre.008.md
Normal file
@@ -0,0 +1,205 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.008.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.008` — reconnect / replay / gap de rétention prouvé
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.007-fix.002
|
||||||
|
workspace.package.version = 0.3.12-pre.7.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour cette base est entièrement vert : audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 61 tests unitaires Worker, 8 `dependency_boundary`, 14 `hardening`, 9 `public_api`, 3 `release_completeness` et doc-tests sans échec.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Brancher la continuité Yellowstone déjà possédée par Transport dans le Worker sans dupliquer la politique de replay :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Transport reconnect snapshot
|
||||||
|
-> Worker source-neutral latest-value projection
|
||||||
|
-> reconnect/replay/gap observability
|
||||||
|
-> proven retention gap => source fault
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker ne calcule ni `from_slot`, ni `SubscribeReplayInfo.first_available`; il ne déclenche aucun repair historique.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
## Façade Transport ajoutée
|
||||||
|
|
||||||
|
`ksp-onchain-transport-lib` expose désormais :
|
||||||
|
|
||||||
|
```text
|
||||||
|
YellowstoneGrpcSubscribeSnapshotSource
|
||||||
|
SolanaYellowstoneGrpcSubscribeSession::snapshot_source()
|
||||||
|
YellowstoneGrpcSubscribeSnapshotSource::current()
|
||||||
|
YellowstoneGrpcSubscribeSnapshotSource::wait_for_change()
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette façade encapsule le `tokio::sync::watch::Receiver` déjà possédé par la session. Elle n'expose ni Tonic, ni protobuf upstream, ni actor interne, ni credentials.
|
||||||
|
|
||||||
|
Le snapshot Transport reste la source autoritaire pour :
|
||||||
|
|
||||||
|
```text
|
||||||
|
state
|
||||||
|
reconnect_count
|
||||||
|
replay_attempt_count
|
||||||
|
continuity_gap_count
|
||||||
|
```
|
||||||
|
|
||||||
|
La sélection/clamp de `from_slot` et l'appel `SubscribeReplayInfo` restent exclusivement dans Transport.
|
||||||
|
|
||||||
|
## Projection Worker
|
||||||
|
|
||||||
|
Le snapshot Worker ajoute le type public source-neutral :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestSourceState
|
||||||
|
Active
|
||||||
|
Reconnecting
|
||||||
|
Closing
|
||||||
|
Closed
|
||||||
|
Failed
|
||||||
|
```
|
||||||
|
|
||||||
|
et les getters :
|
||||||
|
|
||||||
|
```text
|
||||||
|
source_state()
|
||||||
|
source_reconnect_total()
|
||||||
|
source_replay_attempt_total()
|
||||||
|
source_continuity_gap_total()
|
||||||
|
```
|
||||||
|
|
||||||
|
La même watch privée que la processing frontier transporte ces latest-values ; aucune nouvelle watch Worker n'est ajoutée.
|
||||||
|
|
||||||
|
Les champs processing-only `hydration_pending`, `processing_frontier_slot` et `oldest_pending_slot` restent inchangés et sont conservés pendant reconnect.
|
||||||
|
|
||||||
|
## Politique reconnect/replay
|
||||||
|
|
||||||
|
Le source task :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ouvre la session Yellowstone via Transport
|
||||||
|
obtient snapshot_source()
|
||||||
|
publie le snapshot initial Active
|
||||||
|
observe les changements Transport en parallèle de next_update et des hydrations
|
||||||
|
continue les hydrations déjà en vol pendant Reconnecting
|
||||||
|
laisse le stop préempter reconnect/hydration/admission
|
||||||
|
```
|
||||||
|
|
||||||
|
`replay_attempt_count` et `reconnect_count` restent distincts. Un replay attempt peut être observé alors qu'aucun reconnect n'a encore réussi.
|
||||||
|
|
||||||
|
## Gap de rétention prouvé
|
||||||
|
|
||||||
|
Le Worker ne fault que si `continuity_gap_count` augmente.
|
||||||
|
|
||||||
|
Transport incrémente ce compteur uniquement lorsqu'il prouve :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SubscribeReplayInfo.first_available > requested replay slot
|
||||||
|
```
|
||||||
|
|
||||||
|
La hausse est publiée dans le snapshot Worker, puis le source task retourne `source.continuity_gap_proven`; le supervisor classe ensuite le terminal en `ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED`.
|
||||||
|
|
||||||
|
Cette preuve reste conservatrice : elle prouve une couverture de replay demandée devenue indisponible, pas l'existence certaine d'un update filtré perdu.
|
||||||
|
|
||||||
|
Les compteurs Transport observés doivent être monotones ; une régression impossible est rejetée avec `source.continuity_counter_regression`.
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
`pre.008` n'ajoute pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
repair/backfill automatique
|
||||||
|
source secondaire ou failover multi-provider
|
||||||
|
checkpoint durable inter-process
|
||||||
|
getBlock de repair
|
||||||
|
retry Worker spécifique getTransaction null
|
||||||
|
exactly-once
|
||||||
|
preuve de continuité blockchain globale
|
||||||
|
ownership Worker de from_slot
|
||||||
|
ownership Worker de SubscribeReplayInfo
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-onchain-transport-lib/src/grpc_stream.rs
|
||||||
|
crates/ksp-onchain-transport-lib/src/lib.rs
|
||||||
|
crates/ksp-onchain-transport-lib/tests/public_api.rs
|
||||||
|
crates/ksp-onchain-transport-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-onchain-transport-lib/unit_tests/grpc_stream.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/snapshot.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichier ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.008.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
canaris statiques pre.008 replay/frontier/redaction/public API
|
||||||
|
contrôle des versions de fichiers modifiés
|
||||||
|
contrôle diff exact contre pre.007-fix.002
|
||||||
|
unzip -t du delta final
|
||||||
|
reproduction du delta sur la base exacte
|
||||||
|
```
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas installés dans l'environnement d'assemblage. Aucun `cargo check`, Clippy ou test Cargo local n'est déclaré PASS ici.
|
||||||
|
|
||||||
|
## Gate opérateur requis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions durables
|
||||||
|
|
||||||
|
```text
|
||||||
|
Transport reste propriétaire de reconnect/from_slot/ReplayInfo.
|
||||||
|
Worker observe les résultats sûrs, il ne réimplémente pas la stratégie Transport.
|
||||||
|
reconnect != replay attempt != successful reconnect != proven retention gap.
|
||||||
|
proven retention gap => fault/stop, jamais repair implicite dans 0.3.12.
|
||||||
|
processing frontier reste run-local et processing-only.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Suite
|
||||||
|
|
||||||
|
`pre.009` reste consacré au hardening races/retry/backpressure : stop pendant hydration/reconnect, duplicate storm, Store lent, source failure, counter exhaustion et no-orphan, sans élargissement fonctionnel.
|
||||||
97
deltas/0.3.12/pre.009-fix.001.md
Normal file
97
deltas/0.3.12/pre.009-fix.001.md
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.009-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.009-fix.001` — correction du canari duplicate-storm
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.009
|
||||||
|
workspace.package.version = 0.3.12-pre.9
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger l'unique échec du gate opérateur de `pre.009` sans modifier le runtime :
|
||||||
|
|
||||||
|
```text
|
||||||
|
pre_009_duplicate_storm_is_bounded_coalesced_and_abort_leaves_no_orphan_frontier
|
||||||
|
left: None
|
||||||
|
right: Some(9)
|
||||||
|
```
|
||||||
|
|
||||||
|
La predecessor `9` n'est pas une garantie du modèle borné. Après compaction, un pending tardif sur la candidate haute peut rendre la frontier conservative `None`. Le contrat requis est seulement qu'elle n'avance pas à travers le pending et qu'un abort ne l'avance pas artificiellement.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.9.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Correction
|
||||||
|
|
||||||
|
Le canari conserve les preuves de saturation/coalescence/no-orphan mais remplace l'attente exacte `Some(9)` par les invariants :
|
||||||
|
|
||||||
|
```text
|
||||||
|
frontier pré-abort != Some(10)
|
||||||
|
sauvegarde de la frontier sûre pré-abort
|
||||||
|
abort_all -> 0 pending / 0 tâche
|
||||||
|
frontier post-abort == frontier sûre pré-abort
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun test n'est supprimé et aucune garantie productive n'est relâchée.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichier ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.009-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-changements
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun code productif modifié
|
||||||
|
aucune API publique modifiée
|
||||||
|
aucune sémantique frontier/coalescence/hydration modifiée
|
||||||
|
aucun changement Transport/Store/Config/Backfill/Common RAW
|
||||||
|
aucun nouveau retry ou repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
canari statique du test corrigé
|
||||||
|
contrôle diff exact contre pre.009
|
||||||
|
unzip -t
|
||||||
|
reproduction du delta sur pre.009
|
||||||
|
```
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas disponibles dans l'environnement d'assemblage ; aucun gate Cargo local n'est revendiqué.
|
||||||
|
|
||||||
|
## Gate opérateur requis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
152
deltas/0.3.12/pre.009.md
Normal file
152
deltas/0.3.12/pre.009.md
Normal file
@@ -0,0 +1,152 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.009.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.009` — hardening races/retry/backpressure et no-orphan
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.008-fix.002
|
||||||
|
workspace.package.version = 0.3.12-pre.8.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour cette base est entièrement vert : audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 388 tests unitaires Transport, 51 `public_api` Transport, 43 `release_completeness` Transport, 63 tests unitaires Worker, 9 `dependency_boundary`, 15 `hardening`, 9 `public_api` et 3 `release_completeness` Worker sans échec.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Fermer les races terminales prévues pour `pre.009` sans élargir le pipeline :
|
||||||
|
|
||||||
|
```text
|
||||||
|
stop/fault pendant hydration
|
||||||
|
pending/coalescence abortée
|
||||||
|
processing frontier conservative
|
||||||
|
retry ownership Transport-only
|
||||||
|
duplicate storm bornée
|
||||||
|
counter exhaustion transactionnelle
|
||||||
|
no-orphan final
|
||||||
|
```
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.12-pre.9
|
||||||
|
```
|
||||||
|
|
||||||
|
## No-orphan source/hydration
|
||||||
|
|
||||||
|
`RawTransactionIngestHydrationCoordinator::abort_all` reçoit désormais le reporter de processing frontier. Le cleanup :
|
||||||
|
|
||||||
|
```text
|
||||||
|
abort_all des tâches hydration
|
||||||
|
join de toutes les tâches hydration
|
||||||
|
clear de la map pending
|
||||||
|
pending_signal_count = 0
|
||||||
|
discard_all_pending de la frontier
|
||||||
|
```
|
||||||
|
|
||||||
|
`discard_all_pending` ne convertit jamais un pending en settled. Toute slot qui contenait encore au moins un pending est retirée de la projection conservative, même si un autre signal de cette slot avait déjà été settled.
|
||||||
|
|
||||||
|
Cela empêche une frontier terminale d'avancer à travers du travail abandonné lors d'un stop, d'un source fault, d'une erreur hydration ou d'un gap de continuité prouvé.
|
||||||
|
|
||||||
|
## Arithmetic transactionnelle
|
||||||
|
|
||||||
|
Les mutations de `RawTransactionIngestProcessingFrontier` sont durcies :
|
||||||
|
|
||||||
|
```text
|
||||||
|
observe_pending : overflow vérifié avant mutation de l'entrée
|
||||||
|
observe_settled : overflow vérifié avant mutation de l'entrée
|
||||||
|
settle_pending : settled.checked_add avant décrément pending
|
||||||
|
```
|
||||||
|
|
||||||
|
Un `source.frontier_slot_settled_counter_exhausted` laisse donc le pending et la projection inchangés ; aucun état partiellement muté n'est publié.
|
||||||
|
|
||||||
|
## Duplicate storm et backpressure
|
||||||
|
|
||||||
|
Le canari source crée une petite borne déterministe et injecte plusieurs signaux identiques :
|
||||||
|
|
||||||
|
```text
|
||||||
|
hydration in-flight -> abort/join puis 0 tâche et 0 pending
|
||||||
|
3 signaux identiques -> 1 clé coalescée, 3 pending
|
||||||
|
4e signal -> source.hydration_pending_saturated
|
||||||
|
aucune tâche HTTP supplémentaire
|
||||||
|
abort -> 0 pending, 0 tâche, frontier inchangée au dernier niveau sûr
|
||||||
|
```
|
||||||
|
|
||||||
|
La borne productive existante reste inchangée et aucun canal unbounded n'est ajouté.
|
||||||
|
|
||||||
|
## Retry ownership
|
||||||
|
|
||||||
|
Le Worker conserve exactement un appel `get_transaction_observed` dans son chemin hydration et n'ajoute aucun `sleep`/`interval` de retry. Les retries réseau, backoff, rate-limit et reroutage restent exclusivement dans Transport.
|
||||||
|
|
||||||
|
Les tests runtime existants continuent de qualifier Store lent, source failure, stop pendant drain, saturation centrale, drain timeout et counter exhaustion snapshot.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichier ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/pre.009.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-changements
|
||||||
|
|
||||||
|
`pre.009` n'ajoute ni ne modifie :
|
||||||
|
|
||||||
|
```text
|
||||||
|
API publique Worker
|
||||||
|
snapshot public
|
||||||
|
Store/Config/Backfill
|
||||||
|
Common RAW
|
||||||
|
Transport replay semantics
|
||||||
|
from_slot / ReplayInfo
|
||||||
|
repair/failover
|
||||||
|
retry Worker
|
||||||
|
checkpoint durable
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
canaris statiques pre.009 no-orphan/retry ownership
|
||||||
|
contrôle des versions de fichiers modifiés
|
||||||
|
contrôle diff exact contre pre.008-fix.002
|
||||||
|
unzip -t du delta final
|
||||||
|
reproduction du delta sur la base exacte
|
||||||
|
```
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas disponibles dans l'environnement d'assemblage. Aucun gate Cargo local n'est revendiqué.
|
||||||
|
|
||||||
|
## Gate opérateur requis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
## Suite
|
||||||
|
|
||||||
|
`pre.010` reste consacré à la completeness/security cross-layer : fixtures Legacy/V0, dependency firewall, public API exact, redaction, release completeness et scanners, sans nouveau scope fonctionnel.
|
||||||
68
deltas/0.3.12/pre.010.md
Normal file
68
deltas/0.3.12/pre.010.md
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.010.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.010`
|
||||||
|
|
||||||
|
## Base
|
||||||
|
|
||||||
|
`0.3.12-pre.009-fix.001`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Fermer la tranche `pre.010` par un audit cross-layer déterministe de complétude et de sécurité entre Transport, Worker RawTransaction, Common RAW et Store, sans ajouter de comportement runtime.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
- `crates/ksp-worker-raw-transaction-ingest-lib/tests/cross_layer_completeness.rs` ;
|
||||||
|
- `deltas/0.3.12/pre.010.md`.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
- `Cargo.toml` ;
|
||||||
|
- `crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs` ;
|
||||||
|
- `docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md`.
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Décisions
|
||||||
|
|
||||||
|
- la version workspace devient `0.3.12-pre.10` ;
|
||||||
|
- aucun fichier Rust de production n'est modifié ;
|
||||||
|
- le firewall de dépendances est vérifié sur les manifests Transport, Worker, Common RAW, Store API et Store façade ;
|
||||||
|
- les fixtures Legacy/V0 déjà matérialisées dans Transport/Worker/Common RAW deviennent une matrice de complétude explicitement vérifiée ;
|
||||||
|
- les scanners/redaction déjà présents dans chaque couche sont vérifiés comme matrice de sécurité cross-layer ;
|
||||||
|
- l'inventaire public Worker reste strictement inchangé ;
|
||||||
|
- aucun nouveau comportement de reconnect, replay, hydration, persistence, repair ou checkpoint n'est introduit.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
- `python3 scripts/audit_rust_workspace_rules.py` ;
|
||||||
|
- `python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas` ;
|
||||||
|
- canaris statiques `pre.010` ;
|
||||||
|
- vérification d'inventaire du delta ;
|
||||||
|
- `unzip -t` ;
|
||||||
|
- reproduction byte-à-byte du delta sur `0.3.12-pre.009-fix.001`.
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas installés dans l'environnement d'assemblage. Aucun PASS Cargo local n'est revendiqué.
|
||||||
|
|
||||||
|
## Gate opérateur requis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-store-api
|
||||||
|
cargo test -p ksp-store-lib --no-default-features
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
71
deltas/0.3.12/pre.011.md
Normal file
71
deltas/0.3.12/pre.011.md
Normal file
@@ -0,0 +1,71 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.011.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.011`
|
||||||
|
|
||||||
|
## Base
|
||||||
|
|
||||||
|
`0.3.12-pre.010`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Fermer la tranche de gate technique final et de live opt-in prévue par le plan : workspace complet, Clippy strict, suites ciblées, graphes Cargo, duplicates et smokes réellement accessibles, sans nouveau scope fonctionnel.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
- `deltas/0.3.12/pre.011.md`.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
- `Cargo.toml` ;
|
||||||
|
- `docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md`.
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Décisions
|
||||||
|
|
||||||
|
- la version workspace devient `0.3.12-pre.11` ;
|
||||||
|
- aucun fichier Rust de production ou de test n'est modifié ;
|
||||||
|
- le gate `pre.010` réellement communiqué est enregistré sans revendiquer les commandes `cargo tree` ou workspace-all-targets qui n'étaient pas présentes dans ce log ;
|
||||||
|
- `pre.011` exige le workspace complet `--all-targets --all-features`, les suites cross-layer ciblées, les graphes Worker normal/features et `cargo tree --duplicates` ;
|
||||||
|
- les smokes HTTP/WS Devnet sans secret sont distingués des smokes Yellowstone token-gated ;
|
||||||
|
- les tokens Yellowstone restent lus sur stdin et ne doivent pas être passés en argument CLI ni copiés dans les logs/deltas ;
|
||||||
|
- aucun Worker live smoke artificiel n'est ajouté sans secret Yellowstone, hydration HTTP cohérente et Store réellement provisionné ;
|
||||||
|
- un smoke non exécutable faute de secret reste `NON EXÉCUTÉ`, jamais PASS ;
|
||||||
|
- aucun comportement reconnect/replay, frontier, hydration, persistence, Store, Config ou Backfill n'est modifié.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
- `python3 scripts/audit_rust_workspace_rules.py` ;
|
||||||
|
- `python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas` ;
|
||||||
|
- vérification statique de l'inventaire des smokes et de leur caractère `#[ignore]` ;
|
||||||
|
- vérification d'inventaire du delta ;
|
||||||
|
- `unzip -t` ;
|
||||||
|
- reproduction byte-à-byte du delta sur `0.3.12-pre.010`.
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas installés dans l'environnement d'assemblage. Aucun PASS Cargo ou live local n'est revendiqué.
|
||||||
|
|
||||||
|
## Gate opérateur déterministe requis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test --workspace --all-targets --all-features
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-store-api
|
||||||
|
cargo test -p ksp-store-lib --no-default-features
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
Les commandes live opt-in sont documentées dans `docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md` et leurs résultats doivent être enregistrés séparément du gate déterministe.
|
||||||
71
deltas/0.3.12/pre.012.md
Normal file
71
deltas/0.3.12/pre.012.md
Normal file
@@ -0,0 +1,71 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.012.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.012`
|
||||||
|
|
||||||
|
## Base
|
||||||
|
|
||||||
|
`0.3.12-pre.011`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Réconcilier la documentation durable du Worker RawTransaction avec la verticale Yellowstone + hydration HTTP + continuité réellement matérialisée et avec les preuves `pre.011`, sans changer le runtime ni préparer encore la publication.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
- `deltas/0.3.12/pre.012.md`.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
- `Cargo.toml` ;
|
||||||
|
- `README.md` ;
|
||||||
|
- `crates/ksp-worker-raw-transaction-ingest-lib/README.md` ;
|
||||||
|
- `crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md` ;
|
||||||
|
- `docs/architecture/004-COMPONENT_INVENTORY.md` ;
|
||||||
|
- `docs/architecture/005-DEPENDENCY_GRAPH.md` ;
|
||||||
|
- `docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md` ;
|
||||||
|
- `docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md` ;
|
||||||
|
- `docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md`.
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Décisions
|
||||||
|
|
||||||
|
- la version workspace devient `0.3.12-pre.12` ;
|
||||||
|
- aucun fichier Rust de production ou de test n'est modifié ;
|
||||||
|
- la documentation Worker décrit désormais la première source productive Yellowstone et l'hydration HTTP `getTransaction` ;
|
||||||
|
- `RawTransactionIngestWorker::start` reste la fondation sans source, `start_with_runtime_resources` est l'entrée productive ;
|
||||||
|
- les signaux `Transaction`, `TransactionStatus` et `Block` sont hydratés avant Common RAW ; `BlockMeta`/`Slot` restent continuity-only ;
|
||||||
|
- la processing frontier est explicitement run-local et non durable ;
|
||||||
|
- reconnect, `from_slot` et `SubscribeReplayInfo` restent propriétaires de Transport ;
|
||||||
|
- un gap de rétention prouvé provoque un fault Worker et ne déclenche aucun Backfill automatique ;
|
||||||
|
- le graphe durable matérialise l'edge Worker -> `ksp-onchain-transport-lib` sans déplacer Config, backend Store ou clients inférieurs dans le Worker ;
|
||||||
|
- le gate déterministe `pre.011` et les deux smokes keyless PASS sont enregistrés ;
|
||||||
|
- les smokes Yellowstone/Worker non réellement exécutés restent explicitement NON EXÉCUTÉS ;
|
||||||
|
- `CHANGELOG.md`, `ROADMAP.md` et le prompt suivant ne sont pas modifiés.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
- `python3 scripts/audit_rust_workspace_rules.py` ;
|
||||||
|
- `python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas` ;
|
||||||
|
- recherche des formulations durables obsolètes sur le Worker ;
|
||||||
|
- vérification d'inventaire du delta ;
|
||||||
|
- `unzip -t` ;
|
||||||
|
- reproduction byte-à-byte du delta sur `0.3.12-pre.011`.
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
`cargo`, `rustc` et `rustfmt` ne sont pas installés dans l'environnement d'assemblage. Aucun PASS Cargo local n'est revendiqué pour `pre.012`.
|
||||||
|
|
||||||
|
## Gate opérateur requis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
174
deltas/0.3.12/pre.013.md
Normal file
174
deltas/0.3.12/pre.013.md
Normal file
@@ -0,0 +1,174 @@
|
|||||||
|
<!-- file: deltas/0.3.12/pre.013.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-pre.013` — préparation de publication
|
||||||
|
|
||||||
|
## 1. Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.012
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.012` est vert sur rustfmt check, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 66 tests Worker et toutes les suites Worker boundary/hardening/public API/completeness.
|
||||||
|
|
||||||
|
Le gate technique `pre.011` précédent est déjà qualifié : workspace all-targets/all-features, suites ciblées, graphes Cargo/duplicates et smokes live keyless HTTP Devnet + WebSocket Devnet PASS.
|
||||||
|
|
||||||
|
## 2. Objectif
|
||||||
|
|
||||||
|
`pre.013` est la dernière prerelease de préparation de publication de `0.3.12`.
|
||||||
|
|
||||||
|
Conformément à `VER-LIFECYCLE-003` et `PROMPT_STRUCTURE.md`, elle modifie fonctionnellement uniquement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompt de démarrage 0.3.13
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Les seuls fichiers mécaniques supplémentaires sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml racine
|
||||||
|
delta pre.013
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun README, USAGE, plan, validation, architecture, code, test, Config ou manifest de crate n'est rouvert.
|
||||||
|
|
||||||
|
## 3. Version workspace
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.12
|
||||||
|
->
|
||||||
|
0.3.12-pre.13
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. Prompt suivant
|
||||||
|
|
||||||
|
Le nouveau prompt :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/032-V0_3_13_START_PROMPT.md
|
||||||
|
```
|
||||||
|
|
||||||
|
ouvre `0.3.13` exclusivement depuis le futur stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0.3.12
|
||||||
|
```
|
||||||
|
|
||||||
|
Sa mission est bornée à :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WS standard logsSubscribe + hydration
|
||||||
|
WS standard blockSubscribe
|
||||||
|
Helius transactionSubscribe + hydration
|
||||||
|
HTTP live block polling
|
||||||
|
convergence simultanée multi-source
|
||||||
|
observations multiples/coalescence/content conflict/backpressure
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gap repair/hardening multi-source final reste explicitement `0.3.14`.
|
||||||
|
|
||||||
|
Le `pre.001` du prompt interdit tout codage lourd avant : audit de la base stable, réaudit courant Solana/Helius, matrice de matériau/hydration, architecture runtime resources multi-source, threat model, smokes accessibles, graphes, sizing et création du plan `034` + validation `030`.
|
||||||
|
|
||||||
|
## 5. CHANGELOG
|
||||||
|
|
||||||
|
`CHANGELOG.md` reçoit l'entrée `0.3.12` synthétisant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
première source productive Yellowstone
|
||||||
|
hydration HTTP getTransaction observed
|
||||||
|
Common RAW + Store inchangés dans leurs rôles
|
||||||
|
processing frontier run-local
|
||||||
|
reconnect/from_slot/ReplayInfo Transport-owned
|
||||||
|
fault sur gap prouvé sans Backfill automatique
|
||||||
|
hardening duplicate/backpressure/shutdown
|
||||||
|
canaris cross-layer
|
||||||
|
workspace gate final PASS
|
||||||
|
smokes HTTP Devnet + WS Devnet PASS
|
||||||
|
smokes Yellowstone/Worker E2E non exécutés explicitement
|
||||||
|
handoff 0.3.13 / 0.3.14
|
||||||
|
prompt 032
|
||||||
|
```
|
||||||
|
|
||||||
|
## 6. ROADMAP
|
||||||
|
|
||||||
|
L'entrée `0.3.12` passe de `[ ]` à `[X]` et décrit l'état réellement livré. Les entrées `0.3.13` et `0.3.14` restent ouvertes et conservent leur séparation : convergence multi-source d'abord, repair multi-source ensuite.
|
||||||
|
|
||||||
|
## 7. Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/032-V0_3_13_START_PROMPT.md
|
||||||
|
deltas/0.3.12/pre.013.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 8. Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 9. Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## 10. Surfaces explicitement inchangées
|
||||||
|
|
||||||
|
```text
|
||||||
|
README.md
|
||||||
|
docs/**
|
||||||
|
crates/**
|
||||||
|
config/**
|
||||||
|
tests/**
|
||||||
|
```
|
||||||
|
|
||||||
|
Hormis le nouveau prompt sous `prompts/`, aucun prompt existant n'est modifié.
|
||||||
|
|
||||||
|
## 11. Validations exécutées pendant l'assemblage
|
||||||
|
|
||||||
|
Sur l'état exact destiné à l'archive :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 table(s), 824 file(s))
|
||||||
|
|
||||||
|
diff exact : 3 fichiers modifiés + 2 ajoutés, 0 supprimé
|
||||||
|
headers/version workspace : PASS
|
||||||
|
prompt 032 : 16 sections normatives présentes
|
||||||
|
```
|
||||||
|
|
||||||
|
Les contrôles d'archive (`unzip -t`, archive safety et reproduction byte-à-byte sur `pre.012`) sont exécutés après construction du ZIP et doivent être verts avant livraison.
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne revendique aucun Cargo PASS qui n'y a pas réellement été exécuté.
|
||||||
|
|
||||||
|
## 12. Gate opérateur minimum
|
||||||
|
|
||||||
|
Comme aucun code/test/dépendance/feature n'est modifié :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
Un Clippy/test workspace supplémentaire peut être exécuté par l'opérateur, mais `pre.013` ne doit pas réinterpréter le gate technique `pre.011` déjà validé.
|
||||||
|
|
||||||
|
## 13. Décisions prises
|
||||||
|
|
||||||
|
- `0.3.12` est prêt pour une publication stable mécanique après validation de `pre.013` ;
|
||||||
|
- `0.3.13` ne commence qu'après `0.3.12-rel.001` et le tag stable `v0.3.12` ;
|
||||||
|
- le prompt suivant maintient Worker -> Config interdit et impose la composition caller-owned des ressources live ;
|
||||||
|
- aucune correction d'un couloir technique/documentaire antérieur n'est absorbée dans cette tranche de publication.
|
||||||
|
|
||||||
|
## 14. Questions ouvertes
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune question bloquante pour la préparation de publication 0.3.12
|
||||||
|
```
|
||||||
162
deltas/0.3.12/rel.001.md
Normal file
162
deltas/0.3.12/rel.001.md
Normal file
@@ -0,0 +1,162 @@
|
|||||||
|
<!-- file: deltas/0.3.12/rel.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.12-rel.001` — publication stable
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
Base directe attendue :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.013
|
||||||
|
workspace.package.version = 0.3.12-pre.13
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur minimal de `pre.013`, exécuté le **9 septembre 2026**, est propre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check: PASS
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 table(s), 824 file(s))
|
||||||
|
cargo check --workspace: PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
Les couloirs de fermeture antérieurs sont déjà fermés :
|
||||||
|
|
||||||
|
- `pre.011` : gate technique complet avec `cargo test --workspace --all-targets --all-features`, suites ciblées, graphes Cargo Worker normal/features et `cargo tree --duplicates` ;
|
||||||
|
- smokes live keyless exécutés après `pre.011` : HTTP Solana Devnet PASS et WebSocket Solana Devnet PASS ;
|
||||||
|
- `pre.012` : réconciliation documentaire finale des surfaces Yellowstone/hydration/frontier/replay et du statut réel des preuves live ;
|
||||||
|
- `pre.013` : préparation minimale de publication avec `CHANGELOG.md`, `ROADMAP.md` et `prompts/032-V0_3_13_START_PROMPT.md`.
|
||||||
|
|
||||||
|
## Objet
|
||||||
|
|
||||||
|
Publier mécaniquement la version stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12
|
||||||
|
```
|
||||||
|
|
||||||
|
Conformément à `VER-LIFECYCLE-012`, cette tranche ne corrige ni code, ni test, ni documentation durable, ni architecture, ni configuration, ni dépendance, ni prompt.
|
||||||
|
|
||||||
|
Tout défaut nouveau renvoie vers une prerelease appropriée ; `rel.001` n'est pas une tranche de rattrapage.
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
### `Cargo.toml`
|
||||||
|
|
||||||
|
Le header est incrémenté parce que le fichier est réellement modifié :
|
||||||
|
|
||||||
|
```text
|
||||||
|
538 -> 539
|
||||||
|
```
|
||||||
|
|
||||||
|
La version workspace devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.13 -> 0.3.12
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune autre ligne du `Cargo.toml` racine n'est modifiée.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.12/rel.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Surfaces explicitement inchangées
|
||||||
|
|
||||||
|
```text
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
README.md
|
||||||
|
RULES.md
|
||||||
|
prompts/032-V0_3_13_START_PROMPT.md
|
||||||
|
crates/**
|
||||||
|
docs/**
|
||||||
|
config/**
|
||||||
|
```
|
||||||
|
|
||||||
|
La surface préparée en `pre.013` reste byte-identique hors mécanique Cargo et ajout du présent delta.
|
||||||
|
|
||||||
|
## Surface stable publiée
|
||||||
|
|
||||||
|
`0.3.12` stabilise la première verticale réseau productive de `ksp-worker-raw-transaction-ingest-lib` :
|
||||||
|
|
||||||
|
- source Yellowstone gRPC caller-composed et supervisée via `ksp-onchain-transport-lib` ;
|
||||||
|
- updates `Transaction`, `TransactionStatus` et transactions de `Block` projetées en signaux source-neutral ;
|
||||||
|
- coalescence bornée par identité `(network, signature, commitment)` avant hydration ;
|
||||||
|
- hydration HTTP `getTransaction` observed puis conversion Common RAW via `ksp-raw-transaction-lib` avant admission/persistence Store ;
|
||||||
|
- `BlockMeta` et `Slot` réservés à la continuité ;
|
||||||
|
- provenance composite sûre, sans URL, secret ni payload distant ;
|
||||||
|
- processing frontier strictement run-local, distincte d'une preuve durable de complétude ;
|
||||||
|
- reconnect/replay, `from_slot` et `SubscribeReplayInfo` conservés sous ownership Transport ;
|
||||||
|
- distinction explicite entre tentative de replay, reconnexion réussie et gap de rétention prouvé ;
|
||||||
|
- fault Worker sur gap prouvé, sans lancement automatique de Backfill ni campagne de réparation ;
|
||||||
|
- hardening des duplicate storms, overflow de compteurs, backpressure, stop/fault races, abort/join des hydrations et absence de tâches/frontier orphelines ;
|
||||||
|
- canaris cross-layer Transport -> Worker -> Common RAW -> Store, inventaires publics et dependency firewalls fermés ;
|
||||||
|
- gate workspace complet `--all-targets --all-features` validé ;
|
||||||
|
- smokes live keyless HTTP Devnet et WebSocket Devnet réellement PASS ;
|
||||||
|
- smokes Yellowstone token-gated et Worker end-to-end laissés explicitement non exécutés faute de ressources opérateur complètes lors du gate.
|
||||||
|
|
||||||
|
Le Job Backfill historique paramétré et le Worker Ingest continu restent deux producteurs indépendants du même Store. `0.3.12` ne transforme pas le Worker en moteur de campagne historique et ne persiste aucun checkpoint de replay Worker.
|
||||||
|
|
||||||
|
## Suite préparée
|
||||||
|
|
||||||
|
La session suivante part exclusivement de la base stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0.3.12
|
||||||
|
```
|
||||||
|
|
||||||
|
et exécute :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/032-V0_3_13_START_PROMPT.md
|
||||||
|
```
|
||||||
|
|
||||||
|
pour ouvrir :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.13 — WS standard / Helius / HTTP live + convergence multi-source
|
||||||
|
```
|
||||||
|
|
||||||
|
`0.3.13-pre.001` doit commencer par audit/sizing/planification avant tout développement lourd. Le gap repair/hardening multi-source complet reste réservé à `0.3.14`.
|
||||||
|
|
||||||
|
## Gate demandé
|
||||||
|
|
||||||
|
La tranche est purement mécanique. Aucune dépendance, feature, source runtime, configuration ou documentation durable n'est modifiée par `rel.001`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.12
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Après gate propre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
commit : v0.3.12-rel.001
|
||||||
|
tag : v0.3.12
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun tag de prerelease ou `rel.001` n'est requis.
|
||||||
274
deltas/0.3.13/pre.001.md
Normal file
274
deltas/0.3.13/pre.001.md
Normal file
@@ -0,0 +1,274 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.001 — audit règles et convergence live multi-source
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
archive : khadhroony-solana-project-v0.3.12.zip
|
||||||
|
SHA-256 : 85c41eb9f8575ee58edd9863445cbc37b79ef59c4a7cf882679ff24dc8991bf5
|
||||||
|
ZIP bytes : 8260528
|
||||||
|
ZIP entries : 1981
|
||||||
|
workspace.package.version base : 0.3.12
|
||||||
|
stable delta : deltas/0.3.12/rel.001.md
|
||||||
|
prompt : prompts/032-V0_3_13_START_PROMPT.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Exécuter le gate `pre.001` de `0.3.13` avant tout développement : vérifier l'archive stable et les règles, réauditer les surfaces WS standard/Helius/HTTP actuelles, fermer l'architecture de convergence multi-source, les bornes, la stratégie d'observations multiples, le content conflict, les owners retry/reconnect, les smokes et le sizing.
|
||||||
|
|
||||||
|
Aucune nouvelle source réseau productive n'est implémentée dans cette tranche.
|
||||||
|
|
||||||
|
## Type de livraison
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.13-pre.001
|
||||||
|
Cargo : 0.3.13-pre.1
|
||||||
|
archive : ksp-general-0.3.13-pre.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le type `general` est requis par la synchronisation de `Cargo.toml` pour une prerelease non-fix.
|
||||||
|
|
||||||
|
## Vérification archive
|
||||||
|
|
||||||
|
Contrôlé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
unzip -t : PASS
|
||||||
|
absolute entries : 0
|
||||||
|
parent traversal : 0
|
||||||
|
duplicate ZIP entries : 0
|
||||||
|
symlink entries : 0
|
||||||
|
Cargo.lock : absent
|
||||||
|
rust-toolchain.toml : absent
|
||||||
|
package lockfiles : absents
|
||||||
|
.env : absent
|
||||||
|
crate manifests : 21
|
||||||
|
workspace members : 21
|
||||||
|
crates hors workspace : 0
|
||||||
|
members sans crate : 0
|
||||||
|
workspace edition : 2024
|
||||||
|
```
|
||||||
|
|
||||||
|
## Audit règles
|
||||||
|
|
||||||
|
Les règles actives ont été relues depuis `RULES.md` et les neuf documents de `docs/rules/`.
|
||||||
|
|
||||||
|
Le script Rust stable reste vert, mais un contrôle supplémentaire d'unicité des identifiants normatifs a trouvé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
rule ids : 489
|
||||||
|
unique ids : 488
|
||||||
|
collision : KSP-CONFIG-018 x2
|
||||||
|
```
|
||||||
|
|
||||||
|
La preuve historique `deltas/0.1.3/pre.014.md` attribue explicitement `KSP-CONFIG-018` à la règle d'inventaire `.env.example`. La règle desktop plus récente est donc renumérotée `KSP-CONFIG-019` sans modification de sa sémantique.
|
||||||
|
|
||||||
|
Après correction :
|
||||||
|
|
||||||
|
```text
|
||||||
|
rule ids : 489
|
||||||
|
unique ids : 489
|
||||||
|
duplicates : 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune autre contradiction normative active n'a été identifiée.
|
||||||
|
|
||||||
|
## Audit architecture/dépendances
|
||||||
|
|
||||||
|
Confirmé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Worker -> Transport autorisé et déjà productif
|
||||||
|
Worker -> Config interdit et absent
|
||||||
|
Worker -> Job Backfill interdit et absent
|
||||||
|
Worker -> backend Store physique interdit et absent
|
||||||
|
Worker -> reqwest/tonic/yellowstone proto direct absent
|
||||||
|
Transport -> Store absent
|
||||||
|
Transport -> Common RAW absent
|
||||||
|
ksp-store-lib reste la façade de persistence Worker
|
||||||
|
Common RAW reste propriétaire de la canonicalisation
|
||||||
|
```
|
||||||
|
|
||||||
|
`solana-keypair` dans `ksp-wallet-lib` est explicitement admis par `DEP-SOL-004` et ne constitue pas une violation.
|
||||||
|
|
||||||
|
## Audit WS / Helius / HTTP courant
|
||||||
|
|
||||||
|
Décisions :
|
||||||
|
|
||||||
|
```text
|
||||||
|
logsSubscribe standard -> signature/slot -> getTransaction observed
|
||||||
|
blockSubscribe standard -> full/base64, maxSupportedTransactionVersion=1
|
||||||
|
Helius transactionSubscribe -> full/base64 -> getTransaction observed
|
||||||
|
Helius LaserStream WSS ne porte pas blockSubscribe
|
||||||
|
HTTP live -> getSlot + getBlocksWithLimit + getBlock observed
|
||||||
|
```
|
||||||
|
|
||||||
|
La documentation Solana courante demande la préparation `maxSupportedTransactionVersion = 1` pour Transaction V1. V1 n'est pas encore actif sur les clusters publics au 10 septembre 2026, donc la voie `blockSubscribe` V1 reste deterministic-test-gated avant d'être déclarée RAW-direct.
|
||||||
|
|
||||||
|
Le pricing Helius courant place `transactionSubscribe` à partir du tier Developer ; ce fait reste une qualification de smoke, jamais une constante métier KSP.
|
||||||
|
|
||||||
|
## Architecture multi-source fermée
|
||||||
|
|
||||||
|
Décisions principales :
|
||||||
|
|
||||||
|
```text
|
||||||
|
sources simultanées, pas de first-provider-wins
|
||||||
|
source identity opaque SHA-256 et provider-neutral
|
||||||
|
1..32 sources runtime
|
||||||
|
collection validée avant tout spawn
|
||||||
|
coordinator global Worker avant hydration
|
||||||
|
coalescence (network, signature, commitment)
|
||||||
|
une hydration HTTP pour les références cross-source identiques
|
||||||
|
pending global <= admission_queue_capacity effective
|
||||||
|
hydratation in-flight <= persistence_concurrency
|
||||||
|
aucune queue Worker provider privée non bornée
|
||||||
|
```
|
||||||
|
|
||||||
|
Observation/persistence :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identité durable = network + signature
|
||||||
|
première observation = persist_raw_transaction_acquisition atomique
|
||||||
|
observations supplémentaires = record_raw_transaction_observation
|
||||||
|
même identité + même contenu = idempotence
|
||||||
|
même identité + contenu divergent = content conflict terminal
|
||||||
|
```
|
||||||
|
|
||||||
|
La capability Store d'observation additionnelle existe déjà ; aucun changement Store API n'est requis.
|
||||||
|
|
||||||
|
## HTTP live polling
|
||||||
|
|
||||||
|
Contrat de plan retenu :
|
||||||
|
|
||||||
|
```text
|
||||||
|
run start = getSlot(commitment)
|
||||||
|
aucun scan avant ce start slot
|
||||||
|
getBlocksWithLimit pour exclure naturellement skipped slots
|
||||||
|
getBlock observed pour chaque bloc listé
|
||||||
|
poll interval : 100 ms..30 s, default 1 s
|
||||||
|
blocks/cycle : 1..1024, default 128
|
||||||
|
timer/cadence : Worker
|
||||||
|
retry/rate-limit HTTP : Transport
|
||||||
|
```
|
||||||
|
|
||||||
|
## Health et continuité
|
||||||
|
|
||||||
|
Le Worker conservera un inventaire de source privé et borné. La surface publique restera source-neutral avec des counts agrégés seulement.
|
||||||
|
|
||||||
|
Politique `0.3.13` conservative : une source configurée qui devient terminalement `Failed` faute le Worker, car l'équivalence de coverage des autres sources n'est pas prouvée. Le failover non-terminal et le repair appartiennent à `0.3.14`.
|
||||||
|
|
||||||
|
## Sizing retenu
|
||||||
|
|
||||||
|
```text
|
||||||
|
pre.002 runtime resources multi-source
|
||||||
|
pre.003 standard logs + hydration
|
||||||
|
pre.004 standard blockSubscribe
|
||||||
|
pre.005 Helius transactionSubscribe
|
||||||
|
pre.006 HTTP live polling
|
||||||
|
pre.007 supervisor/source inventory
|
||||||
|
pre.008 convergence/observations multiples
|
||||||
|
pre.009 conflict/backpressure/fairness
|
||||||
|
pre.010 snapshots/health
|
||||||
|
pre.011 races/shutdown
|
||||||
|
pre.012 completeness/security
|
||||||
|
pre.013 gate technique/live
|
||||||
|
pre.014 réconciliation documentaire
|
||||||
|
pre.015 préparation publication
|
||||||
|
rel.001 stable
|
||||||
|
```
|
||||||
|
|
||||||
|
Décision : `0.3.13` est maintenue ; aucune rescission n'est nécessaire avant `pre.002`.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md
|
||||||
|
docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md
|
||||||
|
deltas/0.3.13/pre.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/rules/RULES_KSP.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées avant modification
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
PASS : General Rust rule audit clean
|
||||||
|
PASS : Rust export completeness audit 0 candidate
|
||||||
|
PASS : KSP workspace Rust rule audit clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.12
|
||||||
|
PASS : Markdown table audit clean (306 tables, 199 files)
|
||||||
|
```
|
||||||
|
|
||||||
|
La preuve Cargo stable fournie par l'opérateur montre également fmt/check/clippy/tests Worker/trees verts sur `0.3.12`; elle reste une preuve de la base et non une exécution locale de cette prerelease.
|
||||||
|
|
||||||
|
## Validations exécutées après modification
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
PASS : General Rust rule audit clean
|
||||||
|
PASS : Rust export completeness audit 0 candidate
|
||||||
|
PASS : KSP workspace Rust rule audit clean
|
||||||
|
|
||||||
|
supplemental normative rule-id scan
|
||||||
|
PASS : 489 ids, 489 uniques, 0 duplicate
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
PASS : Markdown table audit clean (340 tables, 828 files)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
Le toolchain Cargo/Rustfmt n'est pas installé dans l'environnement d'assemblage :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
Statut : `NON EXÉCUTÉ LOCAL`, jamais PASS.
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
corriger l'identifiant desktop en KSP-CONFIG-019
|
||||||
|
maintenir la release 0.3.13
|
||||||
|
aucune nouvelle dépendance externe
|
||||||
|
maxSupportedTransactionVersion=1 pour les nouvelles voies concernées
|
||||||
|
convergence/hydration globale Worker-owned
|
||||||
|
observation fanout via capability Store existante
|
||||||
|
source failure terminale conservative jusqu'au coverage/repair 0.3.14
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Reportées explicitement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
coverage équivalent entre sources
|
||||||
|
failover non-terminal
|
||||||
|
repair de gaps multi-source
|
||||||
|
sources EARLY
|
||||||
|
campagnes historiques multi-source
|
||||||
|
composition Desk 0.3.15
|
||||||
|
```
|
||||||
171
deltas/0.3.13/pre.002-fix.001.md
Normal file
171
deltas/0.3.13/pre.002-fix.001.md
Normal file
@@ -0,0 +1,171 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.002-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.002-fix.001 — normalisation rustfmt du root Worker
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
delivery précédente : 0.3.13-pre.002
|
||||||
|
Cargo base : 0.3.13-pre.2
|
||||||
|
delta base : deltas/0.3.13/pre.002.md
|
||||||
|
archive delta base : ksp-general-0.3.13-pre.002.zip
|
||||||
|
SHA-256 delta base : 9e7c65d2340fe4918fcfa285cc14beabe4cccd3af1a58aa86e1d24a7ac8bf7b2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur exécuté après application de `pre.002` a identifié un seul défaut : `cargo fmt --all -- --check` demande le tri de l'export `MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES` après les exports du module `runtime`. Les audits Rust/Markdown, `cargo check --workspace`, Clippy strict et les suites Transport/Worker exécutées ensuite sont verts.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger strictement le défaut de format de `pre.002`, sans modification fonctionnelle :
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune API ajoutée ou supprimée
|
||||||
|
aucune signature modifiée
|
||||||
|
aucune dépendance modifiée
|
||||||
|
aucune logique runtime modifiée
|
||||||
|
aucun changement du plan pre.003+
|
||||||
|
réordonnancement rustfmt uniquement dans le root Worker
|
||||||
|
```
|
||||||
|
|
||||||
|
## Type de livraison
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.002-fix.001
|
||||||
|
Cargo : 0.3.13-pre.2.fix.1
|
||||||
|
archive : ksp-general-0.3.13-pre.002-fix.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Conformément à `VER-ID-007` et `VER-ID-010`, le correctif touche un fichier `.rs`; `workspace.package.version` est donc synchronisé avec l'identifiant SemVer technique du fix.
|
||||||
|
|
||||||
|
## Correction
|
||||||
|
|
||||||
|
Dans `crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs`, l'export :
|
||||||
|
|
||||||
|
```text
|
||||||
|
MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES
|
||||||
|
```
|
||||||
|
|
||||||
|
est déplacé après le groupe des exports `runtime`, exactement comme demandé par `rustfmt` dans le gate opérateur.
|
||||||
|
|
||||||
|
L'ordre corrigé est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RAW_TRANSACTION_INGEST_WORKER_KIND_CODE
|
||||||
|
RawTransactionIngestHandle
|
||||||
|
RawTransactionIngestTerminalFuture
|
||||||
|
RawTransactionIngestWorker
|
||||||
|
MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES
|
||||||
|
RawTransactionIngestRuntimeResources
|
||||||
|
RawTransactionIngestYellowstoneSource
|
||||||
|
```
|
||||||
|
|
||||||
|
Le changement est purement lexical. La constante, sa visibilité, son chemin public et sa valeur restent inchangés.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.002-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Versions de fichiers incrémentées
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml : 540 -> 541
|
||||||
|
worker lib.rs : 18 -> 19
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées sur la base pre.002 par l'opérateur
|
||||||
|
|
||||||
|
Le log opérateur fourni pour `0.3.13-pre.2` confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check : FAIL, uniquement diff de tri d'exports dans worker src/lib.rs
|
||||||
|
General Rust rule audit : clean
|
||||||
|
Rust export completeness audit : 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit : clean
|
||||||
|
Markdown table audit : clean (340 tables, 829 files)
|
||||||
|
cargo check --workspace : PASS
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings : PASS
|
||||||
|
cargo test -p ksp-onchain-transport-lib : PASS
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib : PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
Suites explicitement observées :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Transport unit : 389 passed
|
||||||
|
Transport public_api : 52 passed
|
||||||
|
Transport release_completeness : 44 passed
|
||||||
|
Transport doc-tests : 4 passed
|
||||||
|
smokes réseau : ignored / opt-in conformément au contrat
|
||||||
|
Worker unit : 70 passed
|
||||||
|
Worker cross_layer_completeness : 4 passed
|
||||||
|
Worker dependency_boundary : 9 passed
|
||||||
|
Worker hardening : 16 passed
|
||||||
|
Worker public_api : 10 passed
|
||||||
|
Worker release_completeness : 4 passed
|
||||||
|
Worker doc-tests : 0
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage du fix
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
contrôle supplémentaire des collisions d'identifiants normatifs
|
||||||
|
contrôle exhaustif du diff pre.002 -> pre.002-fix.001
|
||||||
|
contrôle du contenu et de l'intégrité ZIP
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
Le toolchain Rust n'est pas disponible dans l'environnement d'assemblage. Les commandes suivantes ne sont donc pas déclarées PASS localement :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Le correctif reproduit exactement le déplacement demandé par le diff `cargo fmt --all -- --check` du gate opérateur.
|
||||||
|
|
||||||
|
## Gate opérateur avant pre.003
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions
|
||||||
|
|
||||||
|
- Le fix ne change pas le périmètre fonctionnel de `pre.002`.
|
||||||
|
- Le supervisor multi-source reste réservé à `pre.007`.
|
||||||
|
- `pre.003` reste la prochaine tranche fonctionnelle après validation de ce correctif.
|
||||||
|
- Aucun document durable n'est modifié pour une correction de format déjà entièrement décrite par ce delta.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune
|
||||||
|
```
|
||||||
323
deltas/0.3.13/pre.002.md
Normal file
323
deltas/0.3.13/pre.002.md
Normal file
@@ -0,0 +1,323 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.002 — runtime resources multi-source bornées
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
delivery précédente : 0.3.13-pre.001
|
||||||
|
Cargo base : 0.3.13-pre.1
|
||||||
|
delta base : deltas/0.3.13/pre.001.md
|
||||||
|
archive delta base : ksp-general-0.3.13-pre.001.zip
|
||||||
|
SHA-256 delta base : ca57546f19bd47b5d3b832a51905159497a5e601c186460db22b04364a1835c4
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur fourni après application de `pre.001` est vert : fmt, audits Rust/Markdown, check workspace, Clippy strict, tests Worker, arbres normal/features et duplicates ont été exécutés sans anomalie signalée.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser le contrat `pre.002` défini par le plan `034` sans démarrer prématurément les nouvelles familles live :
|
||||||
|
|
||||||
|
```text
|
||||||
|
collection runtime privée bornée 1..32
|
||||||
|
identité logique déterministe par source
|
||||||
|
source discriminant capability-owned
|
||||||
|
validation globale avant spawn
|
||||||
|
rejet des doublons logiques
|
||||||
|
rejet cross-network
|
||||||
|
aucun nouveau client réseau
|
||||||
|
aucune nouvelle dépendance
|
||||||
|
```
|
||||||
|
|
||||||
|
Le supervisor simultané appartient toujours à `pre.007`.
|
||||||
|
|
||||||
|
## Type de livraison
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.13-pre.002
|
||||||
|
Cargo : 0.3.13-pre.2
|
||||||
|
archive : ksp-general-0.3.13-pre.002.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
## Runtime resources multi-source
|
||||||
|
|
||||||
|
`RawTransactionIngestRuntimeResources` ne contient plus directement un unique champ Yellowstone. Il possède désormais une collection privée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Vec<RawTransactionIngestLiveSource>
|
||||||
|
```
|
||||||
|
|
||||||
|
Le discriminant privé initial est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Yellowstone(RawTransactionIngestYellowstoneSource)
|
||||||
|
```
|
||||||
|
|
||||||
|
Il est volontairement capability-owned ; aucun provider n'entre dans l'enum Worker.
|
||||||
|
|
||||||
|
Surface publique ajoutée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES = 32
|
||||||
|
RawTransactionIngestRuntimeResources::source_count()
|
||||||
|
RawTransactionIngestRuntimeResources::try_push_yellowstone_source(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
`try_push_yellowstone_source` est transactionnel : le candidat est entièrement rejeté sans mutation de la collection si la borne, le réseau ou l'identité logique sont invalides.
|
||||||
|
|
||||||
|
## Identité logique Yellowstone
|
||||||
|
|
||||||
|
Chaque `RawTransactionIngestYellowstoneSource` possède maintenant un `source_key: [u8; 32]` privé.
|
||||||
|
|
||||||
|
Le hash SHA-256 est dérivé uniquement de données sûres :
|
||||||
|
|
||||||
|
```text
|
||||||
|
KSP domain separator
|
||||||
|
source family = yellowstone
|
||||||
|
network
|
||||||
|
provider identity sûre
|
||||||
|
endpoint identity sûre
|
||||||
|
commitment
|
||||||
|
identité opaque de YellowstoneSubscribeRequest
|
||||||
|
```
|
||||||
|
|
||||||
|
Le rôle et le pool HTTP d'hydration sont volontairement exclus du `source_key` : ils ne définissent pas une nouvelle source live Yellowstone. Ainsi, deux subscriptions identiques ne deviennent pas artificiellement distinctes par simple changement de stratégie d'hydration.
|
||||||
|
|
||||||
|
Sont explicitement absents du matériau en clair et de Debug :
|
||||||
|
|
||||||
|
```text
|
||||||
|
URL
|
||||||
|
API key / credential
|
||||||
|
header
|
||||||
|
transaction payload
|
||||||
|
remote error material
|
||||||
|
request identity bytes
|
||||||
|
source_key bytes/hash
|
||||||
|
```
|
||||||
|
|
||||||
|
Concernant `source_key`, le Debug de la source n'en publie que la longueur ; ses autres champs Debug restent les identités sûres déjà admises par le contrat stable. Le Debug de l'aggregate runtime publie seulement `source_count`.
|
||||||
|
|
||||||
|
## Identité déterministe de SubscribeRequest dans Transport
|
||||||
|
|
||||||
|
`ksp-onchain-transport-lib` expose `YellowstoneSubscribeRequestIdentity` comme valeur opaque et `YellowstoneSubscribeRequest::identity()`.
|
||||||
|
|
||||||
|
Canonicalisation de l'identité :
|
||||||
|
|
||||||
|
```text
|
||||||
|
sept familles de filtres séparées
|
||||||
|
ordre déterministe fourni par les BTreeMap KSP
|
||||||
|
nom de filtre length-framed
|
||||||
|
protobuf exact de chaque filtre length-framed
|
||||||
|
options communes protobuf encodées après vidage des sept maps
|
||||||
|
```
|
||||||
|
|
||||||
|
Les bytes internes restent privés. L'identité implémente `Hash` afin que le Worker puisse l'incorporer à son SHA-256 sans escape hatch de sérialisation.
|
||||||
|
|
||||||
|
Aucune dépendance n'est ajoutée : le codage réutilise `prost` réexporté par `yellowstone-grpc-proto`, déjà dépendance de Transport.
|
||||||
|
|
||||||
|
## Validation globale avant spawn
|
||||||
|
|
||||||
|
Avant activation, `RawTransactionIngestRuntimeResources::validate_network` impose :
|
||||||
|
|
||||||
|
```text
|
||||||
|
collection non vide
|
||||||
|
source_count <= 32
|
||||||
|
chaque source sur le réseau Worker
|
||||||
|
source_key unique dans la collection
|
||||||
|
```
|
||||||
|
|
||||||
|
Les erreurs restent mappées sur le contrat stable `ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID` avec conditions internes stables, sans valeurs sensibles.
|
||||||
|
|
||||||
|
## Gate volontaire d'activation multi-source
|
||||||
|
|
||||||
|
Le plan final exige que toutes les sources configurées soient démarrées simultanément. Cette responsabilité est explicitement réservée à `pre.007`.
|
||||||
|
|
||||||
|
Pour éviter un comportement faux en `pre.002` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
1 source Yellowstone : voie productive stable conservée
|
||||||
|
2..32 sources : composition/validation possible
|
||||||
|
start avec 2..32 sources : fail closed avant spawn
|
||||||
|
jamais de démarrage silencieux de la première source seulement
|
||||||
|
```
|
||||||
|
|
||||||
|
Condition interne :
|
||||||
|
|
||||||
|
```text
|
||||||
|
runtime_resources.multi_source_activation_pending
|
||||||
|
```
|
||||||
|
|
||||||
|
Ce gate sera supprimé uniquement par la tranche supervisor/source inventory.
|
||||||
|
|
||||||
|
## Preuves déterministes ajoutées
|
||||||
|
|
||||||
|
Transport :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identité de request stable malgré ordre d'insertion
|
||||||
|
identité différente si le filtre logique change
|
||||||
|
Debug identité redacted
|
||||||
|
surface identité disponible depuis crate root
|
||||||
|
aucun bytes()/as_bytes() public
|
||||||
|
```
|
||||||
|
|
||||||
|
Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
source_key stable pour identité logique identique
|
||||||
|
source_key sensible au request fingerprint
|
||||||
|
source_key sensible à l'endpoint identity sûre
|
||||||
|
32 sources exactement acceptées
|
||||||
|
33e rejetée sans mutation
|
||||||
|
duplicate source identity rejetée
|
||||||
|
source cross-network rejetée
|
||||||
|
Debug aggregate limité à source_count
|
||||||
|
activation multi-source fail-closed avant pre.007
|
||||||
|
public API canary pour bound/count/push
|
||||||
|
release export inventory mis à jour
|
||||||
|
```
|
||||||
|
|
||||||
|
## Compatibilité et non-régression
|
||||||
|
|
||||||
|
La construction historique :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestRuntimeResources::new(yellowstone_source)
|
||||||
|
```
|
||||||
|
|
||||||
|
reste valide et crée un aggregate d'une source.
|
||||||
|
|
||||||
|
La voie productive stable `start_with_runtime_resources` continue donc à fonctionner pour une seule source Yellowstone sans changement de Common RAW, Store, Config ni protocole réseau.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-onchain-transport-lib/src/grpc_subscribe.rs
|
||||||
|
crates/ksp-onchain-transport-lib/src/lib.rs
|
||||||
|
crates/ksp-onchain-transport-lib/tests/public_api.rs
|
||||||
|
crates/ksp-onchain-transport-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-onchain-transport-lib/unit_tests/grpc_subscribe.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md
|
||||||
|
docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Versions de fichiers incrémentées
|
||||||
|
|
||||||
|
```text
|
||||||
|
grpc_subscribe.rs transport src : 8 -> 9
|
||||||
|
transport lib.rs : 46 -> 47
|
||||||
|
transport public_api.rs : 51 -> 52
|
||||||
|
transport release_completeness.rs : 43 -> 44
|
||||||
|
transport unit grpc_subscribe.rs : 5 -> 6
|
||||||
|
worker lib.rs : 17 -> 18
|
||||||
|
worker runtime.rs : 11 -> 12
|
||||||
|
worker runtime_resources.rs : 12 -> 13
|
||||||
|
worker public_api.rs : 11 -> 12
|
||||||
|
worker release_completeness.rs : 9 -> 10
|
||||||
|
worker unit runtime_resources.rs : 11 -> 12
|
||||||
|
plan 034 : 1 -> 2
|
||||||
|
validation 030 : 1 -> 2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Audit intermédiaire et corrections
|
||||||
|
|
||||||
|
La première passe statique après introduction de l'identité Transport a détecté une rustdoc devenue non adjacente au `YellowstoneSubscribeRequest`. La structure documentaire a été corrigée avant poursuite.
|
||||||
|
|
||||||
|
Une passe ultérieure après ajout des canaris externes a détecté uniquement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
2 fins de fichier avec newline supplémentaire
|
||||||
|
1 double ligne vide entre items Rust
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces trois écarts de format ont été corrigés avant la validation finale.
|
||||||
|
|
||||||
|
Aucun état intermédiaire en échec n'est présenté comme PASS.
|
||||||
|
|
||||||
|
## Validations exécutées localement après correction finale
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
PASS : General Rust rule audit clean
|
||||||
|
PASS : Rust export completeness audit 0 candidate
|
||||||
|
PASS : KSP workspace Rust rule audit clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
PASS : Markdown table audit clean (340 tables, 829 files)
|
||||||
|
|
||||||
|
supplemental normative definition-id scan
|
||||||
|
PASS : 489 definitions, 489 unique, 0 duplicate definition id
|
||||||
|
|
||||||
|
pre-packaging exhaustive diff scan
|
||||||
|
PASS : 14 fichiers modifiés, 1 ajouté, 0 supprimé
|
||||||
|
PASS : chaque header de version des 13 fichiers versionnés modifiés est incrémenté exactement de +1
|
||||||
|
PASS : aucune nouvelle occurrence de unsafe/unwrap/expect/panic/todo/unimplemented dans les lignes Rust production ajoutées
|
||||||
|
```
|
||||||
|
|
||||||
|
Les caches Python recréés par les scripts d'audit sont supprimés après la dernière passe et ne font pas partie de la livraison. L'archive delta est ensuite contrôlée par inventaire exact, test ZIP et scan des chemins/artefacts.
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
Le toolchain Rust n'est pas installé dans l'environnement d'assemblage courant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
Statut : `NON EXÉCUTÉ LOCAL`, jamais PASS.
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
conserver 1..32 comme borne runtime P0
|
||||||
|
source identity opaque et non publique côté Worker
|
||||||
|
faire posséder l'identité exacte SubscribeRequest par Transport
|
||||||
|
ne pas ajouter prost comme dépendance Worker/Transport supplémentaire
|
||||||
|
valider toutes les sources avant spawn
|
||||||
|
rejeter duplicate identity et cross-network avant mutation
|
||||||
|
fail closed pour activation >1 jusqu'à pre.007
|
||||||
|
ne pas démarrer un sous-ensemble silencieux
|
||||||
|
ne modifier ni Common RAW ni Store API
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question bloquante pour `pre.003`.
|
||||||
|
|
||||||
|
Restent volontairement dans leurs tranches prévues :
|
||||||
|
|
||||||
|
```text
|
||||||
|
logsSubscribe standard + hydration : pre.003
|
||||||
|
blockSubscribe standard : pre.004
|
||||||
|
Helius transactionSubscribe : pre.005
|
||||||
|
HTTP polling : pre.006
|
||||||
|
supervisor simultané / source inventory : pre.007
|
||||||
|
convergence globale / observations multiples : pre.008
|
||||||
|
```
|
||||||
210
deltas/0.3.13/pre.003-fix.001.md
Normal file
210
deltas/0.3.13/pre.003-fix.001.md
Normal file
@@ -0,0 +1,210 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.003-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.003-fix.001 — correction des tests Worker après généralisation source-neutral
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
delivery précédente : 0.3.13-pre.003
|
||||||
|
Cargo base : 0.3.13-pre.3
|
||||||
|
delta base : deltas/0.3.13/pre.003.md
|
||||||
|
archive delta base : ksp-general-0.3.13-pre.003.zip
|
||||||
|
SHA-256 delta base : 669b875410799b8a2c5806cddad81740bdd4dd78e36c68f6b268a6b0257f18f4
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.003` confirme que le formatage, les audits et `cargo check --workspace` sont propres, mais que les cibles de tests Worker ne compilent plus après la généralisation source-neutral de `runtime_resources`. `cargo clippy --workspace --all-targets --all-features -- -D warnings` et `cargo test -p ksp-worker-raw-transaction-ingest-lib` échouent sur sept diagnostics provenant du même fichier de tests.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement les tests devenus obsolètes par rapport au contrat production de `pre.003`, sans rouvrir l'API ni modifier la logique runtime :
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun code production modifié
|
||||||
|
aucune API publique ajoutée ou supprimée
|
||||||
|
aucune dépendance modifiée
|
||||||
|
aucune logique réseau/hydration/admission modifiée
|
||||||
|
aucun avancement vers blockSubscribe pre.004
|
||||||
|
```
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.003-fix.001
|
||||||
|
workspace.package.version : 0.3.13-pre.3.fix.1
|
||||||
|
archive : ksp-general-0.3.13-pre.003-fix.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Conformément aux règles de versionnement KSP, le fix modifie un fichier `.rs`; le `workspace.package.version` est donc synchronisé avec l'identifiant SemVer technique du correctif.
|
||||||
|
|
||||||
|
## Diagnostic opérateur
|
||||||
|
|
||||||
|
Les sept diagnostics se répartissent en trois causes.
|
||||||
|
|
||||||
|
### Ancien extracteur mono-Yellowstone supprimé
|
||||||
|
|
||||||
|
Le test `v0_3_13_pre_002_multi_source_activation_fails_closed_until_supervisor_tranche` appelait encore :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestRuntimeResources::into_yellowstone_source
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette méthode a été supprimée en `pre.003` lorsque l'aggregate a été rendu source-neutral. La barrière correspondante est désormais :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestRuntimeResources::validate_single_source_activation
|
||||||
|
```
|
||||||
|
|
||||||
|
Le test utilise donc le contrat actuel et continue de vérifier exactement l'erreur :
|
||||||
|
|
||||||
|
```text
|
||||||
|
runtime_resources.multi_source_activation_pending
|
||||||
|
```
|
||||||
|
|
||||||
|
Les deux erreurs d'inférence `E0282` étaient secondaires à l'absence de l'ancienne méthode et disparaissent avec cette correction.
|
||||||
|
|
||||||
|
### Contexte d'hydration manquant
|
||||||
|
|
||||||
|
Deux tests appelaient encore l'ancienne signature de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
build_hydration_provenance
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.003` a ajouté en premier argument le contexte source-neutral :
|
||||||
|
|
||||||
|
```text
|
||||||
|
&RawTransactionIngestHydrationContext
|
||||||
|
```
|
||||||
|
|
||||||
|
Les tests construisent maintenant ce contexte via :
|
||||||
|
|
||||||
|
```text
|
||||||
|
source.hydration_context()
|
||||||
|
```
|
||||||
|
|
||||||
|
puis le transmettent à `build_hydration_provenance`. Aucun contrat production n'est élargi pour les tests.
|
||||||
|
|
||||||
|
### Accesseurs volontairement absents de RawTransactionMaterial
|
||||||
|
|
||||||
|
Le nouveau test Standard Logs tentait d'appeler :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ingress.material.slot()
|
||||||
|
ingress.material.signature()
|
||||||
|
```
|
||||||
|
|
||||||
|
`RawTransactionMaterial` garde ces champs privés et ne fournit volontairement pas ces accesseurs. Le test ne doit pas provoquer l'ajout d'une API uniquement pour introspection.
|
||||||
|
|
||||||
|
Les deux assertions sont donc supprimées à ce niveau. La même preuve reste effectuée après passage dans l'admission centrale et canonicalisation, via :
|
||||||
|
|
||||||
|
```text
|
||||||
|
acquisition.transaction().slot() == 42
|
||||||
|
acquisition.transaction().reference().signature().as_bytes() == [0; 64]
|
||||||
|
```
|
||||||
|
|
||||||
|
Cela vérifie en plus le chemin réellement consommé par le Worker.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.003-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Versions de fichiers incrémentées
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml : 542 -> 543
|
||||||
|
worker unit_tests/runtime_resources.rs : 13 -> 14
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation opérateur de la base pre.003
|
||||||
|
|
||||||
|
Le log fourni confirme avant fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all : exécuté
|
||||||
|
cargo fmt --all -- --check : PASS / silencieux
|
||||||
|
General Rust rule audit : clean
|
||||||
|
Rust export completeness audit : 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit : clean
|
||||||
|
Markdown table audit : clean (340 tables, 831 files)
|
||||||
|
cargo check --workspace : PASS
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings : FAIL sur compilation des tests Worker
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib : FAIL sur les mêmes 7 diagnostics
|
||||||
|
```
|
||||||
|
|
||||||
|
Diagnostics observés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
E0599 : into_yellowstone_source supprimé
|
||||||
|
E0282 : 2 erreurs d'inférence secondaires
|
||||||
|
E0061 : 2 appels build_hydration_provenance sans hydration context
|
||||||
|
E0599 : RawTransactionMaterial::slot absent
|
||||||
|
E0599 : RawTransactionMaterial::signature absent
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Le toolchain Rust n'est pas disponible dans l'environnement d'assemblage. Les validations statiques KSP exécutables localement et le contrôle exhaustif du delta sont réalisés avant packaging.
|
||||||
|
|
||||||
|
Les commandes Cargo ne sont pas déclarées PASS localement :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Résultats statiques finaux de l'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 tables, 832 files)
|
||||||
|
Normative rule definitions: 489
|
||||||
|
Unique normative IDs: 489
|
||||||
|
Duplicates: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate opérateur avant pre.004
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.004` ne doit démarrer qu'après retour vert de ce gate.
|
||||||
|
|
||||||
|
## Décisions
|
||||||
|
|
||||||
|
- Le fix conserve l'encapsulation de `RawTransactionMaterial`; aucun getter n'est ajouté pour satisfaire un test.
|
||||||
|
- Le fix conserve l'aggregate source-neutral introduit en `pre.003`; l'ancien extracteur mono-Yellowstone n'est pas restauré.
|
||||||
|
- Le fix conserve le contexte d'hydration source-neutral et adapte les tests à sa signature actuelle.
|
||||||
|
- Aucun changement fonctionnel de `pre.003` n'est introduit.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune
|
||||||
|
```
|
||||||
194
deltas/0.3.13/pre.003-fix.002.md
Normal file
194
deltas/0.3.13/pre.003-fix.002.md
Normal file
@@ -0,0 +1,194 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.003-fix.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.003-fix.002 — correction des canaris de dépendance après généralisation source-neutral
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
delivery précédente : 0.3.13-pre.003-fix.001
|
||||||
|
Cargo base : 0.3.13-pre.3.fix.1
|
||||||
|
delta base : deltas/0.3.13/pre.003-fix.001.md
|
||||||
|
archive delta base : ksp-general-0.3.13-pre.003-fix.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.003-fix.001` confirme que le formatage, les audits, `cargo check --workspace`, Clippy strict et les tests unitaires Worker passent. Deux canaris historiques de `tests/dependency_boundary.rs` échouent cependant parce qu'ils vérifient encore des noms et une borne d'implémentation antérieurs à la généralisation source-neutral de `pre.003`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement ces deux canaris afin qu'ils vérifient le contrat production actuel, sans modifier le code runtime :
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun code production modifié
|
||||||
|
aucune API publique ajoutée ou supprimée
|
||||||
|
aucune dépendance modifiée
|
||||||
|
aucune logique réseau/hydration/admission modifiée
|
||||||
|
aucun avancement vers blockSubscribe pre.004
|
||||||
|
```
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.003-fix.002
|
||||||
|
workspace.package.version : 0.3.13-pre.3.fix.2
|
||||||
|
archive : ksp-general-0.3.13-pre.003-fix.002.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le fix modifie un fichier Rust de tests ; le `workspace.package.version` est donc incrémenté conformément aux règles KSP.
|
||||||
|
|
||||||
|
## Diagnostic opérateur
|
||||||
|
|
||||||
|
### Canary hydration historique
|
||||||
|
|
||||||
|
Le test :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0_3_12_pre_004_hydration_contract_uses_only_transport_facade_common_raw_and_existing_ingress
|
||||||
|
```
|
||||||
|
|
||||||
|
attendait encore les helpers mono-Yellowstone :
|
||||||
|
|
||||||
|
```text
|
||||||
|
fn finalize_yellowstone_hydration
|
||||||
|
fn fetch_yellowstone_hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.003` a volontairement rendu l'hydration commune à plusieurs familles de sources et les helpers production actuels sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
fn finalize_hydration
|
||||||
|
async fn fetch_hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
Le canary est mis à jour pour vérifier ces helpers source-neutral sans réintroduire les anciens noms.
|
||||||
|
|
||||||
|
### Canary de borne de coalescence historique
|
||||||
|
|
||||||
|
Le test :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0_3_12_pre_006_productive_source_uses_transport_session_bounded_coalescence_and_existing_admission
|
||||||
|
```
|
||||||
|
|
||||||
|
attendait encore la présence textuelle de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.003` a resserré le budget du coordinator commun sur la borne configurée du Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
max_pending_signals: settings.admission_queue_capacity()
|
||||||
|
```
|
||||||
|
|
||||||
|
Le canary vérifie désormais cette borne runtime effective. Le plafond absolu n'est pas réintroduit dans l'implémentation.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.003-fix.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Versions de fichiers incrémentées
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml : 543 -> 544
|
||||||
|
worker tests/dependency_boundary.rs : 17 -> 18
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation opérateur de la base pre.003-fix.001
|
||||||
|
|
||||||
|
Le log fourni confirme avant ce fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all : exécuté
|
||||||
|
cargo fmt --all -- --check : PASS / silencieux
|
||||||
|
General Rust rule audit : clean
|
||||||
|
Rust export completeness audit : 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit : clean
|
||||||
|
Markdown table audit : clean (340 tables, 832 files)
|
||||||
|
cargo check --workspace : PASS
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings : FAIL lors des tests dependency_boundary
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib : 75 unit tests PASS, puis 2/10 dependency_boundary FAIL
|
||||||
|
```
|
||||||
|
|
||||||
|
Échecs observés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0_3_12_pre_004_hydration_contract_uses_only_transport_facade_common_raw_and_existing_ingress
|
||||||
|
v0_3_12_pre_006_productive_source_uses_transport_session_bounded_coalescence_and_existing_admission
|
||||||
|
```
|
||||||
|
|
||||||
|
Les deux échecs sont des assertions `include_str!` sur l'implémentation et non des erreurs du code production.
|
||||||
|
|
||||||
|
## Validations dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Le toolchain Rust n'est pas disponible dans l'environnement d'assemblage. Les audits statiques KSP et le contrôle exhaustif du delta sont exécutés avant packaging. Les commandes Cargo suivantes ne sont donc pas déclarées PASS localement :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Résultats statiques finaux de l'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 tables, 833 files)
|
||||||
|
Normative rule definitions: 489
|
||||||
|
Unique normative IDs: 489
|
||||||
|
Duplicates: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Les trois chaînes attendues par les canaris corrigés ont également été vérifiées directement dans `runtime_resources.rs` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
fn finalize_hydration : présent
|
||||||
|
fn fetch_hydration : présent
|
||||||
|
max_pending_signals: settings.admission_queue_capacity() : présent
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate opérateur avant pre.004
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.004` ne doit démarrer qu'après retour vert de ce gate.
|
||||||
|
|
||||||
|
## Décisions
|
||||||
|
|
||||||
|
- Le fix conserve les helpers d'hydration source-neutral de `pre.003`; les anciens noms Yellowstone ne sont pas restaurés.
|
||||||
|
- Le fix conserve `settings.admission_queue_capacity()` comme borne effective du coordinator d'hydration ; le plafond absolu n'est pas réinjecté.
|
||||||
|
- Aucun code production n'est modifié.
|
||||||
|
- Aucun changement fonctionnel de `pre.003` n'est introduit.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune
|
||||||
|
```
|
||||||
164
deltas/0.3.13/pre.003-fix.003.md
Normal file
164
deltas/0.3.13/pre.003-fix.003.md
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.003-fix.003.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.003-fix.003 — correction du dernier canari productif historique
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
delivery précédente : 0.3.13-pre.003-fix.002
|
||||||
|
Cargo base : 0.3.13-pre.3.fix.2
|
||||||
|
delta base : deltas/0.3.13/pre.003-fix.002.md
|
||||||
|
archive delta base : ksp-general-0.3.13-pre.003-fix.002.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.003-fix.002` confirme que le formatage, les audits, `cargo check --workspace`, les 75 tests unitaires Worker et 9/10 canaris `dependency_boundary` passent. Un seul canari historique reste obsolète : il recherche encore la chaîne `commitment: commitment.as_str()` alors que `pre.003` a déplacé le commitment dans le contexte d'hydration source-neutral.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement cette assertion textuelle de test afin qu'elle vérifie le contrat production actuel :
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun code production modifié
|
||||||
|
aucune API publique ajoutée ou supprimée
|
||||||
|
aucune dépendance modifiée
|
||||||
|
aucune logique réseau/hydration/admission modifiée
|
||||||
|
aucun avancement vers blockSubscribe pre.004
|
||||||
|
```
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.003-fix.003
|
||||||
|
workspace.package.version : 0.3.13-pre.3.fix.3
|
||||||
|
archive : ksp-general-0.3.13-pre.003-fix.003.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le fix modifie un fichier Rust de tests ; le `workspace.package.version` est donc incrémenté conformément aux règles KSP.
|
||||||
|
|
||||||
|
## Diagnostic opérateur
|
||||||
|
|
||||||
|
Le test :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0_3_12_pre_006_productive_source_uses_transport_session_bounded_coalescence_and_existing_admission
|
||||||
|
```
|
||||||
|
|
||||||
|
attendait encore :
|
||||||
|
|
||||||
|
```text
|
||||||
|
commitment: commitment.as_str()
|
||||||
|
```
|
||||||
|
|
||||||
|
Le coordinator d'hydration source-neutral de `pre.003` porte désormais le commitment dans `RawTransactionIngestHydrationContext`. Le code production courant construit donc la clé avec :
|
||||||
|
|
||||||
|
```text
|
||||||
|
commitment: hydration.commitment.as_str()
|
||||||
|
```
|
||||||
|
|
||||||
|
Le canari est mis à jour pour vérifier cette forme actuelle. Aucun alias, variable ou code de compatibilité artificiel n'est ajouté au runtime.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.003-fix.003.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Versions de fichiers incrémentées
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml : 544 -> 545
|
||||||
|
worker tests/dependency_boundary.rs : 18 -> 19
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation opérateur de la base pre.003-fix.002
|
||||||
|
|
||||||
|
Le log fourni confirme avant ce fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all : exécuté
|
||||||
|
cargo fmt --all -- --check : PASS / silencieux
|
||||||
|
General Rust rule audit : clean
|
||||||
|
Rust export completeness audit : 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit : clean
|
||||||
|
Markdown table audit : clean (340 tables, 833 files)
|
||||||
|
cargo check --workspace : PASS
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings : atteint les tests puis échoue sur le même canari dependency_boundary
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib : 75 unit tests PASS, puis 9/10 dependency_boundary PASS et 1 FAIL
|
||||||
|
```
|
||||||
|
|
||||||
|
Échec restant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
required pre.006 productive-source contract missing: commitment: commitment.as_str()
|
||||||
|
```
|
||||||
|
|
||||||
|
Il s'agit d'une assertion `include_str!` obsolète sur l'implémentation, pas d'une erreur du code production.
|
||||||
|
|
||||||
|
## Validations dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Le toolchain Rust n'est pas disponible dans l'environnement d'assemblage. Les audits statiques KSP et le contrôle exhaustif du delta sont exécutés avant packaging. Les commandes Cargo suivantes ne sont donc pas déclarées PASS localement :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Résultats statiques finaux de l'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 tables, 834 files)
|
||||||
|
Normative rule definitions: 489
|
||||||
|
Unique normative IDs: 489
|
||||||
|
Duplicates: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
La chaîne attendue par le canari corrigé a également été vérifiée directement dans `runtime_resources.rs` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
commitment: hydration.commitment.as_str() : présent
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate opérateur avant pre.004
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.004` ne doit démarrer qu'après retour vert de ce gate.
|
||||||
|
|
||||||
|
## Décisions
|
||||||
|
|
||||||
|
- Le commitment reste porté par le contexte d'hydration source-neutral de `pre.003`.
|
||||||
|
- Le canari historique est aligné sur le contrat actuel ; aucun code production n'est modifié pour satisfaire une recherche textuelle obsolète.
|
||||||
|
- Aucun changement fonctionnel de `pre.003` n'est introduit.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune
|
||||||
|
```
|
||||||
260
deltas/0.3.13/pre.003.md
Normal file
260
deltas/0.3.13/pre.003.md
Normal file
@@ -0,0 +1,260 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.003.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.003 — Standard Solana logsSubscribe + hydration commune
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
delivery précédente : 0.3.13-pre.002-fix.001
|
||||||
|
Cargo base : 0.3.13-pre.2.fix.1
|
||||||
|
delta base : deltas/0.3.13/pre.002-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de la base a été exécuté après `cargo clean` et est vert : `cargo fmt --all -- --check` silencieux, audits Rust/Markdown propres, `cargo check --workspace`, Clippy strict, 389 tests unitaires Transport, 52 tests public API Transport, 44 tests release Transport, 4 doc-tests Transport, puis 70 tests unitaires Worker et toutes ses suites d'intégration sans échec. Les smokes réseau opt-in sont restés ignorés comme prévu.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Ajouter la première nouvelle famille live de `0.3.13` sans dupliquer Transport ni l'admission RAW :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Solana standard WS logsSubscribe
|
||||||
|
-> context.slot + signature
|
||||||
|
-> hydration HTTP getTransaction observed
|
||||||
|
-> Common RAW
|
||||||
|
-> admission centrale Worker
|
||||||
|
-> Store
|
||||||
|
```
|
||||||
|
|
||||||
|
La tranche ne démarre toujours qu'une seule source productive à la fois. La supervision simultanée de plusieurs sources reste réservée à `pre.007`.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.003
|
||||||
|
workspace.package.version : 0.3.13-pre.3
|
||||||
|
archive attendue : ksp-general-0.3.13-pre.003.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
## Source Standard Logs
|
||||||
|
|
||||||
|
Nouvelle surface publique Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestStandardLogsSource
|
||||||
|
```
|
||||||
|
|
||||||
|
Construction caller-composed :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WsEndpointSettings kind solana_standard
|
||||||
|
SolanaLogsSubscribeFilter
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction ne réalise aucune I/O. Elle valide le protocole WebSocket, le commitment, le réseau, la représentation sûre provider/endpoint et l'existence d'une route HTTP `getTransaction` compatible sur le même cluster.
|
||||||
|
|
||||||
|
Au runtime, la source utilise exclusivement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SolanaStandardWsSession::connect
|
||||||
|
logs_subscribe
|
||||||
|
HttpTransportPool::get_transaction_observed
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker ne construit aucun socket/client inférieur et ne dépend directement ni de `reqwest`, ni de `tokio-tungstenite`, ni de `tonic`, ni de `yellowstone-grpc-proto`.
|
||||||
|
|
||||||
|
## Projection et redaction
|
||||||
|
|
||||||
|
Une notification `logsSubscribe` est réduite à son matériau de référence nécessaire :
|
||||||
|
|
||||||
|
```text
|
||||||
|
context.slot
|
||||||
|
signature
|
||||||
|
commitment caller-composed
|
||||||
|
identité source/filter privée
|
||||||
|
```
|
||||||
|
|
||||||
|
Les champs distants `logs` et `err` ne sont pas copiés dans le signal Worker, le snapshot, la provenance textuelle ou `Debug`.
|
||||||
|
|
||||||
|
Les variantes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
All
|
||||||
|
AllWithVotes
|
||||||
|
Mentions(pubkey)
|
||||||
|
```
|
||||||
|
|
||||||
|
participent à une empreinte SHA-256 privée. La valeur du pubkey `Mentions` n'est pas exposée par le Worker.
|
||||||
|
|
||||||
|
## Identité logique
|
||||||
|
|
||||||
|
Le `source_key` Standard Logs est dérivé de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
domain KSP live-source v1
|
||||||
|
family = standard_logs
|
||||||
|
network
|
||||||
|
safe provider identity
|
||||||
|
safe endpoint identity
|
||||||
|
commitment
|
||||||
|
private filter fingerprint
|
||||||
|
```
|
||||||
|
|
||||||
|
Le rôle et le pool HTTP d'hydration n'entrent pas dans cette identité : ils décrivent la stratégie d'enrichissement, pas une nouvelle source live.
|
||||||
|
|
||||||
|
Le rejet transactionnel des doublons et le bound global `1..32` introduits en `pre.002` s'appliquent aussi à Standard Logs.
|
||||||
|
|
||||||
|
## Hydration source-neutral
|
||||||
|
|
||||||
|
Le coordinator privé précédemment utilisé par Yellowstone est généralisé en contrat d'hydration source-neutral et réutilisé par Standard Logs.
|
||||||
|
|
||||||
|
Pour `pre.003`, chaque exécution mono-source possède encore sa propre instance. Le partage réellement global entre plusieurs tâches source sera introduit avec le supervisor `pre.007`.
|
||||||
|
|
||||||
|
Bornes effectives :
|
||||||
|
|
||||||
|
```text
|
||||||
|
hydration in-flight <= persistence_concurrency
|
||||||
|
pending source signals <= admission_queue_capacity
|
||||||
|
```
|
||||||
|
|
||||||
|
Le plafond pending est donc aligné sur les settings effectifs du Worker plutôt que sur le plafond absolu maximal.
|
||||||
|
|
||||||
|
La requête commune `getTransaction observed` utilise maintenant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
encoding = base64
|
||||||
|
maxSupportedTransactionVersion = 1
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette adaptation s'applique également à Yellowstone. Elle prépare l'hydration aux transactions Legacy/V0/V1 mais ne qualifie aucun nouveau chemin RAW-direct V1.
|
||||||
|
|
||||||
|
## Activation runtime
|
||||||
|
|
||||||
|
Le runtime-resource aggregate peut composer Yellowstone et Standard Logs dans la collection privée bornée, mais la barrière de sûreté reste :
|
||||||
|
|
||||||
|
```text
|
||||||
|
1 source Yellowstone : productif
|
||||||
|
1 source Standard Logs : productif
|
||||||
|
2..32 sources : composition validable
|
||||||
|
start 2..32 : runtime_resources.multi_source_activation_pending avant spawn
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune source configurée n'est silencieusement ignorée.
|
||||||
|
|
||||||
|
## Preuves ajoutées
|
||||||
|
|
||||||
|
Unit tests Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
validation standard WS / commitment / route HTTP
|
||||||
|
source_key stable et filter-sensitive
|
||||||
|
source_key indépendant du hydration role
|
||||||
|
All / AllWithVotes / Mentions distingués
|
||||||
|
Mentions/URL/source-key bytes redacted
|
||||||
|
projection reference-only
|
||||||
|
fixture signature+slot -> une hydration HTTP -> matériau Common RAW exact
|
||||||
|
getTransaction base64 + maxSupportedTransactionVersion=1
|
||||||
|
rejet duplicate Standard Logs identity
|
||||||
|
```
|
||||||
|
|
||||||
|
Canaris externes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
public_api : nouvelle construction/composition accessible depuis crate root
|
||||||
|
dependency_boundary : uniquement façades Transport + Common RAW/Store existants
|
||||||
|
hardening : aucune copie logs/err/payload/url et Debug borné
|
||||||
|
release_completeness : nouvelle surface et canaris pre.003 obligatoires
|
||||||
|
```
|
||||||
|
|
||||||
|
Reconnect, resubscribe, IDs distants et backpressure WebSocket restent couverts et possédés par `ksp-onchain-transport-lib`; le Worker réutilise la façade `SolanaStandardWsSession` au lieu de reproduire ces mécanismes.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/README.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md
|
||||||
|
docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.003.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
`pre.003` ne prétend pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
blockSubscribe productif
|
||||||
|
Helius transactionSubscribe productif
|
||||||
|
HTTP live block polling
|
||||||
|
supervision simultanée de plusieurs sources
|
||||||
|
fanout Store d'observations multi-source
|
||||||
|
coordinator partagé entre plusieurs tâches source simultanées
|
||||||
|
gap repair/backfill
|
||||||
|
nouvelle dépendance externe
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Les contrôles réellement exécutés après finalisation de la tranche sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
PASS : General Rust rule audit clean
|
||||||
|
PASS : Rust export completeness audit 0 candidate
|
||||||
|
PASS : KSP workspace Rust rule audit clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
PASS : Markdown table audit clean (340 tables, 831 files)
|
||||||
|
|
||||||
|
supplemental normative definition-id scan
|
||||||
|
PASS : 489 definitions, 489 unique, 0 duplicate definition id
|
||||||
|
|
||||||
|
pre-packaging exhaustive diff scan
|
||||||
|
PASS : 13 fichiers modifiés, 1 ajouté, 0 supprimé
|
||||||
|
PASS : les 13 headers versionnés des fichiers modifiés sont incrémentés exactement de +1
|
||||||
|
PASS : aucune dépendance Cargo nouvelle
|
||||||
|
```
|
||||||
|
|
||||||
|
Le toolchain Rust/Cargo/Rustfmt n'est pas disponible dans cet environnement. Les commandes Cargo post-modification sont donc explicitement `NON EXÉCUTÉ LOCAL`, jamais déclarées PASS.
|
||||||
|
|
||||||
|
Les caches Python produits par les audits sont supprimés avant packaging et ne font pas partie du delta.
|
||||||
|
|
||||||
|
## Gate opérateur requis avant pre.004
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.004` reste bloquée si ce gate révèle une anomalie de format, règle, compilation, API, dépendance ou test.
|
||||||
94
deltas/0.3.13/pre.004-fix.001.md
Normal file
94
deltas/0.3.13/pre.004-fix.001.md
Normal file
@@ -0,0 +1,94 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.004-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.004-fix.001 — retrait de la dépendance serde_json accidentelle
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison précédente : 0.3.13-pre.004
|
||||||
|
Cargo base : 0.3.13-pre.4
|
||||||
|
delta base : deltas/0.3.13/pre.004.md
|
||||||
|
archive delta base : ksp-general-0.3.13-pre.004.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.004` a révélé deux erreurs de compilation liées au même défaut : le Worker nommait directement `serde_json::Value` dans `runtime_resources.rs` alors que `serde_json` n'est pas une dépendance du crate, et le test de qualification Standard Block utilisait directement `serde_json::json!`.
|
||||||
|
|
||||||
|
## Objectif du fix
|
||||||
|
|
||||||
|
Supprimer cette dépendance accidentelle sans ajouter `serde_json` au manifeste du Worker et sans modifier le comportement productif de Standard Block.
|
||||||
|
|
||||||
|
Le chemin production conserve exactement le `meta` décodé par `ksp-onchain-transport-lib`, mais son type concret reste inféré depuis `SolanaBlockTransaction::meta()` et n'est plus nommé par le Worker.
|
||||||
|
|
||||||
|
Le helper de qualification reçoit désormais seulement la version, la position et une closure de construction du matériau. La closure production capture le `meta` opaque et le clone vers le Common RAW. La fixture Legacy/V0/V1 utilise un `RawTransactionWireField::Null`, car elle vérifie la signature embarquée, le slot, le `blockTime`, la version et le `transactionIndex`, pas le contenu de `meta`.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.004-fix.001
|
||||||
|
workspace.package.version : 0.3.13-pre.4.fix.1
|
||||||
|
archive : ksp-general-0.3.13-pre.004-fix.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
`Cargo.toml` passe de la version de fichier `546` à `547`.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
Les deux fichiers Rust passent de la version de fichier `15` à `16`.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.004-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
Ce fix n'ajoute aucune dépendance, ne modifie aucune API publique et ne change ni la qualification Legacy/V0/V1, ni l'admission RAW, ni la frontier, ni le comportement réseau de `blockSubscribe`.
|
||||||
|
|
||||||
|
## Validation dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Le toolchain Rust/Cargo/Rustfmt n'est pas disponible dans l'environnement d'assemblage. Les commandes Cargo post-fix restent donc `NON EXÉCUTÉ LOCAL`. Les audits statiques KSP et le contrôle exhaustif du delta sont exécutés avant packaging.
|
||||||
|
|
||||||
|
## Gate opérateur requis avant pre.005
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Résultats statiques finaux de l'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 tables, 836 files)
|
||||||
|
```
|
||||||
|
|
||||||
|
Contrôle exhaustif du delta :
|
||||||
|
|
||||||
|
```text
|
||||||
|
3 fichiers existants modifiés
|
||||||
|
1 fichier ajouté
|
||||||
|
0 fichier supprimé
|
||||||
|
aucune dépendance ajoutée
|
||||||
|
aucune référence directe serde_json dans les deux fichiers Rust corrigés
|
||||||
|
```
|
||||||
267
deltas/0.3.13/pre.004.md
Normal file
267
deltas/0.3.13/pre.004.md
Normal file
@@ -0,0 +1,267 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.004.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.004 — Standard Solana blockSubscribe RAW-direct qualifié
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
delivery précédente : 0.3.13-pre.003-fix.003
|
||||||
|
Cargo base : 0.3.13-pre.3.fix.3
|
||||||
|
delta base : deltas/0.3.13/pre.003-fix.003.md
|
||||||
|
archive delta base : ksp-general-0.3.13-pre.003-fix.003.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur reçu sur cette base est vert pour toutes les commandes exécutées : `cargo fmt`, `cargo fmt --check`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict et toutes les suites de `ksp-worker-raw-transaction-ingest-lib` (`75` unit, `4` cross-layer, `10` dependency-boundary, `17` hardening, `11` public-api, `4` release-completeness, `0` doc-test). Cette exécution ne contenait pas le test Transport ni les commandes `cargo tree`; ils ne sont donc pas déclarés PASS pour ce gate précis.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Ajouter la deuxième source Solana Standard WebSocket de `0.3.13` sans dupliquer Transport :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Solana standard WS blockSubscribe
|
||||||
|
-> Full + Base64 + maxSupportedTransactionVersion=1 + showRewards=false
|
||||||
|
-> qualification explicite Legacy / V0 / V1
|
||||||
|
-> Common RAW direct par transaction
|
||||||
|
-> admission centrale Worker
|
||||||
|
-> Store
|
||||||
|
```
|
||||||
|
|
||||||
|
La source ne possède pas de fallback HTTP. Tout bloc ou transaction non qualifiable échoue explicitement et sûrement.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.004
|
||||||
|
workspace.package.version : 0.3.13-pre.4
|
||||||
|
archive : ksp-general-0.3.13-pre.004.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
`Cargo.toml` passe également de la version de fichier `545` à `546`.
|
||||||
|
|
||||||
|
## Source Standard Block
|
||||||
|
|
||||||
|
Nouvelle surface publique Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestStandardBlockSource
|
||||||
|
RawTransactionIngestRuntimeResources::from_standard_block_source
|
||||||
|
RawTransactionIngestRuntimeResources::try_push_standard_block_source
|
||||||
|
```
|
||||||
|
|
||||||
|
Construction caller-composed :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WsEndpointSettings kind solana_standard
|
||||||
|
SolanaBlockSubscribeFilter
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction est sans I/O. Elle valide le protocole, le commitment, le réseau et la représentation sûre provider/endpoint avant spawn.
|
||||||
|
|
||||||
|
Au runtime, la source réutilise exclusivement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SolanaStandardWsSession::connect
|
||||||
|
SolanaStandardWsSession::block_subscribe
|
||||||
|
```
|
||||||
|
|
||||||
|
Configuration exacte :
|
||||||
|
|
||||||
|
```text
|
||||||
|
encoding = Base64
|
||||||
|
transactionDetails = Full
|
||||||
|
maxSupportedTransactionVersion = 1
|
||||||
|
showRewards = false
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun client WebSocket direct, `getBlock`, `getTransaction`, `reqwest`, `tokio-tungstenite`, `tonic` ou SDK provider n'est ajouté au Worker.
|
||||||
|
|
||||||
|
## Qualification RAW-direct
|
||||||
|
|
||||||
|
La version est fail-closed :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Legacy -> RawTransactionVersion::Legacy
|
||||||
|
0 -> RawTransactionVersion::Number(0)
|
||||||
|
1 -> RawTransactionVersion::Number(1)
|
||||||
|
>1 -> source.standard_block_transaction_version_unsupported
|
||||||
|
omise/null -> source.standard_block_transaction_version_unqualified
|
||||||
|
```
|
||||||
|
|
||||||
|
Le matériau direct conserve :
|
||||||
|
|
||||||
|
```text
|
||||||
|
signature embarquée dans le wire Base64
|
||||||
|
slot
|
||||||
|
blockTime
|
||||||
|
meta
|
||||||
|
version qualifiée
|
||||||
|
transactionIndex dans le bloc
|
||||||
|
```
|
||||||
|
|
||||||
|
Les fixtures locales construisent et sérialisent des wires Legacy, V0 et V1 avec les types publics de `ksp-raw-transaction-lib`, puis vérifient le Common RAW canonique obtenu. La preuve V1 ne repose donc pas uniquement sur une étiquette `version=1`.
|
||||||
|
|
||||||
|
La parité historique `blockSubscribe` / `getBlock` Legacy reste déjà matérialisée dans `ksp-job-backfill-lib/tests/ws_raw_parity.rs`; `pre.004` rejoue localement les matérialisations Legacy/V0 et ajoute la qualification V1.
|
||||||
|
|
||||||
|
## block:null et formes ambiguës
|
||||||
|
|
||||||
|
Les cas suivants sont des fautes sûres avant progression de frontier :
|
||||||
|
|
||||||
|
```text
|
||||||
|
context.slot différent du slot de notification
|
||||||
|
notification err non null
|
||||||
|
block null
|
||||||
|
transactions omitted/null
|
||||||
|
transaction non Base64
|
||||||
|
version omise/null/>1
|
||||||
|
signature embarquée invalide
|
||||||
|
transaction index non représentable en u32
|
||||||
|
```
|
||||||
|
|
||||||
|
`block: null` n'est jamais assimilé à un bloc vide ou à un slot traité. Le décodage Transport qui distingue `block:null` et `err` est conservé tel quel.
|
||||||
|
|
||||||
|
Pour un bloc non vide, le slot n'est enregistré settled qu'après l'admission réussie de toutes ses transactions. La voie RAW-direct n'incrémente pas `hydration_pending`; le processing frontier ne peut donc pas avancer transitoirement au milieu du traitement d'un même bloc et un stop/faute partiel ne devient jamais une progression.
|
||||||
|
|
||||||
|
## Identité logique et redaction
|
||||||
|
|
||||||
|
Le `source_key` Standard Block est dérivé de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
domain KSP live-source v1
|
||||||
|
family = standard_block
|
||||||
|
network
|
||||||
|
safe provider identity
|
||||||
|
safe endpoint identity
|
||||||
|
commitment
|
||||||
|
private block-filter fingerprint
|
||||||
|
```
|
||||||
|
|
||||||
|
Les filtres `All` et `MentionsAccountOrProgram(pubkey)` sont distingués par empreinte privée. Le pubkey brut n'est pas recopié dans `Debug`, snapshot ou provenance textuelle.
|
||||||
|
|
||||||
|
Le bound global `1..32`, l'invariant réseau unique et le rejet transactionnel des doublons de `pre.002` s'appliquent à cette nouvelle famille.
|
||||||
|
|
||||||
|
## Activation runtime
|
||||||
|
|
||||||
|
Le contrat reste volontairement mono-source avant `pre.007` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
1 source Yellowstone : productif
|
||||||
|
1 source Standard Logs : productif
|
||||||
|
1 source Standard Block : productif
|
||||||
|
2..32 sources : composition validable
|
||||||
|
start 2..32 : runtime_resources.multi_source_activation_pending avant spawn
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune source configurée n'est silencieusement ignorée.
|
||||||
|
|
||||||
|
## Preuves ajoutées
|
||||||
|
|
||||||
|
Unit tests Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
validation protocole/commitment Standard Block
|
||||||
|
source_key stable et filter-sensitive
|
||||||
|
redaction filtre/url/source-key
|
||||||
|
qualification exacte Legacy/V0/V1
|
||||||
|
rejet version omise/null/>1
|
||||||
|
wires Legacy/V0/V1 -> Common RAW direct avec signature/version/index exacts
|
||||||
|
rejet duplicate Standard Block identity
|
||||||
|
```
|
||||||
|
|
||||||
|
Canaris externes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
public_api : construction/composition depuis crate root
|
||||||
|
dependency_boundary : façade Transport seulement, sans fallback HTTP/direct network crate
|
||||||
|
hardening : block:null, erreurs, version, encoding et redaction explicites
|
||||||
|
release_completeness : export et canaris pre.004 obligatoires
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/README.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md
|
||||||
|
docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.004.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
`pre.004` ne prétend pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Helius transactionSubscribe productif
|
||||||
|
HTTP live block polling
|
||||||
|
supervision simultanée de plusieurs sources
|
||||||
|
fanout Store d'observations multi-source
|
||||||
|
fallback HTTP getBlock/getTransaction pour Standard Block
|
||||||
|
RAW-direct pour version >1 ou version ambiguë
|
||||||
|
block:null considéré comme progression
|
||||||
|
nouvelle dépendance externe
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Le toolchain Rust/Cargo/Rustfmt n'est pas disponible dans l'environnement d'assemblage. Les commandes Cargo post-`pre.004` restent donc `NON EXÉCUTÉ LOCAL`. Les audits statiques KSP, le scan des identifiants normatifs et le contrôle exhaustif du diff sont exécutés avant packaging et leurs résultats finaux sont consignés ci-dessous avant livraison.
|
||||||
|
|
||||||
|
## Gate opérateur requis avant pre.005
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.005` ne doit pas commencer si ce gate révèle une anomalie.
|
||||||
|
|
||||||
|
## Résultats statiques finaux de l'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 tables, 835 files)
|
||||||
|
Normative rule definitions: 489
|
||||||
|
Unique normative IDs: 489
|
||||||
|
Duplicates: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Contrôle exhaustif avant packaging :
|
||||||
|
|
||||||
|
```text
|
||||||
|
12 fichiers existants modifiés
|
||||||
|
1 fichier ajouté
|
||||||
|
0 fichier supprimé
|
||||||
|
12/12 headers versionnés incrémentés exactement de +1
|
||||||
|
Cargo.toml hors header : uniquement workspace.package.version 0.3.13-pre.3.fix.3 -> 0.3.13-pre.4
|
||||||
|
aucune dépendance ajoutée, supprimée ou modifiée
|
||||||
|
```
|
||||||
94
deltas/0.3.13/pre.005-fix.001.md
Normal file
94
deltas/0.3.13/pre.005-fix.001.md
Normal file
@@ -0,0 +1,94 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.005-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.005-fix.001 — correction du canari de redaction Helius
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison précédente : 0.3.13-pre.005
|
||||||
|
Cargo base : 0.3.13-pre.5
|
||||||
|
delta base : deltas/0.3.13/pre.005.md
|
||||||
|
archive delta base : ksp-general-0.3.13-pre.005.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 10 septembre 2026 sur `pre.005` est vert pour `cargo fmt`, `cargo fmt --check`, les audits Rust/Markdown, `cargo check --workspace`, Clippy strict, les `84` tests unitaires Worker et les `12` tests `dependency_boundary`. Il s'arrête ensuite sur un seul test de hardening : `v0_3_12_pre_002_production_sources_keep_transport_confined_to_runtime_resources` détecte le mot `credential` dans `runtime_resources.rs`.
|
||||||
|
|
||||||
|
L'occurrence concernée se trouve uniquement dans une Rustdoc décrivant l'encapsulation des paramètres Transport Helius. Aucun secret, aucune credential et aucune valeur sensible n'est stocké en clair dans le Worker par cette chaîne documentaire.
|
||||||
|
|
||||||
|
## Objectif du fix
|
||||||
|
|
||||||
|
Conserver le canari de hardening inchangé et reformuler la Rustdoc production afin qu'elle respecte sa politique textuelle stricte. La phrase emploie désormais `sensitive endpoint material` au lieu de `credentials`.
|
||||||
|
|
||||||
|
Aucune logique Helius, WebSocket, hydration, source identity, admission, Store, frontier ou API publique n'est modifiée.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.005-fix.001
|
||||||
|
workspace.package.version : 0.3.13-pre.5.fix.1
|
||||||
|
archive : ksp-general-0.3.13-pre.005-fix.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
`Cargo.toml` passe de la version de fichier `548` à `549`.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
`runtime_resources.rs` passe de la version de fichier `17` à `18`.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.005-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
Ce fix ne modifie aucune dépendance, aucun code exécutable et aucun test. Il ne change pas le comportement de `transactionSubscribe`, de l'hydration `getTransaction observed`, de la redaction runtime ni de la convergence multi-source.
|
||||||
|
|
||||||
|
## Validation dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Le toolchain Rust/Cargo/Rustfmt n'est pas disponible dans l'environnement d'assemblage. Les commandes Cargo post-fix restent donc `NON EXÉCUTÉ LOCAL`. Les audits statiques KSP et le contrôle exhaustif du delta sont exécutés avant packaging.
|
||||||
|
|
||||||
|
## Gate opérateur requis avant pre.006
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Résultats statiques finaux de l'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 tables, 838 files)
|
||||||
|
```
|
||||||
|
|
||||||
|
Contrôle exhaustif du delta :
|
||||||
|
|
||||||
|
```text
|
||||||
|
2 fichiers existants modifiés
|
||||||
|
1 fichier ajouté
|
||||||
|
0 fichier supprimé
|
||||||
|
aucun test modifié
|
||||||
|
aucune dépendance ajoutée/modifiée/supprimée
|
||||||
|
aucun marqueur secret-like interdit restant dans les sources Rust production du Worker
|
||||||
|
```
|
||||||
268
deltas/0.3.13/pre.005.md
Normal file
268
deltas/0.3.13/pre.005.md
Normal file
@@ -0,0 +1,268 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.005.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.005 — Helius transactionSubscribe + hydration Common RAW
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison précédente : 0.3.13-pre.004-fix.001
|
||||||
|
Cargo base : 0.3.13-pre.4.fix.1
|
||||||
|
delta base : deltas/0.3.13/pre.004-fix.001.md
|
||||||
|
archive delta base : ksp-general-0.3.13-pre.004-fix.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur reçu le 10 septembre 2026 sur cette base est vert pour toutes les commandes exécutées : `cargo fmt`, `cargo fmt --check`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict et toutes les suites de `ksp-worker-raw-transaction-ingest-lib` (`79` unit, `4` cross-layer, `11` dependency-boundary, `18` hardening, `12` public-api, `4` release-completeness, `0` doc-test). Cette exécution ne contenait pas le test Transport ni les commandes `cargo tree`; ils ne sont donc pas déclarés PASS pour ce gate précis.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Ajouter la source Helius WebSocket prévue par la tranche sans dupliquer le chemin Common RAW :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Helius LaserStream transactionSubscribe
|
||||||
|
-> Full + Base64 + maxSupportedTransactionVersion=1 + showRewards=false
|
||||||
|
-> projection privée signature + slot + transactionIndex
|
||||||
|
-> coordinateur d'hydration source-neutral existant
|
||||||
|
-> HTTP getTransaction observed
|
||||||
|
-> Common RAW canonique
|
||||||
|
-> admission centrale Worker
|
||||||
|
-> Store
|
||||||
|
```
|
||||||
|
|
||||||
|
Le nested payload transaction Helius n'est pas utilisé comme RAW-direct. Il reste possédé par Transport et n'est jamais copié dans le signal Worker.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.005
|
||||||
|
workspace.package.version : 0.3.13-pre.5
|
||||||
|
archive : ksp-general-0.3.13-pre.005.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
`Cargo.toml` passe de la version de fichier `547` à `548`.
|
||||||
|
|
||||||
|
## Source Helius Transaction
|
||||||
|
|
||||||
|
Nouvelle surface publique Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestHeliusTransactionSource
|
||||||
|
RawTransactionIngestRuntimeResources::from_helius_transaction_source
|
||||||
|
RawTransactionIngestRuntimeResources::try_push_helius_transaction_source
|
||||||
|
```
|
||||||
|
|
||||||
|
Construction caller-composed :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WsEndpointSettings kind helius_laserstream
|
||||||
|
HeliusTransactionSubscribeFilter
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName d'hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction est sans I/O. Elle refuse un endpoint invalide ou non Helius LaserStream, `Processed`, un réseau/provenance non représentable et l'absence d'une route HTTP `getTransaction` compatible sur le même réseau.
|
||||||
|
|
||||||
|
Au runtime, la source réutilise exclusivement les façades Transport existantes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
HeliusLaserStreamWsSession::connect
|
||||||
|
HeliusLaserStreamWsSession::transaction_subscribe
|
||||||
|
HttpTransportPool::get_transaction_observed via le coordinateur commun
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun client WebSocket direct, SDK Helius, `reqwest`, `tokio-tungstenite`, backend Store ou accès Config n'est ajouté au Worker.
|
||||||
|
|
||||||
|
## Requête Helius qualifiée
|
||||||
|
|
||||||
|
La requête Worker est fixée à :
|
||||||
|
|
||||||
|
```text
|
||||||
|
commitment = Confirmed | Finalized
|
||||||
|
encoding = Base64
|
||||||
|
transactionDetails = Full
|
||||||
|
showRewards = false
|
||||||
|
maxSupportedTransactionVersion = 1
|
||||||
|
```
|
||||||
|
|
||||||
|
Une notification productive doit être `HeliusTransactionNotification::Full`. Une forme `Signature`, `Unknown` ou future malgré la requête `Full` devient `source.helius_transaction_notification_unqualified`; elle n'est jamais convertie en succès silencieux.
|
||||||
|
|
||||||
|
## Projection et hydration
|
||||||
|
|
||||||
|
Le signal privé issu d'une notification `Full` conserve seulement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network
|
||||||
|
signature
|
||||||
|
slot
|
||||||
|
transactionIndex
|
||||||
|
safe route identity
|
||||||
|
private filter fingerprint
|
||||||
|
commitment via hydration context
|
||||||
|
```
|
||||||
|
|
||||||
|
Le nested champ `transaction` de la notification Helius n'est pas lu par `project_helius_transaction_signal` et n'est pas copié dans Worker. La construction Common RAW reste donc unique : `getTransaction observed` en Base64 avec `maxSupportedTransactionVersion = 1`, puis canonicalisation/admission existantes.
|
||||||
|
|
||||||
|
Le coordinateur d'hydration reste borné et source-neutral :
|
||||||
|
|
||||||
|
```text
|
||||||
|
in-flight hydration <= persistence_concurrency
|
||||||
|
pending source signals <= admission_queue_capacity
|
||||||
|
coalescence key = network + signature + commitment
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette voie converge avec Yellowstone et Standard Logs au lieu d'introduire un second moteur HTTP.
|
||||||
|
|
||||||
|
## Identité logique, redaction et secrets
|
||||||
|
|
||||||
|
Le `source_key` Helius est dérivé de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
domain KSP live-source v1
|
||||||
|
family = helius_transaction
|
||||||
|
network
|
||||||
|
safe provider identity
|
||||||
|
safe endpoint identity
|
||||||
|
commitment
|
||||||
|
private normalized Helius filter fingerprint
|
||||||
|
SHA-256 -> [u8; 32]
|
||||||
|
```
|
||||||
|
|
||||||
|
L'empreinte privée couvre `vote`, `failed`, signature exacte, listes `accountInclude`/`accountExclude`/`accountRequired` et `tokenAccounts`. Les listes de pubkeys sont triées avant hash afin que leur ordre ne crée pas artificiellement deux identités logiques. Le rôle/pool HTTP d'hydration reste exclu du `source_key`.
|
||||||
|
|
||||||
|
Les URLs et credentials restent encapsulés par les settings Transport privés utilisés pour ouvrir les connexions ; ils ne sont exposés ni par l'API publique ni par `Debug`. Les valeurs de signature/pubkeys du filtre et les octets de `source_key` ne sont pas rendus dans les diagnostics Worker.
|
||||||
|
|
||||||
|
Les profils Config Helius et `KSP_SECRET_HELIUS_API_KEY` existaient déjà avant cette tranche. Aucun changement Config, `.env`, entitlement, quota, prix ou tier provider n'est codé dans Worker.
|
||||||
|
|
||||||
|
## Activation runtime
|
||||||
|
|
||||||
|
Le contrat reste volontairement mono-source avant `pre.007` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
1 source Yellowstone : productif
|
||||||
|
1 source Standard Logs : productif
|
||||||
|
1 source Standard Block : productif
|
||||||
|
1 source Helius Transaction : productif
|
||||||
|
2..32 sources : composition validable
|
||||||
|
start 2..32 : runtime_resources.multi_source_activation_pending avant spawn
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune source configurée n'est silencieusement ignorée.
|
||||||
|
|
||||||
|
## Preuves ajoutées
|
||||||
|
|
||||||
|
Unit tests Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
validation protocole/commitment/route HTTP Helius
|
||||||
|
requête fixe Full/Base64/maxV1/showRewards=false
|
||||||
|
source_key filter-sensitive, order-normalized et hydration-role-neutral
|
||||||
|
Debug redacted : signature/pubkeys/URL/API key/source-key absents
|
||||||
|
projection Full -> signature/slot/index uniquement
|
||||||
|
Helius reference -> un getTransaction observed -> Common RAW exact
|
||||||
|
provenance helius_ws_http + transaction_get_transaction
|
||||||
|
rejet transactionnel d'une identité Helius dupliquée
|
||||||
|
```
|
||||||
|
|
||||||
|
Canaris externes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
public_api : construction/from/push depuis crate root sans escape hatch des ressources internes
|
||||||
|
dependency_boundary : façades Transport Helius + un seul getTransaction commun
|
||||||
|
hardening : nested payload non copié, secrets/redaction et absence de tier provider
|
||||||
|
release_completeness : export et canaris pre.005 obligatoires
|
||||||
|
```
|
||||||
|
|
||||||
|
## Documentation corrigée
|
||||||
|
|
||||||
|
La documentation Worker est mise à jour pour inclure la source Helius et la convergence d'hydration. Une mention héritée de `README.md` qui classait encore `blockSubscribe` et Helius parmi les sources hors périmètre est supprimée ; `USAGE.md` reste volontairement version-neutral.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/README.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md
|
||||||
|
docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.005.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
`pre.005` ne prétend pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
supervision simultanée de plusieurs sources
|
||||||
|
HTTP live block polling
|
||||||
|
RAW-direct depuis le nested payload Helius
|
||||||
|
SDK Helius dans Worker
|
||||||
|
secret/API key lu directement par Worker
|
||||||
|
nouveau profil Config Helius
|
||||||
|
entitlement/tier/quota/prix provider codé
|
||||||
|
modification de ksp-onchain-transport-lib
|
||||||
|
nouvelle dépendance externe
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Le toolchain Rust/Cargo/Rustfmt n'est pas disponible dans l'environnement d'assemblage. Les commandes Cargo post-`pre.005` restent donc `NON EXÉCUTÉ LOCAL`. Les audits statiques KSP, le scan des identifiants normatifs et le contrôle exhaustif du diff sont exécutés avant packaging.
|
||||||
|
|
||||||
|
## Gate opérateur requis avant pre.006
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.006` ne doit pas commencer si ce gate révèle une anomalie.
|
||||||
|
|
||||||
|
## Résultats statiques finaux de l'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 tables, 837 files)
|
||||||
|
Normative rule definitions: 489
|
||||||
|
Unique normative IDs: 489
|
||||||
|
Duplicates: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
## Contrôle exhaustif avant packaging
|
||||||
|
|
||||||
|
```text
|
||||||
|
12 fichiers existants modifiés
|
||||||
|
1 fichier ajouté
|
||||||
|
0 fichier supprimé
|
||||||
|
12/12 headers de fichiers existants incrémentés exactement de +1
|
||||||
|
Cargo.toml : version workspace uniquement hors header
|
||||||
|
aucune dépendance ajoutée/modifiée/supprimée
|
||||||
|
```
|
||||||
111
deltas/0.3.13/pre.006-fix.001.md
Normal file
111
deltas/0.3.13/pre.006-fix.001.md
Normal file
@@ -0,0 +1,111 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.006-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.006-fix.001 — correction du canari historique après live block discovery
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison précédente : 0.3.13-pre.006
|
||||||
|
Cargo base : 0.3.13-pre.6
|
||||||
|
delta base : deltas/0.3.13/pre.006.md
|
||||||
|
archive delta base : ksp-general-0.3.13-pre.006.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 10 septembre 2026 sur `pre.006` est vert pour `cargo fmt`, `cargo fmt --check`, les audits Rust/Markdown, `cargo check --workspace`, Clippy strict, les `88` tests unitaires Worker et les `13` tests `dependency_boundary`. Il s'arrête ensuite sur un seul canari de hardening : `pre_010_production_surface_has_no_historical_backfill_or_retriever_contract` détecte la chaîne `Backfill` dans la Rustdoc de la nouvelle source HTTP live block polling.
|
||||||
|
|
||||||
|
La même vérification interdisait également globalement `Discovery`/`discovery`. Cette hypothèse n'est plus valide depuis `pre.006` : la découverte run-local via `getBlocksWithLimit` est une primitive légitime du live polling et ne constitue ni un retriever historique ni un Backfill.
|
||||||
|
|
||||||
|
## Objectif du fix
|
||||||
|
|
||||||
|
Conserver la séparation Worker/Backfill sans interdire la terminologie de live block discovery introduite par le poller HTTP :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Rustdoc HTTP polling : suppression de la mention Backfill
|
||||||
|
canari historique : conserve les interdictions retriever/backfill/checkpoint/historical
|
||||||
|
mais ne classe plus Discovery/discovery comme marqueur historique
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune logique de polling, cadence, Transport, admission, Store, processing frontier, provenance, source identity ou API publique n'est modifiée.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.006-fix.001
|
||||||
|
workspace.package.version : 0.3.13-pre.6.fix.1
|
||||||
|
archive : ksp-general-0.3.13-pre.006-fix.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
`Cargo.toml` passe de la version de fichier `550` à `551`.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
Versions de fichiers :
|
||||||
|
|
||||||
|
```text
|
||||||
|
runtime_resources.rs : 19 -> 20
|
||||||
|
hardening.rs : 15 -> 16
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.006-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
Ce fix ne modifie aucune dépendance et n'ajoute aucune source. Il ne change ni la borne de run, ni `getSlot`, ni `getBlocksWithLimit`, ni `getBlock observed`, ni les règles Legacy/V0/V1, ni la sémantique `block:null`.
|
||||||
|
|
||||||
|
## Validation dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Le toolchain Rust/Cargo/Rustfmt n'est pas disponible dans l'environnement d'assemblage. Les commandes Cargo post-fix restent donc `NON EXÉCUTÉ LOCAL`. Les audits statiques KSP et le contrôle exhaustif du delta sont exécutés avant packaging.
|
||||||
|
|
||||||
|
## Gate opérateur requis avant pre.007
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Résultats statiques finaux de l'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 tables, 840 files)
|
||||||
|
Normative rule definitions: 489
|
||||||
|
Unique normative IDs: 489
|
||||||
|
Duplicates: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Contrôle ciblé de la surface production Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-retriever : 0
|
||||||
|
raw_transaction_retriever : 0
|
||||||
|
RawTransactionRetriever : 0
|
||||||
|
Backfill/backfill : 0
|
||||||
|
Checkpoint/checkpoint : 0
|
||||||
|
historical : 0
|
||||||
|
```
|
||||||
|
|
||||||
|
`Discovery`/`discovery` n'est plus un marqueur interdit global : `pre.006` l'emploie pour la découverte live run-local de slots/blocs par `getBlocksWithLimit`.
|
||||||
322
deltas/0.3.13/pre.006.md
Normal file
322
deltas/0.3.13/pre.006.md
Normal file
@@ -0,0 +1,322 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.006.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.006 — HTTP live block polling run-local
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison précédente : 0.3.13-pre.005-fix.001
|
||||||
|
Cargo base : 0.3.13-pre.5.fix.1
|
||||||
|
delta base : deltas/0.3.13/pre.005-fix.001.md
|
||||||
|
archive delta base : ksp-general-0.3.13-pre.005-fix.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur reçu le 10 septembre 2026 sur cette base est entièrement vert pour les commandes exécutées : `cargo fmt`, `cargo fmt --check`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict et toutes les suites de `ksp-worker-raw-transaction-ingest-lib` (`84` unit, `4` cross-layer, `12` dependency-boundary, `19` hardening, `13` public-api, `4` release-completeness, `0` doc-test). Cette exécution ne contenait ni le test Transport ni les commandes `cargo tree`; ils ne sont donc pas déclarés PASS pour ce gate précis.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Ajouter la source HTTP live prévue par la tranche sans la transformer en Backfill :
|
||||||
|
|
||||||
|
```text
|
||||||
|
getSlot(commitment)
|
||||||
|
-> borne inférieure du run
|
||||||
|
getBlocksWithLimit(next_scan_slot, bounded_limit, commitment)
|
||||||
|
-> découverte des blocs disponibles
|
||||||
|
getBlock observed(slot, Full/Base64/maxV1/rewards=false)
|
||||||
|
-> Common RAW direct Legacy/V0/V1
|
||||||
|
-> admission centrale Worker
|
||||||
|
-> Store
|
||||||
|
```
|
||||||
|
|
||||||
|
Le polling ne remonte jamais avant la borne observée au démarrage et ne crée aucun checkpoint durable inter-run.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.006
|
||||||
|
workspace.package.version : 0.3.13-pre.6
|
||||||
|
archive : ksp-general-0.3.13-pre.006.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
`Cargo.toml` passe également de la version de fichier `549` à `550`.
|
||||||
|
|
||||||
|
## Source HTTP Block Polling
|
||||||
|
|
||||||
|
Nouvelle surface publique Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestHttpBlockPollingSource
|
||||||
|
RawTransactionIngestRuntimeResources::from_http_block_polling_source
|
||||||
|
RawTransactionIngestRuntimeResources::try_push_http_block_polling_source
|
||||||
|
DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL
|
||||||
|
MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL
|
||||||
|
MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL
|
||||||
|
DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE
|
||||||
|
MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE
|
||||||
|
MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction caller-owned reçoit un `HttpTransportPool`, un `HttpRoleName` et un commitment `Confirmed` ou `Finalized`. Elle est sans I/O et vérifie statiquement que le rôle possède des routes compatibles `getSlot`, `getBlocksWithLimit` et `getBlock` sur un même réseau.
|
||||||
|
|
||||||
|
Bornes publiques :
|
||||||
|
|
||||||
|
```text
|
||||||
|
poll interval : défaut 1 s ; min 100 ms ; max 30 s
|
||||||
|
max discovered blocks/cycle : défaut 128 ; min 1 ; max 1024
|
||||||
|
```
|
||||||
|
|
||||||
|
Les limitations physiques, la sélection d'endpoint, le rate-limit, les retries et le backoff HTTP restent possédés par `ksp-onchain-transport-lib`.
|
||||||
|
|
||||||
|
## Borne de run et progression
|
||||||
|
|
||||||
|
Au démarrage :
|
||||||
|
|
||||||
|
```text
|
||||||
|
start_slot = getSlot(commitment)
|
||||||
|
next_scan_slot = start_slot
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker ne demande aucun slot inférieur à `start_slot`. Pour chaque cycle, il relit le tip courant, puis appelle `getBlocksWithLimit` depuis `next_scan_slot`. La réponse doit être strictement croissante et ne peut contenir aucun slot inférieur à la frontier de scan.
|
||||||
|
|
||||||
|
Chaque slot listé est matérialisé par `getBlock observed`. Si le résultat vaut `null` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
next_scan_slot = slot
|
||||||
|
aucun settled
|
||||||
|
aucun Common RAW
|
||||||
|
reprise au cycle suivant
|
||||||
|
```
|
||||||
|
|
||||||
|
Le cas `null` n'est donc ni assimilé à un bloc vide ni converti en progression silencieuse.
|
||||||
|
|
||||||
|
Les slots non listés par `getBlocksWithLimit` sont traités comme non produits/skipped dans le run courant. Lorsque la liste reçue ne remplit pas le batch demandé et qu'aucun slot n'est bloqué par `null`, la frontier peut rejoindre le tip observé du cycle sans lancer de scan historique implicite.
|
||||||
|
|
||||||
|
## RAW-direct et version
|
||||||
|
|
||||||
|
Le poller demande exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
encoding = Base64
|
||||||
|
transactionDetails = Full
|
||||||
|
maxSupportedTransactionVersion = 1
|
||||||
|
showRewards = false
|
||||||
|
commitment = Confirmed | Finalized
|
||||||
|
```
|
||||||
|
|
||||||
|
Chaque transaction qualifiée produit directement `RawTransactionMaterial` avec :
|
||||||
|
|
||||||
|
```text
|
||||||
|
signature embarquée dans le wire Base64
|
||||||
|
slot
|
||||||
|
blockTime
|
||||||
|
meta
|
||||||
|
version Legacy | 0 | 1
|
||||||
|
transactionIndex dérivé de la position dans le bloc
|
||||||
|
```
|
||||||
|
|
||||||
|
Les cas suivants sont fail-closed : transaction non Base64, transactions omises/nulles, version omise/nulle, version `>1`, signature invalide ou index non représentable en `u32`.
|
||||||
|
|
||||||
|
La voie HTTP Block Polling est RAW-direct et n'utilise pas le coordinateur d'hydration. Le slot n'est marqué `settled` qu'après l'envoi réussi de toutes les transactions du bloc vers l'admission centrale ; `hydration_pending` n'est jamais artificiellement incrémenté.
|
||||||
|
|
||||||
|
## Provenance observée
|
||||||
|
|
||||||
|
La provenance de chaque transaction reprend l'endpoint et le provider réellement gagnants de `getBlock observed` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
origin = Live
|
||||||
|
protocol = solana_http
|
||||||
|
method = block_polling_get_block
|
||||||
|
endpoint = endpoint observé
|
||||||
|
provider = provider observé
|
||||||
|
commitment = source commitment
|
||||||
|
capture session = WorkerId
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun URL, header HTTP ou corps brut n'est exposé par `HttpObservedValue` ou les diagnostics Worker.
|
||||||
|
|
||||||
|
## Identité logique
|
||||||
|
|
||||||
|
Le `source_key` HTTP polling est dérivé de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
domain KSP live-source v1
|
||||||
|
family = http_block_polling
|
||||||
|
network
|
||||||
|
polling role
|
||||||
|
commitment
|
||||||
|
private route-profile fingerprint
|
||||||
|
SHA-256 -> [u8; 32]
|
||||||
|
```
|
||||||
|
|
||||||
|
L'empreinte de profil est construite à partir des identités sûres cluster/provider/endpoint et des capacités pertinentes de chaque route active, après tri déterministe. La cadence et le batch de découverte n'entrent volontairement pas dans l'identité logique : modifier un réglage de timing ne crée pas une deuxième source conceptuelle.
|
||||||
|
|
||||||
|
Le bound global `1..32`, l'invariant réseau unique et le rejet transactionnel des doublons de `pre.002` s'appliquent à cette famille.
|
||||||
|
|
||||||
|
## Activation runtime
|
||||||
|
|
||||||
|
Le contrat reste volontairement mono-source avant `pre.007` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
1 source Yellowstone : productif
|
||||||
|
1 source Standard Logs : productif
|
||||||
|
1 source Standard Block : productif
|
||||||
|
1 source Helius Transaction : productif
|
||||||
|
1 source HTTP Block Polling : productif
|
||||||
|
2..32 sources : composition validable
|
||||||
|
start 2..32 : runtime_resources.multi_source_activation_pending avant spawn
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune source configurée n'est ignorée silencieusement.
|
||||||
|
|
||||||
|
## Preuves ajoutées
|
||||||
|
|
||||||
|
Unit tests Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
bornes cadence/batch et commitment
|
||||||
|
présence exacte des capacités HTTP requises
|
||||||
|
source_key stable malgré changement de timing et différent si route logique différente
|
||||||
|
redaction URL sensible dans Debug
|
||||||
|
validation stricte de la découverte croissante
|
||||||
|
qualification Legacy/V0/V1 et rejet version ambiguë/future
|
||||||
|
rejet transactionnel d'une identité HTTP polling dupliquée
|
||||||
|
```
|
||||||
|
|
||||||
|
Canaris externes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
public_api : constructeurs/from/push et bornes publiques disponibles depuis crate root
|
||||||
|
dependency_boundary : Transport facade uniquement, aucun Config/Backfill/backend/direct HTTP client
|
||||||
|
hardening : run-local, timer borné, getBlock observed unique, aucun task par tick, aucun hydration_pending direct
|
||||||
|
release_completeness : exports et canaris pre.006 obligatoires
|
||||||
|
```
|
||||||
|
|
||||||
|
Les anciens canaris qui interdisaient globalement `get_block_observed` ou `tokio::time::sleep` sont resserrés sur le coordinateur d'hydration : ces primitives restent interdites dans l'hydration mais sont désormais légitimes, explicites et bornées dans la seule source HTTP polling.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
`README.md`, le README du crate, `USAGE.md`, le plan et le journal de validation sont alignés sur la cinquième famille productive. `USAGE.md` reste version-neutral : il décrit la construction et le comportement sans journaliser le numéro de prerelease.
|
||||||
|
|
||||||
|
Une incohérence du README de crate, qui classait encore le live HTTP polling parmi les éléments hors périmètre, est corrigée dans cette tranche.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
README.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/README.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md
|
||||||
|
docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.006.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
`pre.006` ne prétend pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
supervision simultanée de plusieurs sources
|
||||||
|
checkpoint durable du polling
|
||||||
|
Backfill implicite
|
||||||
|
scan de slots antérieurs au start_slot du run
|
||||||
|
retry/backoff HTTP implémenté dans Worker
|
||||||
|
client reqwest direct dans Worker
|
||||||
|
Config lu dans Worker
|
||||||
|
getBlock=null assimilé à un bloc vide
|
||||||
|
transaction version >1 ou ambiguë acceptée
|
||||||
|
modification de ksp-onchain-transport-lib
|
||||||
|
nouvelle dépendance externe
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Le toolchain Rust/Cargo/Rustfmt n'est pas disponible dans l'environnement d'assemblage. Les commandes Cargo post-`pre.006` restent donc `NON EXÉCUTÉ LOCAL`. Les audits statiques KSP, le scan normatif, le contrôle des versions de fichiers et le diff exhaustif sont exécutés avant packaging.
|
||||||
|
|
||||||
|
## Gate opérateur requis avant pre.007
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.007` ne doit pas commencer si ce gate révèle une anomalie.
|
||||||
|
|
||||||
|
## Résultats statiques finaux de l'assemblage
|
||||||
|
|
||||||
|
Les valeurs ci-dessous sont mises à jour après la dernière passe d'audits sur l'état exact livré.
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 tables, 839 files)
|
||||||
|
Normative rule definitions: 489
|
||||||
|
Unique normative IDs: 489
|
||||||
|
Duplicates: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
## Contrôle exhaustif avant packaging
|
||||||
|
|
||||||
|
Comparaison avec la base `0.3.13-pre.005-fix.001` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
added : 1
|
||||||
|
deltas/0.3.13/pre.006.md
|
||||||
|
modified : 13
|
||||||
|
Cargo.toml
|
||||||
|
README.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/README.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md
|
||||||
|
docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md
|
||||||
|
deleted : 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Les `13/13` fichiers existants modifiés portent une version de fichier exactement `+1`. Le diff de `Cargo.toml` est limité au header `549 -> 550` et à `workspace.package.version = "0.3.13-pre.6"`; aucun membre workspace ni aucune dépendance n'est modifié.
|
||||||
|
|
||||||
|
La revue statique ciblée confirme dans le code Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
get_block_observed( : 1 occurrence, source HTTP polling uniquement
|
||||||
|
get_blocks_with_limit( : 1 occurrence, source HTTP polling uniquement
|
||||||
|
tokio::time::sleep : 1 occurrence, cadence du poller uniquement
|
||||||
|
aucun reqwest/tokio-tungstenite/tonic/yellowstone-grpc-proto direct
|
||||||
|
aucun ksp-config-lib/ksp-job-backfill-lib/ksp-store-postgres-lib direct
|
||||||
|
aucun unbounded_channel
|
||||||
|
```
|
||||||
218
deltas/0.3.13/pre.007.md
Normal file
218
deltas/0.3.13/pre.007.md
Normal file
@@ -0,0 +1,218 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.007.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.007 — supervision et inventaire multi-source
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison précédente : 0.3.13-pre.006-fix.001
|
||||||
|
Cargo base : 0.3.13-pre.6.fix.1
|
||||||
|
delta base : deltas/0.3.13/pre.006-fix.001.md
|
||||||
|
archive delta base : ksp-general-0.3.13-pre.006-fix.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué le 10 septembre 2026 est vert sur toutes les commandes exécutées : `cargo fmt`, `cargo fmt --check`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict et toutes les suites de `ksp-worker-raw-transaction-ingest-lib` (`88` unit, `4` cross-layer, `13` dependency-boundary, `20` hardening, `14` public-api, `4` release-completeness, `0` doc-test). Le log reçu ne contient pas les sorties séparées `cargo test -p ksp-onchain-transport-lib` ni les trois commandes `cargo tree`; elles ne sont donc pas déclarées PASS dans ce delta.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Activer réellement la collection `1..32` déjà composable depuis `pre.002` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestRuntimeResources
|
||||||
|
-> validation complète réseau / doublons / borne
|
||||||
|
-> spawn de toutes les sources configurées
|
||||||
|
-> inventaire privé source_key -> latest state/frontier
|
||||||
|
-> projection frontier source-neutral agrégée
|
||||||
|
-> terminalité conservatrice si une source disparaît ou échoue
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun provider n'est traité comme primaire, fallback ou standby.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.007
|
||||||
|
workspace.package.version : 0.3.13-pre.7
|
||||||
|
archive : ksp-general-0.3.13-pre.007.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
`Cargo.toml` passe de la version de fichier `551` à `552`.
|
||||||
|
|
||||||
|
## Supervisor multi-source
|
||||||
|
|
||||||
|
`RawTransactionIngestWorker::start_with_runtime_resources` ne vérifie plus `validate_single_source_activation`. Après validation du réseau, le runtime délègue à `RawTransactionIngestRuntimeResources::run_live_sources`.
|
||||||
|
|
||||||
|
Le supervisor privé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crée un stop channel source-owned
|
||||||
|
crée un JoinSet borné par la collection 1..32
|
||||||
|
spawn chaque source configurée exactement une fois
|
||||||
|
clone uniquement les handles nécessaires
|
||||||
|
rejoint toutes les tasks avant de rendre son résultat
|
||||||
|
```
|
||||||
|
|
||||||
|
La vieille erreur `runtime_resources.multi_source_activation_pending` disparaît du code production.
|
||||||
|
|
||||||
|
## Terminalité conservatrice
|
||||||
|
|
||||||
|
La politique reste fail-closed :
|
||||||
|
|
||||||
|
```text
|
||||||
|
stop Worker
|
||||||
|
-> fanout stop vers toutes les sources
|
||||||
|
-> join complet
|
||||||
|
-> Ok seulement si les sources ferment sans erreur
|
||||||
|
|
||||||
|
source Err
|
||||||
|
-> stop immédiat des autres sources
|
||||||
|
-> join complet
|
||||||
|
-> faute source terminale
|
||||||
|
|
||||||
|
source Ok avant stop Worker
|
||||||
|
-> fermeture inattendue d'une source configurée
|
||||||
|
-> stop des autres sources
|
||||||
|
-> faute terminale
|
||||||
|
|
||||||
|
JoinError
|
||||||
|
-> faute terminale sûre
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker ne suppose pas qu'une autre source couvre les mêmes filtres, le même provider universe ou la même complétude.
|
||||||
|
|
||||||
|
## Inventaire privé et processing frontier
|
||||||
|
|
||||||
|
`RawTransactionIngestSourceInventory` et `RawTransactionIngestSourceInventoryPublisher` restent privés au module runtime resources. L'inventaire est initialisé depuis les `source_key` déjà validées et possède exactement une entrée latest-value par source du run.
|
||||||
|
|
||||||
|
Chaque source conserve son propre publisher local. Un wrapper privé reporte ses projections dans l'inventaire puis publie une projection globale conservative :
|
||||||
|
|
||||||
|
```text
|
||||||
|
hydration_pending : somme saturante temporaire des sources
|
||||||
|
oldest_pending_slot : minimum global
|
||||||
|
processing_frontier_slot : None si une source n'a pas encore de frontier
|
||||||
|
sinon minimum de toutes les frontiers
|
||||||
|
source_state : Failed > Reconnecting > Closing > Active > Closed/None
|
||||||
|
reconnect/replay/gap : agrégation saturante, jamais wrap arithmétique
|
||||||
|
```
|
||||||
|
|
||||||
|
Le choix du minimum n'est pas une preuve de coverage blockchain. La frontier reste run-local et source-neutral.
|
||||||
|
|
||||||
|
## Propriété d'ownership
|
||||||
|
|
||||||
|
Les sources physiques continuent d'appartenir à leurs façades Transport existantes. Le Worker ne crée ni client reqwest, ni socket alternatif, ni SDK provider. Le supervisor ajoute uniquement l'ownership des tasks source et de leur lifecycle commun.
|
||||||
|
|
||||||
|
Une source qui termine avant le stop est explicitement marquée `Closed` ou `Failed` dans l'inventaire avant le résultat du supervisor. Les identités `source_key` restent privées et ne sont jamais projetées dans `RawTransactionIngestSnapshot` ou le crate root.
|
||||||
|
|
||||||
|
## Preuves ajoutées
|
||||||
|
|
||||||
|
Unit tests déterministes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
inventaire 2 sources : pending/frontier/state/counters agrégés conservativement
|
||||||
|
stop supervisor : trois tasks source actives reçoivent le stop et sont toutes jointes
|
||||||
|
source failure : la source sœur reçoit le stop et est jointe avant retour de faute
|
||||||
|
```
|
||||||
|
|
||||||
|
Canaris externes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
dependency_boundary : run_live_sources, JoinSet, inventaire privé, vieille barrière absente
|
||||||
|
hardening : stop/fault/join fail-closed, aucune policy primary/standby, aucun canal unbounded
|
||||||
|
public_api : inventaire/live-source/source_key non exportés
|
||||||
|
```
|
||||||
|
|
||||||
|
Les canaris historiques `0.3.12` qui cherchaient `run_single_live_source` sont mis à jour vers le supervisor multi-source au lieu d'être supprimés.
|
||||||
|
|
||||||
|
## Frontière de tranche
|
||||||
|
|
||||||
|
`pre.007` active la simultanéité, mais ne ferme pas encore la convergence transactionnelle cross-source :
|
||||||
|
|
||||||
|
```text
|
||||||
|
pas de coalescence globale de la même signature entre plusieurs sources
|
||||||
|
pas d'unique hydration cross-source garantie
|
||||||
|
pas de fusion de plusieurs provenance/observation seeds vers une acquisition canonique unique
|
||||||
|
pas encore de record_raw_transaction_observation pour les observations supplémentaires convergées
|
||||||
|
pas encore de fairness durcie duplicate-storm
|
||||||
|
pas encore de source_total/source_active/source_failed publics
|
||||||
|
```
|
||||||
|
|
||||||
|
Les coordinators d'hydration des sources reference-bearing restent donc source-local pendant cette tranche. La fermeture de la coalescence globale et des observations multiples appartient à `pre.008`, puis les bornes/fairness renforcées à `pre.009`.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
Le README du crate décrit désormais le supervisor 1..32, la terminalité conservative et la frontier agrégée. `USAGE.md` reste version-neutral et montre comment composer plusieurs sources via les `try_push_*` existants sans mentionner le numéro de prerelease.
|
||||||
|
|
||||||
|
Le plan et la validation consignent le gate opérateur réellement reçu et distinguent explicitement ce qui reste reporté à `pre.008+`.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/README.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md
|
||||||
|
docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.007.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Le toolchain Rust/Cargo/Rustfmt n'est pas disponible dans l'environnement d'assemblage. Les commandes Cargo post-`pre.007` restent donc `NON EXÉCUTÉ LOCAL`. Les audits statiques KSP, le scan normatif, les versions de fichiers et le diff exhaustif sont exécutés sur l'état exact avant packaging.
|
||||||
|
|
||||||
|
## Résultats statiques finaux de l'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 tables, 841 files)
|
||||||
|
Normative rule definitions: 489
|
||||||
|
Unique normative IDs: 489
|
||||||
|
Duplicates: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Contrôle ciblé du supervisor :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ancienne barrière validate_single_source_activation : absente
|
||||||
|
runtime_resources.multi_source_activation_pending : absent du code production
|
||||||
|
inventaire source public : absent
|
||||||
|
source_key public : absent
|
||||||
|
unbounded_channel : absent
|
||||||
|
nouvelle dépendance : aucune
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate opérateur requis avant pre.008
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
85
deltas/0.3.13/pre.008-fix.001.md
Normal file
85
deltas/0.3.13/pre.008-fix.001.md
Normal file
@@ -0,0 +1,85 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.008-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.008-fix.001 — correction compilation convergence persistence
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison précédente : 0.3.13-pre.008
|
||||||
|
Cargo base : 0.3.13-pre.8
|
||||||
|
delta base : deltas/0.3.13/pre.008.md
|
||||||
|
archive delta base : ksp-general-0.3.13-pre.008.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 10 septembre 2026 sur `pre.008` est vert pour `cargo fmt`, `cargo fmt --check` et les audits Rust/Markdown. `cargo check --workspace` met ensuite en évidence deux défauts de compilation dans `ksp-worker-raw-transaction-ingest-lib` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
unused import : pub(crate) use self::persistence::persist_raw_transaction_ingest_acquisition
|
||||||
|
E0004 : match non exhaustif sur RawObservationWriteOutcome #[non_exhaustive]
|
||||||
|
```
|
||||||
|
|
||||||
|
Clippy strict reproduit les deux défauts et les tests ne peuvent pas être compilés avant leur correction.
|
||||||
|
|
||||||
|
## Objectif du fix
|
||||||
|
|
||||||
|
Corriger strictement ces deux défauts sans modifier le contrat de convergence `pre.008` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
re-export crate-root : conservé conformément aux règles KSP
|
||||||
|
appels internes : via crate::persist_raw_transaction_ingest_acquisition(...)
|
||||||
|
non-exhaustive enum : wildcard fail-closed vers une erreur runtime stable
|
||||||
|
```
|
||||||
|
|
||||||
|
Le wildcard ne transforme aucune variante future en succès implicite. Toute variante `RawObservationWriteOutcome` inconnue est rejetée comme `persistence.additional_observation_outcome_invalid`.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.008-fix.001
|
||||||
|
workspace.package.version : 0.3.13-pre.8.fix.1
|
||||||
|
archive : ksp-general-0.3.13-pre.008-fix.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
`Cargo.toml` passe de la version de fichier `553` à `554`.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
`persistence.rs` passe de la version de fichier `2` à `3`.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.008-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
Ce fix ne modifie aucune dépendance, aucune API publique, aucun cache de convergence, aucune clé de coalescence, aucune sémantique d'hydration, aucune observation/provenance et aucune règle Store de première acquisition ou d'observation supplémentaire.
|
||||||
|
|
||||||
|
## Validation dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Le toolchain Rust/Cargo/Rustfmt n'est pas disponible dans l'environnement d'assemblage. Les commandes Cargo post-fix restent donc `NON EXÉCUTÉ LOCAL`. Les audits statiques KSP et le contrôle exhaustif du delta sont exécutés avant packaging.
|
||||||
|
|
||||||
|
## Gate opérateur requis avant pre.009
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user