Compare commits
21 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 85937b8cbb | |||
| 107986bf85 | |||
| a8cabbebd5 | |||
| 636a0c5f43 | |||
| bf46782db3 | |||
| 02919e959e | |||
| d231eba8be | |||
| 637382f364 | |||
| bf6ef5b426 | |||
| 822e8fcd86 | |||
| 63e1a866a3 | |||
| 22c88c449d | |||
| ed6c0ac10c | |||
| 355c5d1e9a | |||
| 61d8b1dee2 | |||
| 0db32a865c | |||
| a4061ee1e1 | |||
| ec64be340e | |||
| f1c383c392 | |||
| 984c327162 | |||
| e2689d1b04 |
14
CHANGELOG.md
14
CHANGELOG.md
@@ -1,8 +1,20 @@
|
|||||||
<!-- file: CHANGELOG.md -->
|
<!-- file: CHANGELOG.md -->
|
||||||
<!-- version: 30 -->
|
<!-- version: 31 -->
|
||||||
|
|
||||||
# Changelog KSP
|
# Changelog KSP
|
||||||
|
|
||||||
|
## 0.3.11 — Fondation runtime source-neutral du Worker RawTransaction ingest — 2026-09-08
|
||||||
|
|
||||||
|
`0.3.11` introduit `ksp-worker-raw-transaction-ingest-lib` comme premier Worker concret KSP pour l’acquisition continue de `RawTransaction`, tout en fermant volontairement la release **sans source réseau productive**. La crate consomme `ksp-worker-api`, `ksp-raw-transaction-lib` et la seule façade `ksp-store-lib`; elle ne dépend ni de Config, ni d’un Job, ni d’un backend Store physique, ni encore de `ksp-onchain-transport-lib`. Son identité stable est `raw_transaction_ingest`, ses settings bornent la capacité d’admission, la concurrence de persistence et la deadline de drain, et son `start` utilise exclusivement le runtime Tokio actif fourni par le caller.
|
||||||
|
|
||||||
|
Le runtime possède un supervisor privé, des tâches source/persistence détenues, une admission centrale `mpsc` bornée avec backpressure, la canonicalisation/assembly Common RAW, une observation key déterministe et la persistence atomique en mode `Normal`. Les outcomes distinguent insert, idempotence et tombstone purgée ; un contenu divergent devient `worker_raw_transaction_ingest.content_conflict`, les erreurs Store restent classifiées sans texte distant, et aucun provider gagnant n’est choisi implicitement. Les snapshots concrets latest-value se projettent directement sur `WorkerSnapshotSource`, utilisent séquences/compteurs checked et conservent le terminal pour les lecteurs tardifs.
|
||||||
|
|
||||||
|
Le hardening final ferme les races stop/fault, les source failures, la saturation observable, le Store lent/fautif et le shutdown borné. À l’expiration de `shutdown_drain_timeout`, le Worker ferme les admissions, abort les tâches restantes, les rejoint toutes puis publie `drain_timeout`; aucune tâche privée ne survit au terminal. Les tests externes de hardening/release verrouillent les sept codes d’erreur publics, les huit modules de production, les vingt-quatre exports crate-root, la redaction, le dependency firewall et l’absence de surface historique/Backfill/Transport/Config/backend direct.
|
||||||
|
|
||||||
|
Le gate technique final `pre.011` passe rustfmt check, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 56 tests Worker, `cargo test --workspace --all-targets --all-features`, les arbres Cargo Worker normal/features et `cargo tree --duplicates`. La réconciliation `pre.012` ajoute ensuite les README/USAGE durables du Worker et réaligne l’architecture sur l’état matériel : la fondation `0.3.11` n’a aucun adapter HTTP/WS/Yellowstone productif, aucune API publique d’enqueue/source registration et aucun replay/gap-repair live.
|
||||||
|
|
||||||
|
La trajectoire live reste découpée : `0.3.12` ouvre Yellowstone transactions/blocks/status + hydration HTTP + continuité de run, `0.3.13` ajoute WS standard/Helius/HTTP live et la convergence multi-source, puis `0.3.14` ferme le gap repair/hardening multi-source. `prompts/031-V0_3_12_START_PROMPT.md` ouvre `0.3.12` uniquement depuis le tag stable `v0.3.11`; son `pre.001` doit réauditer les surfaces Yellowstone/Solana/provider actuelles, l’injection Transport, l’hydration, le frontier, `from_slot`/ReplayInfo, les smokes accessibles et le sizing avant toute implémentation lourde.
|
||||||
|
|
||||||
## 0.3.10 — Lower-layer RAW Transaction commune + qualification cross-source — 2026-09-08
|
## 0.3.10 — Lower-layer RAW Transaction commune + qualification cross-source — 2026-09-08
|
||||||
|
|
||||||
`0.3.10` introduit `ksp-raw-transaction-lib` comme lower-layer source-neutral commune aux producteurs `RawTransaction`. La canonicalisation RAW v1 auparavant locale à `ksp-job-backfill-lib` est extraite sans modifier les golden bytes/hash existants ; le Backfill consomme désormais cette common crate sans créer de dépendance vers un Worker. La bibliothèque construit les modèles `RawTransaction`/`RawTransactionObservation` de `ksp-store-api`, conserve l'identité canonique `(network, signature)`, borne et redacted les entrées hostiles, et reste volontairement sans Transport, Config, runtime async, Job, Worker ou backend Store physique.
|
`0.3.10` introduit `ksp-raw-transaction-lib` comme lower-layer source-neutral commune aux producteurs `RawTransaction`. La canonicalisation RAW v1 auparavant locale à `ksp-job-backfill-lib` est extraite sans modifier les golden bytes/hash existants ; le Backfill consomme désormais cette common crate sans créer de dépendance vers un Worker. La bibliothèque construit les modèles `RawTransaction`/`RawTransactionObservation` de `ksp-store-api`, conserve l'identité canonique `(network, signature)`, borne et redacted les entrées hostiles, et reste volontairement sans Transport, Config, runtime async, Job, Worker ou backend Store physique.
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
# file: Cargo.toml
|
# file: Cargo.toml
|
||||||
# version: 505
|
# version: 518
|
||||||
|
|
||||||
[workspace]
|
[workspace]
|
||||||
resolver = "3"
|
resolver = "3"
|
||||||
members = ["crates/ksp-app-backfill-desk", "crates/ksp-app-config-desk", "crates/ksp-app-solprices-desk", "crates/ksp-app-store-desk", "crates/ksp-app-wallet-desk", "crates/ksp-config-lib", "crates/ksp-core-lib", "crates/ksp-interface-lib", "crates/ksp-job-api", "crates/ksp-job-backfill-lib", "crates/ksp-logging-lib", "crates/ksp-offchain-transport-lib", "crates/ksp-onchain-transport-lib", "crates/ksp-program-api", "crates/ksp-raw-transaction-lib", "crates/ksp-store-api", "crates/ksp-store-lib", "crates/ksp-store-postgres-lib", "crates/ksp-wallet-lib", "crates/ksp-worker-api"]
|
members = ["crates/ksp-app-backfill-desk", "crates/ksp-app-config-desk", "crates/ksp-app-solprices-desk", "crates/ksp-app-store-desk", "crates/ksp-app-wallet-desk", "crates/ksp-config-lib", "crates/ksp-core-lib", "crates/ksp-interface-lib", "crates/ksp-job-api", "crates/ksp-job-backfill-lib", "crates/ksp-logging-lib", "crates/ksp-offchain-transport-lib", "crates/ksp-onchain-transport-lib", "crates/ksp-program-api", "crates/ksp-raw-transaction-lib", "crates/ksp-store-api", "crates/ksp-store-lib", "crates/ksp-store-postgres-lib", "crates/ksp-wallet-lib", "crates/ksp-worker-api", "crates/ksp-worker-raw-transaction-ingest-lib"]
|
||||||
|
|
||||||
[workspace.package]
|
[workspace.package]
|
||||||
version = "0.3.10"
|
version = "0.3.11"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
license = "MIT"
|
license = "MIT"
|
||||||
repository = "https://git.sasedev.com/Sasedev/khadhroony-solana-project"
|
repository = "https://git.sasedev.com/Sasedev/khadhroony-solana-project"
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: README.md -->
|
<!-- file: README.md -->
|
||||||
<!-- version: 11 -->
|
<!-- version: 12 -->
|
||||||
|
|
||||||
# Khadhroony Solana Project
|
# Khadhroony Solana Project
|
||||||
|
|
||||||
@@ -57,7 +57,7 @@ La couche RAW dispose d'une façade Store backend-neutral et d'un premier job hi
|
|||||||
|
|
||||||
`ksp-worker-api` fournit désormais la fondation générique des services continus : identité Worker, lifecycle borné, health/activity, intention de stop coopératif et observation latest-value. Cette API reste Core-only, runtime-neutral et sans connaissance Solana, Transport, Store ou Job. Elle ne démarre ni n'arrête elle-même un runtime concret.
|
`ksp-worker-api` fournit désormais la fondation générique des services continus : identité Worker, lifecycle borné, health/activity, intention de stop coopératif et observation latest-value. Cette API reste Core-only, runtime-neutral et sans connaissance Solana, Transport, Store ou Job. Elle ne démarre ni n'arrête elle-même un runtime concret.
|
||||||
|
|
||||||
Le futur `ksp-worker-raw-transaction-ingest-lib` sera un consumer concret distinct du Job Backfill : il fonctionnera en continu entre Start et Stop, sans scope historique métier, afin d'acquérir et persister des `RawTransaction` selon les sources/capabilities configurées. `ksp-job-backfill-lib` conserve de son côté son rôle historique paramétré et borné. Les deux producteurs restent indépendants et convergent uniquement vers les mêmes contrats RAW/Store.
|
`ksp-worker-raw-transaction-ingest-lib` est désormais le premier Worker concret distinct du Job Backfill. Sa fondation source-neutral possède le lifecycle Start/Stop, l'admission bornée, la canonicalisation Common RAW, la persistance Store backend-neutral, le shutdown borné et les snapshots latest-value, mais aucune source réseau productive ni dépendance Transport. Les adapters live/catch-up seront ajoutés dans les tranches suivantes sans transformer le Worker en campagne historique. `ksp-job-backfill-lib` conserve de son côté son rôle historique paramétré et borné ; les deux producteurs restent indépendants et convergent uniquement vers les mêmes contrats RAW/Store.
|
||||||
|
|
||||||
## Points d'entrée
|
## Points d'entrée
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: ROADMAP.md -->
|
<!-- file: ROADMAP.md -->
|
||||||
<!-- version: 108 -->
|
<!-- version: 109 -->
|
||||||
|
|
||||||
# Roadmap KSP
|
# Roadmap KSP
|
||||||
|
|
||||||
@@ -103,7 +103,7 @@ RAW -> STRUCTURAL -> DECODED -> DOMAIN
|
|||||||
- [X] `0.3.8` — `ksp-app-store-desk` V1 RAW livrée sur le gabarit KSP courant comme application Tauri read-only backend-neutral. La Desk compose Config + Logging + `ksp-store-lib`, expose health/runtime sûrs, DataTables `serverSide` pour `RawTransaction`, `RawAccountState` et leurs observations, détails bornés, provenance sûre et états de rétention/tombstone, sans SQL/backend physique/Transport dans l'application. La nouvelle inspection random-access `offset + limit + counts exacts` reste distincte de la pagination machine cursor/keyset conservée pour workers/backfills/replays. Le gate final et les builds Linux `.deb`/`.rpm` sont verts.
|
- [X] `0.3.8` — `ksp-app-store-desk` V1 RAW livrée sur le gabarit KSP courant comme application Tauri read-only backend-neutral. La Desk compose Config + Logging + `ksp-store-lib`, expose health/runtime sûrs, DataTables `serverSide` pour `RawTransaction`, `RawAccountState` et leurs observations, détails bornés, provenance sûre et états de rétention/tombstone, sans SQL/backend physique/Transport dans l'application. La nouvelle inspection random-access `offset + limit + counts exacts` reste distincte de la pagination machine cursor/keyset conservée pour workers/backfills/replays. Le gate final et les builds Linux `.deb`/`.rpm` sont verts.
|
||||||
- [X] `0.3.9` — `ksp-worker-api` stable comme API générique et volontairement courte pour services continus : identité/kind bornés, lifecycle explicite, health/activity sûrs, stop token partagé, séquence/snapshot latest-value et `WorkerSnapshotSource` object-safe, avec dépendance Core-only et sans runtime/start-stop universel, Solana, Transport, Store, Config ou Tauri. Après freeze Worker API, l'audit `RawTransaction` a été consolidé dans une synthèse Store-centrique exhaustive séparant possibilités/support/preuve et classant HTTP, WS standard/provider, Yellowstone gRPC, blocs/slots, discovery + hydration, replay de continuité, archives et sources EARLY. Le Job Backfill historique paramétré et le Worker Ingest live start/stop sont deux producteurs indépendants du même Store ; `mainnet` est désormais l'identité KSP canonique et `mainnet-beta` un alias legacy/externe. Le handoff retient `ksp-raw-transaction-lib` comme lower-layer commune de canonicalisation RAW v1 pour `0.3.10`, sans edge Job ↔ Worker.
|
- [X] `0.3.9` — `ksp-worker-api` stable comme API générique et volontairement courte pour services continus : identité/kind bornés, lifecycle explicite, health/activity sûrs, stop token partagé, séquence/snapshot latest-value et `WorkerSnapshotSource` object-safe, avec dépendance Core-only et sans runtime/start-stop universel, Solana, Transport, Store, Config ou Tauri. Après freeze Worker API, l'audit `RawTransaction` a été consolidé dans une synthèse Store-centrique exhaustive séparant possibilités/support/preuve et classant HTTP, WS standard/provider, Yellowstone gRPC, blocs/slots, discovery + hydration, replay de continuité, archives et sources EARLY. Le Job Backfill historique paramétré et le Worker Ingest live start/stop sont deux producteurs indépendants du même Store ; `mainnet` est désormais l'identité KSP canonique et `mainnet-beta` un alias legacy/externe. Le handoff retient `ksp-raw-transaction-lib` comme lower-layer commune de canonicalisation RAW v1 pour `0.3.10`, sans edge Job ↔ Worker.
|
||||||
- [X] `0.3.10` — Lower-layer RAW Transaction commune stabilisée : `ksp-raw-transaction-lib` possède la canonicalisation RAW v1 source-neutral partagée, le Backfill est migré sans changement des golden bytes/hash, Transport expose `get_block_observed`, le wire Solana Legacy/V0/V1 est matérialisé et les canaris HTTP/WS standard/Helius/Yellowstone qualifient précisément les voies directes ou nécessitant hydration. La release se ferme sans runtime Worker concret ; celui-ci commence en `0.3.11` selon le redécoupage `0.3.11`–`0.3.14`.
|
- [X] `0.3.10` — Lower-layer RAW Transaction commune stabilisée : `ksp-raw-transaction-lib` possède la canonicalisation RAW v1 source-neutral partagée, le Backfill est migré sans changement des golden bytes/hash, Transport expose `get_block_observed`, le wire Solana Legacy/V0/V1 est matérialisé et les canaris HTTP/WS standard/Helius/Yellowstone qualifient précisément les voies directes ou nécessitant hydration. La release se ferme sans runtime Worker concret ; celui-ci commence en `0.3.11` selon le redécoupage `0.3.11`–`0.3.14`.
|
||||||
- [ ] `0.3.11` — Introduire la **fondation runtime** de `ksp-worker-raw-transaction-ingest-lib` sans source live complexe : crate/dependency firewall, settings source-neutral, handle/start-stop, lifecycle, snapshots, supervisor, bounded channels et pipeline de persistence/déduplication déterministe. Aucun protocole live n'est encore requis pour fermer cette release.
|
- [X] `0.3.11` — Fondation runtime de `ksp-worker-raw-transaction-ingest-lib` livrée : crate/dependency firewall, settings source-neutral, handle/start-stop, lifecycle, snapshots, supervisor, admission bornée, canonicalisation Common RAW, persistence/déduplication Store déterministe, backpressure et shutdown/fault hardening. La release se ferme volontairement sans source réseau productive ; Yellowstone + hydration HTTP commencent en `0.3.12`.
|
||||||
- [ ] `0.3.12` — Ajouter au Worker la voie **Yellowstone + hydration HTTP** : transactions/blocks/status, projection vers la common RAW, provenance sûre, `from_slot`, replay info, frontier de run et continuité propre au run, avec fixtures déterministes et smokes accessibles sans prétendre à un RAW-direct lorsque les métadonnées restent incomplètes.
|
- [ ] `0.3.12` — Ajouter au Worker la voie **Yellowstone + hydration HTTP** : transactions/blocks/status, projection vers la common RAW, provenance sûre, `from_slot`, replay info, frontier de run et continuité propre au run, avec fixtures déterministes et smokes accessibles sans prétendre à un RAW-direct lorsque les métadonnées restent incomplètes.
|
||||||
- [ ] `0.3.13` — Ajouter les voies **WS standard / Helius / HTTP live** et leur convergence multi-source : `logsSubscribe + getTransaction`, `blockSubscribe`, `transactionSubscribe + hydration`, polling blocs HTTP, observations multiples, content conflict explicite, coalescence et backpressure sans second actor Transport.
|
- [ ] `0.3.13` — Ajouter les voies **WS standard / Helius / HTTP live** et leur convergence multi-source : `logsSubscribe + getTransaction`, `blockSubscribe`, `transactionSubscribe + hydration`, polling blocs HTTP, observations multiples, content conflict explicite, coalescence et backpressure sans second actor Transport.
|
||||||
- [ ] `0.3.14` — Fermer le Worker par le **gap repair/hardening multi-source** : replay natif, source redondante, scan HTTP blocs, hydration de réparation, unresolved gaps visibles, politiques degraded/unhealthy/faulted, shutdown pendant repair, smokes provider accessibles et adapter EARLY seulement si réellement prouvé.
|
- [ ] `0.3.14` — Fermer le Worker par le **gap repair/hardening multi-source** : replay natif, source redondante, scan HTTP blocs, hydration de réparation, unresolved gaps visibles, politiques degraded/unhealthy/faulted, shutdown pendant repair, smokes provider accessibles et adapter EARLY seulement si réellement prouvé.
|
||||||
|
|||||||
20
crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
Normal file
20
crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
Normal file
@@ -0,0 +1,20 @@
|
|||||||
|
# file: crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
|
||||||
|
# version: 1
|
||||||
|
|
||||||
|
[package]
|
||||||
|
name = "ksp-worker-raw-transaction-ingest-lib"
|
||||||
|
version.workspace = true
|
||||||
|
edition.workspace = true
|
||||||
|
repository.workspace = true
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
ksp-core-lib = { path = "../ksp-core-lib" }
|
||||||
|
ksp-logging-lib = { path = "../ksp-logging-lib" }
|
||||||
|
ksp-raw-transaction-lib = { path = "../ksp-raw-transaction-lib" }
|
||||||
|
ksp-store-lib = { path = "../ksp-store-lib", default-features = false }
|
||||||
|
ksp-worker-api = { path = "../ksp-worker-api" }
|
||||||
|
sha2 = { workspace = true }
|
||||||
|
tokio = { workspace = true, features = ["macros", "rt", "sync", "time"] }
|
||||||
|
|
||||||
|
[lints]
|
||||||
|
workspace = true
|
||||||
158
crates/ksp-worker-raw-transaction-ingest-lib/README.md
Normal file
158
crates/ksp-worker-raw-transaction-ingest-lib/README.md
Normal file
@@ -0,0 +1,158 @@
|
|||||||
|
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/README.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# ksp-worker-raw-transaction-ingest-lib
|
||||||
|
|
||||||
|
`ksp-worker-raw-transaction-ingest-lib` est le premier Worker concret de KSP pour l'alimentation continue de la couche RAW Transaction.
|
||||||
|
|
||||||
|
La crate fournit la fondation runtime **source-neutral** du Worker : identité et settings bornés, lifecycle Start/Stop, admission privée bornée, canonicalisation via `ksp-raw-transaction-lib`, persistance backend-neutral via `ksp-store-lib`, supervision des tâches, shutdown borné et snapshots latest-value projetables sur `ksp-worker-api`.
|
||||||
|
|
||||||
|
La fondation ne contient volontairement encore **aucune source réseau productive**. Elle ne dépend pas de `ksp-onchain-transport-lib` et n'expose pas d'API publique permettant au caller d'injecter directement des transactions dans la queue interne. Les adapters live/catch-up sont des responsabilités ultérieures du même Worker, pas de son API de fondation.
|
||||||
|
|
||||||
|
## Identité et réseau
|
||||||
|
|
||||||
|
Une exécution est liée à :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawNetworkId
|
||||||
|
WorkerId
|
||||||
|
Worker kind = raw_transaction_ingest
|
||||||
|
```
|
||||||
|
|
||||||
|
Le réseau logique doit être identique à celui du `Store` remis au démarrage. La transaction canonique conserve l'identité durable définie par la couche RAW commune ; le Worker n'ajoute ni provider, ni endpoint, ni protocole à cette identité.
|
||||||
|
|
||||||
|
## Runtime
|
||||||
|
|
||||||
|
Le point d'entrée public est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestWorker::start(settings, Arc<Store>)
|
||||||
|
-> RawTransactionIngestHandle
|
||||||
|
```
|
||||||
|
|
||||||
|
Le démarrage est synchrone mais nécessite qu'un runtime Tokio courant appartienne déjà au caller. Le Worker ne crée pas de runtime global et n'expose aucun `JoinHandle` public.
|
||||||
|
|
||||||
|
`RawTransactionIngestHandle` permet de :
|
||||||
|
|
||||||
|
- demander un stop coopératif et idempotent ;
|
||||||
|
- obtenir une source de snapshots concrets latest-value ;
|
||||||
|
- utiliser cette même source via `WorkerSnapshotSource` ;
|
||||||
|
- attendre le terminal après drain/abort+join des tâches possédées.
|
||||||
|
|
||||||
|
La destruction du dernier handle de contrôle ferme aussi la voie de contrôle privée ; le runtime termine alors selon les mêmes règles de shutdown.
|
||||||
|
|
||||||
|
## Admission et Common RAW
|
||||||
|
|
||||||
|
La queue centrale est un `tokio::sync::mpsc` privé borné par `admission_queue_capacity`. Les sources internes doivent subir la backpressure du channel ; aucune queue non bornée ni silent drop n'est autorisé.
|
||||||
|
|
||||||
|
Chaque ingress admis est :
|
||||||
|
|
||||||
|
1. vérifié contre le réseau attendu ;
|
||||||
|
2. canonicalisé exclusivement par `ksp-raw-transaction-lib` ;
|
||||||
|
3. associé à une observation key déterministe sous le domaine `ksp.raw_transaction_ingest.observation.v1` ;
|
||||||
|
4. assemblé en acquisition RAW commune ;
|
||||||
|
5. remis à la persistence Store.
|
||||||
|
|
||||||
|
La crate ne possède pas un second format RAW et ne duplique pas le canonicaliseur commun.
|
||||||
|
|
||||||
|
## Persistence Store
|
||||||
|
|
||||||
|
La persistance passe uniquement par `ksp-store-lib` avec `default-features = false` dans la crate Worker. Aucun backend physique n'est imposé ou importé directement.
|
||||||
|
|
||||||
|
L'écriture utilise le mode normal d'acquisition atomique `RawTransaction + RawTransactionObservation`. Les outcomes distingués sont notamment :
|
||||||
|
|
||||||
|
```text
|
||||||
|
entity inserted
|
||||||
|
entity already present
|
||||||
|
entity skipped purged
|
||||||
|
observation inserted
|
||||||
|
observation already present
|
||||||
|
observation not recorded for purged entity
|
||||||
|
content conflict
|
||||||
|
store failure
|
||||||
|
```
|
||||||
|
|
||||||
|
`persistence_concurrency` borne le nombre d'écritures Store simultanées. Un conflit de contenu est terminal et n'est jamais converti en succès idempotent.
|
||||||
|
|
||||||
|
## Shutdown et faults
|
||||||
|
|
||||||
|
Le shutdown possède une deadline bornée par `shutdown_drain_timeout`.
|
||||||
|
|
||||||
|
Avant publication terminale, le supervisor :
|
||||||
|
|
||||||
|
- arrête les nouvelles admissions ;
|
||||||
|
- signale le stop aux sources privées ;
|
||||||
|
- draine le travail déjà admis tant que la deadline le permet ;
|
||||||
|
- récolte les persistences et sources possédées ;
|
||||||
|
- en cas de timeout, abort les tâches restantes puis les rejoint avant le terminal.
|
||||||
|
|
||||||
|
Les codes d'erreur publics du Worker sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.settings_invalid
|
||||||
|
worker_raw_transaction_ingest.runtime_invalid
|
||||||
|
worker_raw_transaction_ingest.store_failed
|
||||||
|
worker_raw_transaction_ingest.content_conflict
|
||||||
|
worker_raw_transaction_ingest.counter_exhausted
|
||||||
|
worker_raw_transaction_ingest.source_failed
|
||||||
|
worker_raw_transaction_ingest.drain_timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
Les diagnostics ne recopient pas de payload RAW, URL, credential, signature hostile ou texte backend/provider arbitraire.
|
||||||
|
|
||||||
|
## Snapshots
|
||||||
|
|
||||||
|
`RawTransactionIngestSnapshotSource` est latest-value. Les listeners peuvent rater des états intermédiaires mais obtiennent toujours une valeur complète et monotone par `WorkerSnapshotSequence`.
|
||||||
|
|
||||||
|
Le snapshot concret expose notamment :
|
||||||
|
|
||||||
|
```text
|
||||||
|
lifecycle / health / activity
|
||||||
|
admission_queue_capacity / admission_queue_depth
|
||||||
|
persistence_concurrency / in_flight_persistence
|
||||||
|
admitted_total / canonicalized_total / persisted_total
|
||||||
|
entity_inserted_total / entity_already_present_total / entity_skipped_purged_total
|
||||||
|
observation_inserted_total / observation_already_present_total
|
||||||
|
content_conflict_total / store_failure_total / source_failure_total
|
||||||
|
backpressure_wait_total
|
||||||
|
```
|
||||||
|
|
||||||
|
Les compteurs ne wrapent jamais silencieusement.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
Les dépendances normales sont exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-core-lib
|
||||||
|
ksp-logging-lib
|
||||||
|
ksp-raw-transaction-lib
|
||||||
|
ksp-store-lib (default-features = false)
|
||||||
|
ksp-worker-api
|
||||||
|
sha2
|
||||||
|
tokio (macros, rt, sync, time)
|
||||||
|
```
|
||||||
|
|
||||||
|
La crate ne dépend pas de Config, Job, `ksp-store-api` directement, backend Store concret, Transport, Tauri ou SDK provider.
|
||||||
|
|
||||||
|
## Hors périmètre de la fondation source-neutral
|
||||||
|
|
||||||
|
Cette surface ne possède pas encore :
|
||||||
|
|
||||||
|
- adapter HTTP/WS/Yellowstone productif ;
|
||||||
|
- sélection Config de sources/endpoints/credentials ;
|
||||||
|
- discovery/hydration/replay de continuité live ;
|
||||||
|
- hot reconfiguration de listeners/sources ;
|
||||||
|
- application Desk ou process autonome ;
|
||||||
|
- campagne historique/backfill ;
|
||||||
|
- décodage STRUCTURAL/DECODED/DOMAIN.
|
||||||
|
|
||||||
|
Ces extensions doivent conserver la séparation avec `ksp-job-backfill-lib` et réutiliser les mêmes contrats RAW/Store.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- [`USAGE.md`](USAGE.md) — utilisation de la façade publique actuelle ;
|
||||||
|
- [`../ksp-worker-api/README.md`](../ksp-worker-api/README.md) — contrats Worker génériques ;
|
||||||
|
- [`../ksp-raw-transaction-lib/README.md`](../ksp-raw-transaction-lib/README.md) — canonicalisation RAW commune ;
|
||||||
|
- [`../../docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md`](../../docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md) — séparation Worker/Job ;
|
||||||
|
- [`../../docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md`](../../docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md) — architecture d'acquisition RawTransaction.
|
||||||
206
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
Normal file
206
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
Normal file
@@ -0,0 +1,206 @@
|
|||||||
|
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Utilisation de ksp-worker-raw-transaction-ingest-lib
|
||||||
|
|
||||||
|
Cette page décrit l'utilisation de la façade publique source-neutral de `ksp-worker-raw-transaction-ingest-lib`. Le caller possède la composition du runtime Tokio et du `Store` ; la crate Worker ne construit ni Config, ni backend physique, ni Transport.
|
||||||
|
|
||||||
|
## Construire les settings
|
||||||
|
|
||||||
|
Les identités sont validées par leurs couches propriétaires :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn worker_settings() -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-mainnet-0001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::with_defaults(
|
||||||
|
network,
|
||||||
|
worker_id,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker kind est fixe :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
assert_eq!(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE,
|
||||||
|
"raw_transaction_ingest",
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
Pour des limites explicites :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let settings = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::new(
|
||||||
|
network,
|
||||||
|
worker_id,
|
||||||
|
512,
|
||||||
|
16,
|
||||||
|
std::time::Duration::from_secs(10),
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
Bornes publiques :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission_queue_capacity 1 ..= 65_536 défaut 256
|
||||||
|
persistence_concurrency 1 ..= 64 défaut 8
|
||||||
|
shutdown_drain_timeout 100 ms ..= 30 s défaut 5 s
|
||||||
|
```
|
||||||
|
|
||||||
|
Une valeur hors borne retourne `worker_raw_transaction_ingest.settings_invalid` avec uniquement le nom stable du champ invalide.
|
||||||
|
|
||||||
|
## Démarrer sur le runtime du caller
|
||||||
|
|
||||||
|
`RawTransactionIngestWorker::start` doit être appelé depuis un contexte possédant déjà un runtime Tokio courant. Le `Store` est passé dans un `Arc` et doit cibler exactement le même `RawNetworkId` que les settings.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
async fn start_worker(
|
||||||
|
settings: ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings,
|
||||||
|
store: std::sync::Arc<ksp_store_lib::Store>,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle> {
|
||||||
|
return ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWorker::start(settings, store);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le démarrage retourne avant la fin du Worker. Aucun `JoinHandle` Tokio n'est exposé au caller.
|
||||||
|
|
||||||
|
Un appel hors runtime Tokio courant retourne une erreur `worker_raw_transaction_ingest.runtime_invalid`. Un mismatch réseau Store/settings est également rejeté avant le spawn.
|
||||||
|
|
||||||
|
## Observer le snapshot concret
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let current = source.current();
|
||||||
|
|
||||||
|
let sequence = current.worker_snapshot().sequence();
|
||||||
|
let state = current.worker_snapshot().state();
|
||||||
|
let queue_depth = current.admission_queue_depth();
|
||||||
|
let in_flight = current.in_flight_persistence();
|
||||||
|
```
|
||||||
|
|
||||||
|
Pour attendre une valeur plus récente :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let observed = source.current().worker_snapshot().sequence();
|
||||||
|
let newer = source.wait_for_change(observed).await;
|
||||||
|
assert!(newer.worker_snapshot().sequence().is_after(observed));
|
||||||
|
```
|
||||||
|
|
||||||
|
Le flux est latest-value : les transitions intermédiaires peuvent être coalescées. Ne pas l'utiliser comme journal d'événements exhaustif.
|
||||||
|
|
||||||
|
## Utiliser la projection Worker API
|
||||||
|
|
||||||
|
La même source implémente `ksp_worker_api::WorkerSnapshotSource` :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let source = handle.worker_snapshot_source();
|
||||||
|
let current = ksp_worker_api::WorkerSnapshotSource::current(&source);
|
||||||
|
let observed = current.sequence();
|
||||||
|
let newer = ksp_worker_api::WorkerSnapshotSource::wait_for_change(&source, observed).await;
|
||||||
|
assert!(newer.sequence().is_after(observed));
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette projection commune ne contient que les dimensions génériques Worker. Les compteurs d'admission/persistence restent disponibles sur `RawTransactionIngestSnapshot`.
|
||||||
|
|
||||||
|
## Lire les compteurs concrets
|
||||||
|
|
||||||
|
Les compteurs principaux sont monotones :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
|
||||||
|
let admitted = snapshot.admitted_total();
|
||||||
|
let canonicalized = snapshot.canonicalized_total();
|
||||||
|
let persisted = snapshot.persisted_total();
|
||||||
|
let inserted = snapshot.entity_inserted_total();
|
||||||
|
let existing = snapshot.entity_already_present_total();
|
||||||
|
let purged = snapshot.entity_skipped_purged_total();
|
||||||
|
let observations = snapshot.observation_inserted_total();
|
||||||
|
let conflicts = snapshot.content_conflict_total();
|
||||||
|
let store_failures = snapshot.store_failure_total();
|
||||||
|
let source_failures = snapshot.source_failure_total();
|
||||||
|
let backpressure = snapshot.backpressure_wait_total();
|
||||||
|
```
|
||||||
|
|
||||||
|
`admission_queue_depth()` et `in_flight_persistence()` sont des gauges latest-value, pas des totaux cumulés.
|
||||||
|
|
||||||
|
L'épuisement d'un compteur ou de la séquence est terminal et utilise `worker_raw_transaction_ingest.counter_exhausted` au lieu de wrapper silencieusement.
|
||||||
|
|
||||||
|
## Demander un stop et attendre le terminal
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let accepted = handle.request_stop();
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
|
||||||
|
match terminal {
|
||||||
|
std::result::Result::Ok(state) => {
|
||||||
|
assert!(state.is_terminal());
|
||||||
|
}
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
}
|
||||||
|
|
||||||
|
if accepted {
|
||||||
|
// La première demande de stop a été remise au runtime vivant.
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`request_stop()` est idempotent. Il exprime une intention coopérative ; le terminal n'est publié qu'après traitement du drain et des tâches possédées.
|
||||||
|
|
||||||
|
Le shutdown est borné par `shutdown_drain_timeout`. Si les tâches ne peuvent pas être drainées à temps, le Worker les abort puis les rejoint avant de publier `worker_raw_transaction_ingest.drain_timeout`.
|
||||||
|
|
||||||
|
## Interpréter les terminaux faultés
|
||||||
|
|
||||||
|
Les codes Worker publics sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
settings_invalid settings techniques hors contrat
|
||||||
|
runtime_invalid invariant runtime/lifecycle impossible
|
||||||
|
store_failed erreur Store non-conflict classifiée
|
||||||
|
content_conflict transaction canonique incompatible avec le contenu durable
|
||||||
|
counter_exhausted compteur/séquence monotone arrivé à sa borne
|
||||||
|
source_failed tâche source privée terminée en erreur
|
||||||
|
drain_timeout drain du shutdown hors deadline
|
||||||
|
```
|
||||||
|
|
||||||
|
Un consumer doit traiter le `ErrorCode` comme contrat stable et ne pas dépendre d'un texte backend/provider arbitraire.
|
||||||
|
|
||||||
|
## Frontière d'admission
|
||||||
|
|
||||||
|
La queue d'admission et le type ingress sont volontairement privés. La façade publique actuelle ne propose ni `send`, ni `enqueue`, ni registration d'un adapter réseau.
|
||||||
|
|
||||||
|
Les adapters d'acquisition qui alimentent cette queue appartiennent à l'implémentation du Worker et doivent conserver :
|
||||||
|
|
||||||
|
```text
|
||||||
|
backpressure mpsc bornée
|
||||||
|
stop prioritaire sur send bloqué
|
||||||
|
canonicalisation via ksp-raw-transaction-lib
|
||||||
|
persistance via ksp-store-lib
|
||||||
|
aucun backend physique direct
|
||||||
|
aucun provider dans l'identité canonique
|
||||||
|
```
|
||||||
|
|
||||||
|
Le caller ne doit pas contourner cette frontière en écrivant directement dans les structures privées du Worker.
|
||||||
|
|
||||||
|
## Composition supérieure
|
||||||
|
|
||||||
|
Le pattern de composition attendu reste :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Config / application / service owner
|
||||||
|
-> construit le Store
|
||||||
|
-> choisit ultérieurement les sources/adapters supportés
|
||||||
|
-> construit RawTransactionIngestSettings
|
||||||
|
-> démarre RawTransactionIngestWorker
|
||||||
|
-> observe RawTransactionIngestSnapshotSource
|
||||||
|
-> demande stop lorsque nécessaire
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker ne reçoit pas de requête historique métier. Une campagne `signature/program_id/plage/limite` appartient à `ksp-job-backfill-lib`, pas à ce runtime continu.
|
||||||
110
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
Normal file
110
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
use sha2::Digest; // rust-rules: trait-import
|
||||||
|
|
||||||
|
const RAW_TRANSACTION_INGEST_OBSERVATION_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.observation.v1\0";
|
||||||
|
|
||||||
|
/// Crate-private source-neutral ingress admitted by the central bounded Worker queue.
|
||||||
|
pub(crate) struct RawTransactionIngress {
|
||||||
|
/// Complete Common RAW material supplied by one private source task.
|
||||||
|
pub(crate) material: ksp_raw_transaction_lib::RawTransactionMaterial,
|
||||||
|
/// Logical network expected to match both Worker settings and canonical material.
|
||||||
|
pub(crate) network: ksp_store_lib::RawNetworkId,
|
||||||
|
/// Safe source-independent provenance attached to the acquisition observation.
|
||||||
|
pub(crate) provenance: ksp_store_lib::RawAcquisitionProvenance,
|
||||||
|
/// Opaque deterministic source-owned key material used only for Worker observation-key derivation.
|
||||||
|
pub(crate) source_key: [u8; 32],
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Receiver side of the bounded central RAW transaction admission queue owned by the Worker supervisor.
|
||||||
|
pub(crate) struct RawTransactionAdmission {
|
||||||
|
backpressure_wait_observed: bool,
|
||||||
|
capacity: usize,
|
||||||
|
receiver: tokio::sync::mpsc::Receiver<crate::RawTransactionIngress>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionAdmission {
|
||||||
|
/// Creates one bounded admission queue and returns its crate-private source sender.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn new(capacity: usize) -> (crate::RawTransactionAdmission, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) {
|
||||||
|
let (sender, receiver) = tokio::sync::mpsc::channel(capacity);
|
||||||
|
return (Self { backpressure_wait_observed: false, capacity, receiver }, sender);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest receiver-side queue depth without consuming an ingress entry.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn queue_depth(&self) -> usize {
|
||||||
|
return self.receiver.len();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Closes new admissions while preserving already queued ingress for deterministic drain.
|
||||||
|
pub(crate) fn close(&mut self) {
|
||||||
|
self.receiver.close();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Receives and converts one queued ingress into a common RAW transaction acquisition.
|
||||||
|
pub(crate) async fn receive(
|
||||||
|
&mut self,
|
||||||
|
expected_network: &ksp_store_lib::RawNetworkId,
|
||||||
|
) -> ksp_core_lib::Result<std::option::Option<ksp_raw_transaction_lib::RawTransactionAcquisition>> {
|
||||||
|
self.backpressure_wait_observed = self.receiver.len() == self.capacity;
|
||||||
|
let ingress = match self.receiver.recv().await {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Ok(std::option::Option::None),
|
||||||
|
};
|
||||||
|
let acquisition = canonicalize_ingress(expected_network, ingress);
|
||||||
|
return match acquisition {
|
||||||
|
std::result::Result::Ok(value) => std::result::Result::Ok(std::option::Option::Some(value)),
|
||||||
|
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Takes and clears the source-neutral signal that the previous dequeue observed the bounded queue at full capacity.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn take_backpressure_wait_observed(&mut self) -> bool {
|
||||||
|
let observed = self.backpressure_wait_observed;
|
||||||
|
self.backpressure_wait_observed = false;
|
||||||
|
return observed;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn canonicalize_ingress(
|
||||||
|
expected_network: &ksp_store_lib::RawNetworkId,
|
||||||
|
ingress: crate::RawTransactionIngress,
|
||||||
|
) -> ksp_core_lib::Result<ksp_raw_transaction_lib::RawTransactionAcquisition> {
|
||||||
|
if &ingress.network != expected_network {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("admission.network_mismatch"));
|
||||||
|
}
|
||||||
|
let transaction = ksp_raw_transaction_lib::canonicalize_raw_transaction(ingress.material);
|
||||||
|
let transaction = match transaction {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("admission.canonicalization_invalid")),
|
||||||
|
};
|
||||||
|
if transaction.reference().network() != expected_network {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("admission.material_network_mismatch"));
|
||||||
|
}
|
||||||
|
let observation_key = observation_key(transaction.reference(), &ingress.source_key);
|
||||||
|
return std::result::Result::Ok(ksp_raw_transaction_lib::assemble_raw_transaction_acquisition(transaction, observation_key, ingress.provenance));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn hash_bytes(hasher: &mut sha2::Sha256, value: &[u8]) {
|
||||||
|
hasher.update((value.len() as u64).to_be_bytes());
|
||||||
|
hasher.update(value);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn observation_key(reference: &ksp_store_lib::RawTransactionReference, source_key: &[u8; 32]) -> ksp_store_lib::RawObservationKey {
|
||||||
|
let mut hasher = sha2::Sha256::new();
|
||||||
|
hasher.update(RAW_TRANSACTION_INGEST_OBSERVATION_DOMAIN);
|
||||||
|
hash_bytes(&mut hasher, reference.network().as_str().as_bytes());
|
||||||
|
hash_bytes(&mut hasher, reference.signature().as_bytes());
|
||||||
|
hash_bytes(&mut hasher, source_key);
|
||||||
|
let bytes: [u8; 32] = hasher.finalize().into();
|
||||||
|
return ksp_store_lib::RawObservationKey::new(bytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/admission.rs"]
|
||||||
|
mod tests;
|
||||||
54
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
Normal file
54
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
// version: 6
|
||||||
|
|
||||||
|
/// Error code used when durable Store content conflicts with one admitted canonical RAW transaction.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "content_conflict");
|
||||||
|
/// Error code used when one monotone RAW transaction ingest Worker counter or snapshot sequence cannot advance without wrapping.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "counter_exhausted");
|
||||||
|
/// Error code used when private Worker tasks cannot drain before the configured shutdown deadline.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "drain_timeout");
|
||||||
|
/// Error code used when the RAW transaction ingest Worker reaches an invalid runtime or lifecycle condition.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "runtime_invalid");
|
||||||
|
/// Error code used when RAW transaction ingest Worker settings violate one bounded runtime invariant.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "settings_invalid");
|
||||||
|
/// Error code used when one private source task terminates with a classified failure.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "source_failed");
|
||||||
|
/// Error code used when Store persistence fails for one admitted canonical RAW transaction.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "store_failed");
|
||||||
|
|
||||||
|
/// Creates one terminal counter-exhaustion error without exposing runtime material.
|
||||||
|
pub(crate) fn counter_exhausted_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED, "RAW transaction ingest Worker counter exhausted")
|
||||||
|
.with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one terminal content-conflict error without copying conflicting material into diagnostics.
|
||||||
|
pub(crate) fn content_conflict_error() -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT, "RAW transaction ingest Store content conflict");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one runtime-domain error carrying only one stable internal condition code.
|
||||||
|
pub(crate) fn runtime_error(condition: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID, "invalid RAW transaction ingest Worker runtime state")
|
||||||
|
.with_context("condition", condition);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one settings-domain error carrying only the stable invalid field name.
|
||||||
|
pub(crate) fn settings_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID, "invalid RAW transaction ingest Worker settings")
|
||||||
|
.with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one terminal Store error while retaining only the already-safe lower-layer ErrorCode.
|
||||||
|
pub(crate) fn store_error(store_code: ksp_core_lib::ErrorCode) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED, "RAW transaction ingest Store persistence failed")
|
||||||
|
.with_context("store_domain", store_code.domain())
|
||||||
|
.with_context("store_code", store_code.code());
|
||||||
|
}
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/identity.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
/// Stable Worker kind code used by the continuous RAW transaction ingest vertical.
|
||||||
|
pub const RAW_TRANSACTION_INGEST_WORKER_KIND_CODE: &str = "raw_transaction_ingest";
|
||||||
98
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
Normal file
98
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
Normal file
@@ -0,0 +1,98 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
// version: 8
|
||||||
|
|
||||||
|
#![warn(missing_docs)]
|
||||||
|
#![deny(unreachable_pub)]
|
||||||
|
#![forbid(unsafe_code)]
|
||||||
|
|
||||||
|
//! Source-neutral runtime foundation for continuous KSP RAW transaction ingestion.
|
||||||
|
//!
|
||||||
|
//! This tranche owns the concrete Worker family identity, validated technical settings
|
||||||
|
//! and the caller-runtime-owned lifecycle with private child-task supervision. This tranche also
|
||||||
|
//! owns bounded source-neutral admission, common RAW canonicalization/assembly and backend-neutral
|
||||||
|
//! Store persistence in normal mode plus concrete latest-value snapshots projected onto Worker API;
|
||||||
|
//! no live source or Transport dependency exists.
|
||||||
|
|
||||||
|
mod admission;
|
||||||
|
mod error;
|
||||||
|
mod identity;
|
||||||
|
mod persistence;
|
||||||
|
mod runtime;
|
||||||
|
mod settings;
|
||||||
|
mod snapshot;
|
||||||
|
|
||||||
|
/// Error code used when durable Store content conflicts with one admitted canonical RAW transaction.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT;
|
||||||
|
/// Error code used when one monotone Worker counter or snapshot sequence cannot advance without wrapping.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED;
|
||||||
|
/// Error code used when private Worker tasks cannot drain before the configured shutdown deadline.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT;
|
||||||
|
/// Error code used when the RAW transaction ingest Worker reaches an invalid runtime or lifecycle condition.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID;
|
||||||
|
/// Error code used when RAW transaction ingest Worker settings violate one bounded runtime invariant.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID;
|
||||||
|
/// Error code used when one private source task terminates with a classified failure.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED;
|
||||||
|
/// Error code used when Store persistence fails for one admitted canonical RAW transaction.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED;
|
||||||
|
/// Stable Worker kind code used by the continuous RAW transaction ingest vertical.
|
||||||
|
pub use self::identity::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE;
|
||||||
|
/// Cloneable external control handle for one continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime::RawTransactionIngestHandle;
|
||||||
|
/// Runtime-neutral boxed future resolving after one RAW transaction ingest Worker has fully reached a terminal lifecycle state.
|
||||||
|
pub use self::runtime::RawTransactionIngestTerminalFuture;
|
||||||
|
/// Entry point owning synchronous validation and task launch for one RAW transaction ingest Worker run.
|
||||||
|
pub use self::runtime::RawTransactionIngestWorker;
|
||||||
|
/// Default bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY;
|
||||||
|
/// Default number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY;
|
||||||
|
/// Default cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT;
|
||||||
|
/// Maximum bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY;
|
||||||
|
/// Maximum number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY;
|
||||||
|
/// Maximum cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT;
|
||||||
|
/// Minimum bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY;
|
||||||
|
/// Minimum number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY;
|
||||||
|
/// Minimum cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT;
|
||||||
|
/// Validated source-neutral runtime settings for one continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::RawTransactionIngestSettings;
|
||||||
|
/// Complete safe latest-value snapshot of one continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::snapshot::RawTransactionIngestSnapshot;
|
||||||
|
/// Runtime-neutral boxed future resolving to one newer concrete RAW transaction ingest Worker snapshot.
|
||||||
|
pub use self::snapshot::RawTransactionIngestSnapshotFuture;
|
||||||
|
/// Cloneable latest-value source exposing concrete and common Worker snapshots from one shared watch state.
|
||||||
|
pub use self::snapshot::RawTransactionIngestSnapshotSource;
|
||||||
|
|
||||||
|
/// Receiver-side owner of the private bounded RAW transaction admission queue.
|
||||||
|
pub(crate) use self::admission::RawTransactionAdmission;
|
||||||
|
/// Crate-private source-neutral ingress sent through the bounded central admission queue.
|
||||||
|
pub(crate) use self::admission::RawTransactionIngress;
|
||||||
|
/// Creates one terminal content-conflict error without copying conflicting material into diagnostics.
|
||||||
|
pub(crate) use self::error::content_conflict_error;
|
||||||
|
/// Creates one terminal counter-exhaustion error without exposing runtime material.
|
||||||
|
pub(crate) use self::error::counter_exhausted_error;
|
||||||
|
/// Creates one runtime-domain error without copying runtime/provider/Store values into diagnostics.
|
||||||
|
pub(crate) use self::error::runtime_error;
|
||||||
|
/// Creates one settings-domain error without copying caller-supplied values into diagnostics.
|
||||||
|
pub(crate) use self::error::settings_error;
|
||||||
|
/// Creates one terminal Store error while retaining only the already-safe lower-layer ErrorCode.
|
||||||
|
pub(crate) use self::error::store_error;
|
||||||
|
/// Canonical entity disposition produced by one successful Worker Store persistence attempt.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestEntityPersistence;
|
||||||
|
/// Observation disposition produced by one successful Worker Store persistence attempt.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestObservationPersistence;
|
||||||
|
/// Classified successful outcome of one atomic Worker Store persistence attempt.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestPersistenceOutcome;
|
||||||
|
/// Private backend-neutral Store persistence port used by the Worker and deterministic tests.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestPersistencePort;
|
||||||
|
/// Persists one already-canonical Worker acquisition through the private Store port in `Normal` mode.
|
||||||
|
pub(crate) use self::persistence::persist_raw_transaction_ingest_acquisition;
|
||||||
|
/// Private latest-value publisher and checked counter owner shared by the Worker supervisor.
|
||||||
|
pub(crate) use self::snapshot::RawTransactionIngestSnapshotPublisher;
|
||||||
138
crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
Normal file
138
crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
Normal file
@@ -0,0 +1,138 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
/// Canonical entity disposition produced by one Worker Store persistence attempt.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub(crate) enum RawTransactionIngestEntityPersistence {
|
||||||
|
/// The canonical RAW transaction was inserted for the first time.
|
||||||
|
Inserted,
|
||||||
|
/// Identical canonical RAW transaction content was already durable.
|
||||||
|
AlreadyPresent,
|
||||||
|
/// Normal persistence respected an existing purge tombstone.
|
||||||
|
SkippedPurged,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Observation disposition produced by one Worker Store persistence attempt.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub(crate) enum RawTransactionIngestObservationPersistence {
|
||||||
|
/// The deterministic acquisition observation was inserted for the first time.
|
||||||
|
Inserted,
|
||||||
|
/// The same deterministic acquisition observation was already durable.
|
||||||
|
AlreadyPresent,
|
||||||
|
/// No observation was recorded because the canonical entity was intentionally skipped.
|
||||||
|
NotRecorded,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Classified successful outcome of one atomic Worker Store persistence attempt.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub(crate) struct RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
/// Returns the canonical entity persistence disposition.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) const fn entity(self) -> crate::RawTransactionIngestEntityPersistence {
|
||||||
|
return self.entity;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the acquisition-observation persistence disposition.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) const fn observation(self) -> crate::RawTransactionIngestObservationPersistence {
|
||||||
|
return self.observation;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Private backend-neutral Store persistence port used by the Worker and deterministic tests.
|
||||||
|
pub(crate) trait RawTransactionIngestPersistencePort: std::marker::Send + std::marker::Sync {
|
||||||
|
/// Returns whether the persistence target belongs to the requested logical network.
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool;
|
||||||
|
|
||||||
|
/// Executes one atomic canonical RAW transaction plus observation Store write.
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
transaction: ksp_store_lib::RawTransaction,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>>;
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistencePort for ksp_store_lib::Store {
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool {
|
||||||
|
return self.runtime_snapshot().network() == network;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
transaction: ksp_store_lib::RawTransaction,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>> {
|
||||||
|
return ksp_store_lib::RawTransactionWrite::persist_raw_transaction_acquisition(self, transaction, observation, mode);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Persists one already-canonical Worker acquisition through the private Store port in `Normal` mode.
|
||||||
|
pub(crate) async fn persist_raw_transaction_ingest_acquisition<P>(
|
||||||
|
port: &P,
|
||||||
|
acquisition: ksp_raw_transaction_lib::RawTransactionAcquisition,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>
|
||||||
|
where
|
||||||
|
P: crate::RawTransactionIngestPersistencePort + ?Sized,
|
||||||
|
{
|
||||||
|
let reference = acquisition.transaction().reference().clone();
|
||||||
|
if !port.network_matches(reference.network()) {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("persistence.store_network_mismatch"));
|
||||||
|
}
|
||||||
|
if acquisition.observation().transaction() != &reference {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("persistence.acquisition_reference_mismatch"));
|
||||||
|
}
|
||||||
|
let (transaction, observation) = acquisition.into_parts();
|
||||||
|
let result = port.persist_acquisition(transaction, observation, ksp_store_lib::RawTransactionAcquisitionMode::Normal).await;
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(outcome) => map_store_outcome(outcome),
|
||||||
|
std::result::Result::Err(error) => map_store_error(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn map_store_error(error: ksp_core_lib::Error) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome> {
|
||||||
|
if error.code() == ksp_store_lib::ERROR_CODE_RAW_CONFLICT {
|
||||||
|
return std::result::Result::Err(crate::content_conflict_error());
|
||||||
|
}
|
||||||
|
return std::result::Result::Err(crate::store_error(error.code()));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn map_store_outcome(outcome: ksp_store_lib::RawAcquisitionWriteOutcome) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome> {
|
||||||
|
let entity = outcome.entity();
|
||||||
|
let observation = outcome.observation();
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::Inserted && observation == ksp_store_lib::RawObservationWriteOutcome::Inserted {
|
||||||
|
return std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::Inserted,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent && observation == ksp_store_lib::RawObservationWriteOutcome::Inserted {
|
||||||
|
return std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent && observation == ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent {
|
||||||
|
return std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::AlreadyPresent,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::SkippedPurged && observation == ksp_store_lib::RawObservationWriteOutcome::NotRecorded {
|
||||||
|
return std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::SkippedPurged,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::NotRecorded,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return std::result::Result::Err(crate::runtime_error("persistence.store_outcome_invalid"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/persistence.rs"]
|
||||||
|
mod tests;
|
||||||
557
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
Normal file
557
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
Normal file
@@ -0,0 +1,557 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
// version: 7
|
||||||
|
|
||||||
|
type PersistencePort = std::sync::Arc<dyn crate::RawTransactionIngestPersistencePort + 'static>;
|
||||||
|
type PersistenceTasks = tokio::task::JoinSet<ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>>;
|
||||||
|
type SourceTasks = tokio::task::JoinSet<ksp_core_lib::Result<()>>;
|
||||||
|
|
||||||
|
/// Runtime-neutral boxed future resolving after one RAW transaction ingest Worker has fully reached a terminal lifecycle state.
|
||||||
|
pub type RawTransactionIngestTerminalFuture<'a> =
|
||||||
|
std::pin::Pin<std::boxed::Box<dyn std::future::Future<Output = ksp_core_lib::Result<ksp_worker_api::WorkerState>> + std::marker::Send + 'a>>;
|
||||||
|
|
||||||
|
/// Cloneable external control handle for one continuous RAW transaction ingest Worker.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct RawTransactionIngestHandle {
|
||||||
|
snapshots: crate::RawTransactionIngestSnapshotSource,
|
||||||
|
stop_sender: tokio::sync::watch::Sender<bool>,
|
||||||
|
stop_token: ksp_worker_api::WorkerStopToken,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestHandle {
|
||||||
|
/// Requests cooperative stop and returns `true` only for the first request accepted by the live runtime.
|
||||||
|
#[must_use]
|
||||||
|
pub fn request_stop(&self) -> bool {
|
||||||
|
if !self.stop_token.request_stop() {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return self.stop_sender.send(true).is_ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns an independent latest-value source for concrete RAW transaction ingest Worker snapshots.
|
||||||
|
#[must_use]
|
||||||
|
pub fn snapshot_source(&self) -> crate::RawTransactionIngestSnapshotSource {
|
||||||
|
return self.snapshots.clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns an independent source implementing the common [`ksp_worker_api::WorkerSnapshotSource`] projection.
|
||||||
|
#[must_use]
|
||||||
|
pub fn worker_snapshot_source(&self) -> crate::RawTransactionIngestSnapshotSource {
|
||||||
|
return self.snapshots.clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Waits until the private runtime task has published one terminal lifecycle state after draining owned work.
|
||||||
|
#[must_use]
|
||||||
|
pub fn wait_terminal(&self) -> crate::RawTransactionIngestTerminalFuture<'_> {
|
||||||
|
let source = self.snapshots.clone();
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
loop {
|
||||||
|
let current = source.current();
|
||||||
|
let state = current.worker_snapshot().state();
|
||||||
|
if state.is_terminal() {
|
||||||
|
return std::result::Result::Ok(state);
|
||||||
|
}
|
||||||
|
let observed = current.worker_snapshot().sequence();
|
||||||
|
let changed = source.wait_for_change(observed).await;
|
||||||
|
let state = changed.worker_snapshot().state();
|
||||||
|
if state.is_terminal() {
|
||||||
|
return std::result::Result::Ok(state);
|
||||||
|
}
|
||||||
|
if changed.worker_snapshot().sequence() == observed && source.is_closed() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("terminal.closed_before_terminal"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionIngestHandle {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
let snapshot = self.snapshots.current();
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionIngestHandle")
|
||||||
|
.field("stop_requested", &self.stop_token.is_stop_requested())
|
||||||
|
.field("sequence", &snapshot.worker_snapshot().sequence())
|
||||||
|
.field("state", &snapshot.worker_snapshot().state())
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Entry point owning synchronous validation and task launch for one RAW transaction ingest Worker run.
|
||||||
|
pub struct RawTransactionIngestWorker;
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestWorker {
|
||||||
|
/// Starts one Worker on the caller-owned current Tokio runtime while retaining the caller-owned Store facade through an `Arc`.
|
||||||
|
pub fn start(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
store: std::sync::Arc<ksp_store_lib::Store>,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle> {
|
||||||
|
let runtime = match current_runtime_handle() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let store_snapshot = store.runtime_snapshot();
|
||||||
|
if let std::result::Result::Err(error) = validate_store_network(&settings, store_snapshot.network()) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
return start_foundation(settings, runtime, std::option::Option::Some(store));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn begin_stopping(
|
||||||
|
lifecycle: &mut ksp_worker_api::WorkerLifecycle,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
if lifecycle.mark_stopping().is_err() {
|
||||||
|
return std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = snapshots.publish_state(lifecycle.state(), admission_queue_depth, in_flight_persistence) {
|
||||||
|
return std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn current_runtime_handle() -> ksp_core_lib::Result<tokio::runtime::Handle> {
|
||||||
|
return match tokio::runtime::Handle::try_current() {
|
||||||
|
std::result::Result::Ok(value) => std::result::Result::Ok(value),
|
||||||
|
std::result::Result::Err(_) => std::result::Result::Err(crate::runtime_error("start.runtime_unavailable")),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn drain_children(
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
children: &mut SourceTasks,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let mut fault = std::option::Option::None;
|
||||||
|
while let std::option::Option::Some(joined) = children.join_next().await {
|
||||||
|
let current = source_completion(joined, state, 0, 0, snapshots);
|
||||||
|
fault = merge_fault(fault, current);
|
||||||
|
}
|
||||||
|
return fault;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn drain_persistence(
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
admission: &crate::RawTransactionAdmission,
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let mut fault = std::option::Option::None;
|
||||||
|
while let std::option::Option::Some(joined) = persistence.join_next().await {
|
||||||
|
let current = persistence_completion(joined, lifecycle.state(), admission.queue_depth(), persistence.len(), snapshots);
|
||||||
|
fault = merge_fault(fault, current);
|
||||||
|
}
|
||||||
|
return fault;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn drain_admission_and_persistence(
|
||||||
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
admission: &mut crate::RawTransactionAdmission,
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
port: &std::option::Option<PersistencePort>,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let mut fault = std::option::Option::None;
|
||||||
|
admission.close();
|
||||||
|
loop {
|
||||||
|
while persistence.len() >= settings.persistence_concurrency() {
|
||||||
|
let joined = persistence.join_next().await;
|
||||||
|
let current = match joined {
|
||||||
|
std::option::Option::Some(value) => persistence_completion(value, lifecycle.state(), admission.queue_depth(), persistence.len(), snapshots),
|
||||||
|
std::option::Option::None => std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID),
|
||||||
|
};
|
||||||
|
fault = merge_fault(fault, current);
|
||||||
|
}
|
||||||
|
let received = admission.receive(settings.network()).await;
|
||||||
|
let backpressure_wait_observed = admission.take_backpressure_wait_observed();
|
||||||
|
match received {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(acquisition)) => {
|
||||||
|
let spawned = spawn_persistence(persistence, port, acquisition);
|
||||||
|
let published = snapshots.record_admission_success(lifecycle.state(), admission.queue_depth(), persistence.len(), backpressure_wait_observed);
|
||||||
|
if !spawned && fault.is_none() {
|
||||||
|
fault = std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = published
|
||||||
|
&& fault.is_none()
|
||||||
|
{
|
||||||
|
fault = std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Ok(std::option::Option::None) => break,
|
||||||
|
std::result::Result::Err(_) => {
|
||||||
|
let published = snapshots.record_admission_failure(lifecycle.state(), admission.queue_depth(), persistence.len(), backpressure_wait_observed);
|
||||||
|
match published {
|
||||||
|
std::result::Result::Ok(()) => {
|
||||||
|
if fault.is_none() {
|
||||||
|
fault = std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
if fault.is_none() {
|
||||||
|
fault = std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let persistence_fault = drain_persistence(lifecycle, admission, persistence, snapshots).await;
|
||||||
|
fault = merge_fault(fault, persistence_fault);
|
||||||
|
return fault;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn drain_owned_work(
|
||||||
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
children: &mut SourceTasks,
|
||||||
|
admission: &mut crate::RawTransactionAdmission,
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
port: &std::option::Option<PersistencePort>,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let drain = async {
|
||||||
|
let mut fault = drain_admission_and_persistence(settings, lifecycle, admission, persistence, port, snapshots).await;
|
||||||
|
let child_fault = drain_children(lifecycle.state(), children, snapshots).await;
|
||||||
|
fault = merge_fault(fault, child_fault);
|
||||||
|
return fault;
|
||||||
|
};
|
||||||
|
let timed = tokio::time::timeout(settings.shutdown_drain_timeout(), drain).await;
|
||||||
|
return match timed {
|
||||||
|
std::result::Result::Ok(fault) => fault,
|
||||||
|
std::result::Result::Err(_) => {
|
||||||
|
admission.close();
|
||||||
|
persistence.abort_all();
|
||||||
|
children.abort_all();
|
||||||
|
while persistence.join_next().await.is_some() {}
|
||||||
|
while children.join_next().await.is_some() {}
|
||||||
|
std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT)
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn finish_faulted(
|
||||||
|
lifecycle: &mut ksp_worker_api::WorkerLifecycle,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
code: ksp_core_lib::ErrorCode,
|
||||||
|
) {
|
||||||
|
if lifecycle.fault(code).is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if snapshots.publish_state(lifecycle.state(), 0, 0).is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn finish_stopped(lifecycle: &mut ksp_worker_api::WorkerLifecycle, snapshots: &mut crate::RawTransactionIngestSnapshotPublisher) {
|
||||||
|
if lifecycle.mark_stopped().is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if snapshots.publish_state(lifecycle.state(), 0, 0).is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn merge_fault(
|
||||||
|
first: std::option::Option<ksp_core_lib::ErrorCode>,
|
||||||
|
next: std::option::Option<ksp_core_lib::ErrorCode>,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
if first.is_some() {
|
||||||
|
return first;
|
||||||
|
}
|
||||||
|
return next;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persistence_completion(
|
||||||
|
joined: std::result::Result<ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>, tokio::task::JoinError>,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
return match joined {
|
||||||
|
std::result::Result::Ok(std::result::Result::Ok(outcome)) => {
|
||||||
|
match snapshots.record_persistence_success(state, admission_queue_depth, in_flight_persistence, outcome) {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::None,
|
||||||
|
std::result::Result::Err(error) => std::option::Option::Some(error.code()),
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Ok(std::result::Result::Err(error)) => {
|
||||||
|
let code = error.code();
|
||||||
|
let published = snapshots.record_persistence_fault(state, admission_queue_depth, in_flight_persistence, code);
|
||||||
|
match published {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::Some(code),
|
||||||
|
std::result::Result::Err(snapshot_error) => std::option::Option::Some(snapshot_error.code()),
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Err(_) => std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn source_completion(
|
||||||
|
joined: std::result::Result<ksp_core_lib::Result<()>, tokio::task::JoinError>,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
match joined {
|
||||||
|
std::result::Result::Ok(std::result::Result::Ok(())) => return std::option::Option::None,
|
||||||
|
std::result::Result::Ok(std::result::Result::Err(_)) | std::result::Result::Err(_) => {},
|
||||||
|
}
|
||||||
|
let published = snapshots.record_source_failure(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
return match published {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED),
|
||||||
|
std::result::Result::Err(error) => std::option::Option::Some(error.code()),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn run_supervisor<Spawner>(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
mut lifecycle: ksp_worker_api::WorkerLifecycle,
|
||||||
|
port: std::option::Option<PersistencePort>,
|
||||||
|
mut stop_receiver: tokio::sync::watch::Receiver<bool>,
|
||||||
|
mut snapshots: crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
source_spawner: Spawner,
|
||||||
|
) where
|
||||||
|
Spawner:
|
||||||
|
FnOnce(&mut SourceTasks, tokio::sync::watch::Receiver<bool>, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) + std::marker::Send + 'static,
|
||||||
|
{
|
||||||
|
if *stop_receiver.borrow() {
|
||||||
|
let stopping_fault = begin_stopping(&mut lifecycle, &mut snapshots, 0, 0);
|
||||||
|
match stopping_fault {
|
||||||
|
std::option::Option::Some(code) => finish_faulted(&mut lifecycle, &mut snapshots, code),
|
||||||
|
std::option::Option::None => finish_stopped(&mut lifecycle, &mut snapshots),
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if lifecycle.mark_running().is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = snapshots.publish_state(lifecycle.state(), 0, 0) {
|
||||||
|
finish_faulted(&mut lifecycle, &mut snapshots, error.code());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let mut children = SourceTasks::new();
|
||||||
|
let mut persistence = PersistenceTasks::new();
|
||||||
|
let (source_stop_sender, source_stop_receiver) = tokio::sync::watch::channel(false);
|
||||||
|
let (mut admission, admission_sender) = crate::RawTransactionAdmission::new(settings.admission_queue_capacity());
|
||||||
|
source_spawner(&mut children, source_stop_receiver, admission_sender);
|
||||||
|
let mut fault = supervise_until_stop(
|
||||||
|
&settings,
|
||||||
|
&lifecycle,
|
||||||
|
&source_stop_sender,
|
||||||
|
&mut stop_receiver,
|
||||||
|
&mut children,
|
||||||
|
&mut admission,
|
||||||
|
&mut persistence,
|
||||||
|
&port,
|
||||||
|
&mut snapshots,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
let stopping_fault = begin_stopping(&mut lifecycle, &mut snapshots, admission.queue_depth(), persistence.len());
|
||||||
|
fault = merge_fault(fault, stopping_fault);
|
||||||
|
let drain_fault = drain_owned_work(&settings, &lifecycle, &mut children, &mut admission, &mut persistence, &port, &mut snapshots).await;
|
||||||
|
match drain_fault {
|
||||||
|
std::option::Option::Some(code) if code == crate::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT => {
|
||||||
|
fault = std::option::Option::Some(code);
|
||||||
|
},
|
||||||
|
std::option::Option::Some(code) => {
|
||||||
|
fault = merge_fault(fault, std::option::Option::Some(code));
|
||||||
|
},
|
||||||
|
std::option::Option::None => {},
|
||||||
|
}
|
||||||
|
match fault {
|
||||||
|
std::option::Option::Some(code) => finish_faulted(&mut lifecycle, &mut snapshots, code),
|
||||||
|
std::option::Option::None => finish_stopped(&mut lifecycle, &mut snapshots),
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn spawn_persistence(
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
port: &std::option::Option<PersistencePort>,
|
||||||
|
acquisition: ksp_raw_transaction_lib::RawTransactionAcquisition,
|
||||||
|
) -> bool {
|
||||||
|
let port = match port {
|
||||||
|
std::option::Option::Some(value) => std::sync::Arc::clone(value),
|
||||||
|
std::option::Option::None => return false,
|
||||||
|
};
|
||||||
|
let _abort_handle = persistence.spawn(async move {
|
||||||
|
return crate::persist_raw_transaction_ingest_acquisition(port.as_ref(), acquisition).await;
|
||||||
|
});
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn start_foundation(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
runtime: tokio::runtime::Handle,
|
||||||
|
store_guard: std::option::Option<std::sync::Arc<ksp_store_lib::Store>>,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle> {
|
||||||
|
return start_foundation_with_source_spawner(settings, runtime, store_guard, |_children, _stop_receiver, _admission_sender| {});
|
||||||
|
}
|
||||||
|
|
||||||
|
fn start_foundation_with_port_and_source_spawner<Spawner>(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
runtime: tokio::runtime::Handle,
|
||||||
|
port: std::option::Option<PersistencePort>,
|
||||||
|
source_spawner: Spawner,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle>
|
||||||
|
where
|
||||||
|
Spawner:
|
||||||
|
FnOnce(&mut SourceTasks, tokio::sync::watch::Receiver<bool>, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) + std::marker::Send + 'static,
|
||||||
|
{
|
||||||
|
let kind = match ksp_worker_api::WorkerKindCode::new(crate::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("start.worker_kind_invalid")),
|
||||||
|
};
|
||||||
|
let mut lifecycle = ksp_worker_api::WorkerLifecycle::new(settings.worker_id().clone(), kind);
|
||||||
|
if lifecycle.start().is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("start.lifecycle_invalid"));
|
||||||
|
}
|
||||||
|
let stop_token = ksp_worker_api::WorkerStopToken::new();
|
||||||
|
let (stop_sender, stop_receiver) = tokio::sync::watch::channel(false);
|
||||||
|
let (snapshots, snapshot_source) = crate::RawTransactionIngestSnapshotPublisher::new(&settings, &lifecycle);
|
||||||
|
let handle = crate::RawTransactionIngestHandle { snapshots: snapshot_source, stop_sender, stop_token };
|
||||||
|
std::mem::drop(runtime.spawn(run_supervisor(settings, lifecycle, port, stop_receiver, snapshots, source_spawner)));
|
||||||
|
return std::result::Result::Ok(handle);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn start_foundation_with_source_spawner<Spawner>(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
runtime: tokio::runtime::Handle,
|
||||||
|
store_guard: std::option::Option<std::sync::Arc<ksp_store_lib::Store>>,
|
||||||
|
source_spawner: Spawner,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle>
|
||||||
|
where
|
||||||
|
Spawner:
|
||||||
|
FnOnce(&mut SourceTasks, tokio::sync::watch::Receiver<bool>, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) + std::marker::Send + 'static,
|
||||||
|
{
|
||||||
|
let port = match store_guard {
|
||||||
|
std::option::Option::Some(store) => {
|
||||||
|
let port: PersistencePort = store;
|
||||||
|
std::option::Option::Some(port)
|
||||||
|
},
|
||||||
|
std::option::Option::None => std::option::Option::None,
|
||||||
|
};
|
||||||
|
return start_foundation_with_port_and_source_spawner(settings, runtime, port, source_spawner);
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn supervise_until_stop(
|
||||||
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
source_stop_sender: &tokio::sync::watch::Sender<bool>,
|
||||||
|
stop_receiver: &mut tokio::sync::watch::Receiver<bool>,
|
||||||
|
children: &mut SourceTasks,
|
||||||
|
admission: &mut crate::RawTransactionAdmission,
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
port: &std::option::Option<PersistencePort>,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let mut admission_open = true;
|
||||||
|
loop {
|
||||||
|
if *stop_receiver.borrow() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if children.is_empty() && !admission_open && persistence.is_empty() {
|
||||||
|
let changed = stop_receiver.changed().await;
|
||||||
|
if changed.is_err() || *stop_receiver.borrow() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
tokio::select! {
|
||||||
|
biased;
|
||||||
|
changed = stop_receiver.changed() => {
|
||||||
|
if changed.is_err() || *stop_receiver.borrow() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
joined = children.join_next(), if !children.is_empty() => {
|
||||||
|
let source_fault = match joined {
|
||||||
|
std::option::Option::Some(value) => source_completion(value, lifecycle.state(), admission.queue_depth(), persistence.len(), snapshots),
|
||||||
|
std::option::Option::None => std::option::Option::None,
|
||||||
|
};
|
||||||
|
if source_fault.is_some() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return source_fault;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
joined = persistence.join_next(), if !persistence.is_empty() => {
|
||||||
|
match joined {
|
||||||
|
std::option::Option::Some(value) => {
|
||||||
|
let fault = persistence_completion(value, lifecycle.state(), admission.queue_depth(), persistence.len(), snapshots);
|
||||||
|
if fault.is_some() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return fault;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::option::Option::None => {},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
received = admission.receive(settings.network()), if admission_open && persistence.len() < settings.persistence_concurrency() => {
|
||||||
|
match received {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(acquisition)) => {
|
||||||
|
let backpressure_wait_observed = admission.take_backpressure_wait_observed();
|
||||||
|
let spawned = spawn_persistence(persistence, port, acquisition);
|
||||||
|
let published = snapshots.record_admission_success(
|
||||||
|
lifecycle.state(),
|
||||||
|
admission.queue_depth(),
|
||||||
|
persistence.len(),
|
||||||
|
backpressure_wait_observed,
|
||||||
|
);
|
||||||
|
if !spawned {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = published {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Ok(std::option::Option::None) => {
|
||||||
|
let _backpressure_wait_observed = admission.take_backpressure_wait_observed();
|
||||||
|
admission_open = false;
|
||||||
|
},
|
||||||
|
std::result::Result::Err(_) => {
|
||||||
|
let backpressure_wait_observed = admission.take_backpressure_wait_observed();
|
||||||
|
let published = snapshots.record_admission_failure(
|
||||||
|
lifecycle.state(),
|
||||||
|
admission.queue_depth(),
|
||||||
|
persistence.len(),
|
||||||
|
backpressure_wait_observed,
|
||||||
|
);
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return match published {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID),
|
||||||
|
std::result::Result::Err(error) => std::option::Option::Some(error.code()),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_store_network(settings: &crate::RawTransactionIngestSettings, store_network: &ksp_store_lib::RawNetworkId) -> ksp_core_lib::Result<()> {
|
||||||
|
if settings.network() != store_network {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("start.store_network_mismatch"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/runtime.rs"]
|
||||||
|
mod tests;
|
||||||
131
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
Normal file
131
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
Normal file
@@ -0,0 +1,131 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
|
||||||
|
// version: 4
|
||||||
|
|
||||||
|
/// Default bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub const DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY: usize = 256;
|
||||||
|
/// Default number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub const DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY: usize = 8;
|
||||||
|
/// Default cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub const DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
|
||||||
|
/// Maximum bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub const MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY: usize = 65_536;
|
||||||
|
/// Maximum number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub const MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY: usize = 64;
|
||||||
|
/// Maximum cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub const MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
|
||||||
|
/// Minimum bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub const MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY: usize = 1;
|
||||||
|
/// Minimum number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub const MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY: usize = 1;
|
||||||
|
/// Minimum cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub const MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(100);
|
||||||
|
|
||||||
|
/// Validated source-neutral runtime settings for one continuous RAW transaction ingest Worker.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct RawTransactionIngestSettings {
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
worker_id: ksp_worker_api::WorkerId,
|
||||||
|
admission_queue_capacity: usize,
|
||||||
|
persistence_concurrency: usize,
|
||||||
|
shutdown_drain_timeout: std::time::Duration,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestSettings {
|
||||||
|
/// Creates settings from validated network/Worker identities and explicit bounded runtime limits.
|
||||||
|
pub fn new(
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
worker_id: ksp_worker_api::WorkerId,
|
||||||
|
admission_queue_capacity: usize,
|
||||||
|
persistence_concurrency: usize,
|
||||||
|
shutdown_drain_timeout: std::time::Duration,
|
||||||
|
) -> ksp_core_lib::Result<Self> {
|
||||||
|
if let std::result::Result::Err(error) = validate_inclusive_usize(
|
||||||
|
admission_queue_capacity,
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
"admission_queue_capacity",
|
||||||
|
) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = validate_inclusive_usize(
|
||||||
|
persistence_concurrency,
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
"persistence_concurrency",
|
||||||
|
) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
if !(crate::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT..=crate::MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT)
|
||||||
|
.contains(&shutdown_drain_timeout)
|
||||||
|
{
|
||||||
|
return std::result::Result::Err(crate::settings_error("shutdown_drain_timeout"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(Self { network, worker_id, admission_queue_capacity, persistence_concurrency, shutdown_drain_timeout });
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates settings using the stable V1 runtime defaults for one validated network and Worker identity.
|
||||||
|
#[must_use]
|
||||||
|
pub fn with_defaults(network: ksp_store_lib::RawNetworkId, worker_id: ksp_worker_api::WorkerId) -> Self {
|
||||||
|
return Self {
|
||||||
|
network,
|
||||||
|
worker_id,
|
||||||
|
admission_queue_capacity: crate::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
persistence_concurrency: crate::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
shutdown_drain_timeout: crate::DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the Store-compatible logical network binding.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn network(&self) -> &ksp_store_lib::RawNetworkId {
|
||||||
|
return &self.network;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the validated logical identity of this Worker instance.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn worker_id(&self) -> &ksp_worker_api::WorkerId {
|
||||||
|
return &self.worker_id;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the bounded central admission queue capacity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn admission_queue_capacity(&self) -> usize {
|
||||||
|
return self.admission_queue_capacity;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the bounded maximum number of concurrent Store persistence operations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn persistence_concurrency(&self) -> usize {
|
||||||
|
return self.persistence_concurrency;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the bounded cooperative shutdown drain deadline.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn shutdown_drain_timeout(&self) -> std::time::Duration {
|
||||||
|
return self.shutdown_drain_timeout;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionIngestSettings {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionIngestSettings")
|
||||||
|
.field("network", &self.network)
|
||||||
|
.field("worker_id", &self.worker_id)
|
||||||
|
.field("admission_queue_capacity", &self.admission_queue_capacity)
|
||||||
|
.field("persistence_concurrency", &self.persistence_concurrency)
|
||||||
|
.field("shutdown_drain_timeout", &self.shutdown_drain_timeout)
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_inclusive_usize(value: usize, minimum: usize, maximum: usize, field: &'static str) -> ksp_core_lib::Result<()> {
|
||||||
|
if !(minimum..=maximum).contains(&value) {
|
||||||
|
return std::result::Result::Err(crate::settings_error(field));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/settings.rs"]
|
||||||
|
mod tests;
|
||||||
508
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
Normal file
508
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
Normal file
@@ -0,0 +1,508 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
/// Runtime-neutral boxed future resolving to one newer concrete RAW transaction ingest Worker snapshot.
|
||||||
|
pub type RawTransactionIngestSnapshotFuture<'a> =
|
||||||
|
std::pin::Pin<std::boxed::Box<dyn std::future::Future<Output = crate::RawTransactionIngestSnapshot> + std::marker::Send + 'a>>;
|
||||||
|
|
||||||
|
/// Complete safe latest-value snapshot of one continuous RAW transaction ingest Worker.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct RawTransactionIngestSnapshot {
|
||||||
|
worker: ksp_worker_api::WorkerSnapshot,
|
||||||
|
admission_queue_capacity: usize,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
persistence_concurrency: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
admitted_total: u64,
|
||||||
|
canonicalized_total: u64,
|
||||||
|
persisted_total: u64,
|
||||||
|
entity_inserted_total: u64,
|
||||||
|
entity_already_present_total: u64,
|
||||||
|
entity_skipped_purged_total: u64,
|
||||||
|
observation_inserted_total: u64,
|
||||||
|
observation_already_present_total: u64,
|
||||||
|
content_conflict_total: u64,
|
||||||
|
store_failure_total: u64,
|
||||||
|
source_failure_total: u64,
|
||||||
|
backpressure_wait_total: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestSnapshot {
|
||||||
|
/// Returns the common Worker projection carried by this concrete snapshot.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn worker_snapshot(&self) -> &ksp_worker_api::WorkerSnapshot {
|
||||||
|
return &self.worker;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the configured bounded admission queue capacity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn admission_queue_capacity(&self) -> usize {
|
||||||
|
return self.admission_queue_capacity;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest observed number of queued ingress entries awaiting supervisor admission.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn admission_queue_depth(&self) -> usize {
|
||||||
|
return self.admission_queue_depth;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the configured maximum number of concurrent Store persistence operations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn persistence_concurrency(&self) -> usize {
|
||||||
|
return self.persistence_concurrency;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest observed number of in-flight Store persistence operations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn in_flight_persistence(&self) -> usize {
|
||||||
|
return self.in_flight_persistence;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of ingress entries removed from the bounded admission queue.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn admitted_total(&self) -> u64 {
|
||||||
|
return self.admitted_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of admitted ingress entries successfully canonicalized through the Common RAW contract.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn canonicalized_total(&self) -> u64 {
|
||||||
|
return self.canonicalized_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of successful atomic Store persistence outcomes.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn persisted_total(&self) -> u64 {
|
||||||
|
return self.persisted_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of newly inserted canonical RAW transaction entities.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn entity_inserted_total(&self) -> u64 {
|
||||||
|
return self.entity_inserted_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of identical canonical RAW transaction entities already durable.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn entity_already_present_total(&self) -> u64 {
|
||||||
|
return self.entity_already_present_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of durable purge tombstones respected by normal persistence.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn entity_skipped_purged_total(&self) -> u64 {
|
||||||
|
return self.entity_skipped_purged_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of newly inserted deterministic acquisition observations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn observation_inserted_total(&self) -> u64 {
|
||||||
|
return self.observation_inserted_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of deterministic acquisition observations already durable.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn observation_already_present_total(&self) -> u64 {
|
||||||
|
return self.observation_already_present_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of durable Store content conflicts observed by this run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn content_conflict_total(&self) -> u64 {
|
||||||
|
return self.content_conflict_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of non-conflict Store persistence failures observed by this run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn store_failure_total(&self) -> u64 {
|
||||||
|
return self.store_failure_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of source-task failures observed by this run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_failure_total(&self) -> u64 {
|
||||||
|
return self.source_failure_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of supervisor dequeues that observed the bounded admission queue at full capacity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn backpressure_wait_total(&self) -> u64 {
|
||||||
|
return self.backpressure_wait_total;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionIngestSnapshot {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionIngestSnapshot")
|
||||||
|
.field("worker", &self.worker)
|
||||||
|
.field("admission_queue_capacity", &self.admission_queue_capacity)
|
||||||
|
.field("admission_queue_depth", &self.admission_queue_depth)
|
||||||
|
.field("persistence_concurrency", &self.persistence_concurrency)
|
||||||
|
.field("in_flight_persistence", &self.in_flight_persistence)
|
||||||
|
.field("admitted_total", &self.admitted_total)
|
||||||
|
.field("canonicalized_total", &self.canonicalized_total)
|
||||||
|
.field("persisted_total", &self.persisted_total)
|
||||||
|
.field("entity_inserted_total", &self.entity_inserted_total)
|
||||||
|
.field("entity_already_present_total", &self.entity_already_present_total)
|
||||||
|
.field("entity_skipped_purged_total", &self.entity_skipped_purged_total)
|
||||||
|
.field("observation_inserted_total", &self.observation_inserted_total)
|
||||||
|
.field("observation_already_present_total", &self.observation_already_present_total)
|
||||||
|
.field("content_conflict_total", &self.content_conflict_total)
|
||||||
|
.field("store_failure_total", &self.store_failure_total)
|
||||||
|
.field("source_failure_total", &self.source_failure_total)
|
||||||
|
.field("backpressure_wait_total", &self.backpressure_wait_total)
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cloneable latest-value source exposing concrete and common Worker snapshots from one shared watch state.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct RawTransactionIngestSnapshotSource {
|
||||||
|
receiver: tokio::sync::watch::Receiver<crate::RawTransactionIngestSnapshot>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestSnapshotSource {
|
||||||
|
/// Returns the complete current concrete snapshot without replaying intermediate updates.
|
||||||
|
#[must_use]
|
||||||
|
pub fn current(&self) -> crate::RawTransactionIngestSnapshot {
|
||||||
|
return self.receiver.borrow().clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reports whether the private publisher has closed after the runtime task returned.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn is_closed(&self) -> bool {
|
||||||
|
return self.receiver.has_changed().is_err();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Waits for one concrete snapshot newer than `observed`, coalescing intermediate updates to the latest value.
|
||||||
|
#[must_use]
|
||||||
|
pub fn wait_for_change(&self, observed: ksp_worker_api::WorkerSnapshotSequence) -> crate::RawTransactionIngestSnapshotFuture<'_> {
|
||||||
|
let mut receiver = self.receiver.clone();
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
loop {
|
||||||
|
let current = receiver.borrow_and_update().clone();
|
||||||
|
if current.worker_snapshot().sequence().is_after(observed) {
|
||||||
|
return current;
|
||||||
|
}
|
||||||
|
let changed = receiver.changed().await;
|
||||||
|
if changed.is_err() {
|
||||||
|
return receiver.borrow_and_update().clone();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ksp_worker_api::WorkerSnapshotSource for crate::RawTransactionIngestSnapshotSource {
|
||||||
|
fn current(&self) -> ksp_worker_api::WorkerSnapshot {
|
||||||
|
return self.receiver.borrow().worker_snapshot().clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn wait_for_change(&self, observed: ksp_worker_api::WorkerSnapshotSequence) -> ksp_worker_api::WorkerSnapshotFuture<'_> {
|
||||||
|
let mut receiver = self.receiver.clone();
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
loop {
|
||||||
|
let current = receiver.borrow_and_update().worker_snapshot().clone();
|
||||||
|
if current.sequence().is_after(observed) {
|
||||||
|
return current;
|
||||||
|
}
|
||||||
|
let changed = receiver.changed().await;
|
||||||
|
if changed.is_err() {
|
||||||
|
return receiver.borrow_and_update().worker_snapshot().clone();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionIngestSnapshotSource {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
let current = self.receiver.borrow();
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionIngestSnapshotSource")
|
||||||
|
.field("sequence", ¤t.worker_snapshot().sequence())
|
||||||
|
.field("state", ¤t.worker_snapshot().state())
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Private latest-value publisher and checked counter owner shared by the Worker supervisor.
|
||||||
|
pub(crate) struct RawTransactionIngestSnapshotPublisher {
|
||||||
|
sender: tokio::sync::watch::Sender<crate::RawTransactionIngestSnapshot>,
|
||||||
|
snapshot: crate::RawTransactionIngestSnapshot,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestSnapshotPublisher {
|
||||||
|
/// Creates the initial `Starting` snapshot stream for one validated Worker run.
|
||||||
|
pub(crate) fn new(
|
||||||
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
) -> (crate::RawTransactionIngestSnapshotPublisher, crate::RawTransactionIngestSnapshotSource) {
|
||||||
|
let worker = ksp_worker_api::WorkerSnapshot::new(
|
||||||
|
lifecycle.id().clone(),
|
||||||
|
lifecycle.kind().clone(),
|
||||||
|
ksp_worker_api::WorkerSnapshotSequence::initial(),
|
||||||
|
lifecycle.state(),
|
||||||
|
ksp_worker_api::WorkerHealth::Unknown,
|
||||||
|
ksp_worker_api::WorkerActivity::Unknown,
|
||||||
|
);
|
||||||
|
let snapshot = crate::RawTransactionIngestSnapshot {
|
||||||
|
worker,
|
||||||
|
admission_queue_capacity: settings.admission_queue_capacity(),
|
||||||
|
admission_queue_depth: 0,
|
||||||
|
persistence_concurrency: settings.persistence_concurrency(),
|
||||||
|
in_flight_persistence: 0,
|
||||||
|
admitted_total: 0,
|
||||||
|
canonicalized_total: 0,
|
||||||
|
persisted_total: 0,
|
||||||
|
entity_inserted_total: 0,
|
||||||
|
entity_already_present_total: 0,
|
||||||
|
entity_skipped_purged_total: 0,
|
||||||
|
observation_inserted_total: 0,
|
||||||
|
observation_already_present_total: 0,
|
||||||
|
content_conflict_total: 0,
|
||||||
|
store_failure_total: 0,
|
||||||
|
source_failure_total: 0,
|
||||||
|
backpressure_wait_total: 0,
|
||||||
|
};
|
||||||
|
let (sender, receiver) = tokio::sync::watch::channel(snapshot.clone());
|
||||||
|
return (Self { sender, snapshot }, crate::RawTransactionIngestSnapshotSource { receiver });
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Forces one terminal latest value without advancing the sequence, used only when sequence publication itself is exhausted or invalid.
|
||||||
|
pub(crate) fn force_terminal(&mut self, state: ksp_worker_api::WorkerState) {
|
||||||
|
let worker = ksp_worker_api::WorkerSnapshot::new(
|
||||||
|
self.snapshot.worker.id().clone(),
|
||||||
|
self.snapshot.worker.kind().clone(),
|
||||||
|
self.snapshot.worker.sequence(),
|
||||||
|
state,
|
||||||
|
health_for_state(state, self.snapshot.worker.health()),
|
||||||
|
ksp_worker_api::WorkerActivity::Idle,
|
||||||
|
);
|
||||||
|
self.snapshot.worker = worker;
|
||||||
|
self.snapshot.admission_queue_depth = 0;
|
||||||
|
self.snapshot.in_flight_persistence = 0;
|
||||||
|
self.sender.send_replace(self.snapshot.clone());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publishes one lifecycle/depth refresh with a strictly newer common sequence.
|
||||||
|
pub(crate) fn publish_state(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one dequeued ingress that failed canonicalization and publishes the resulting latest value.
|
||||||
|
pub(crate) fn record_admission_failure(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
backpressure_wait_observed: bool,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
let admitted_total = match checked_counter(self.snapshot.admitted_total, "admitted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let backpressure_wait_total =
|
||||||
|
match checked_optional_counter(self.snapshot.backpressure_wait_total, backpressure_wait_observed, "backpressure_wait_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.admitted_total = admitted_total;
|
||||||
|
self.snapshot.backpressure_wait_total = backpressure_wait_total;
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one successfully canonicalized admitted ingress and publishes the resulting latest value.
|
||||||
|
pub(crate) fn record_admission_success(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
backpressure_wait_observed: bool,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
let admitted_total = match checked_counter(self.snapshot.admitted_total, "admitted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let backpressure_wait_total =
|
||||||
|
match checked_optional_counter(self.snapshot.backpressure_wait_total, backpressure_wait_observed, "backpressure_wait_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let canonicalized_total = match checked_counter(self.snapshot.canonicalized_total, "canonicalized_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.admitted_total = admitted_total;
|
||||||
|
self.snapshot.backpressure_wait_total = backpressure_wait_total;
|
||||||
|
self.snapshot.canonicalized_total = canonicalized_total;
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one failed private source task without retaining provider-specific error material.
|
||||||
|
pub(crate) fn record_source_failure(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
let source_failure_total = match checked_counter(self.snapshot.source_failure_total, "source_failure_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.source_failure_total = source_failure_total;
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one successful Store persistence outcome and publishes the resulting latest value.
|
||||||
|
pub(crate) fn record_persistence_success(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
outcome: crate::RawTransactionIngestPersistenceOutcome,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
let persisted_total = match checked_counter(self.snapshot.persisted_total, "persisted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let mut entity_inserted_total = self.snapshot.entity_inserted_total;
|
||||||
|
let mut entity_already_present_total = self.snapshot.entity_already_present_total;
|
||||||
|
let mut entity_skipped_purged_total = self.snapshot.entity_skipped_purged_total;
|
||||||
|
let mut observation_inserted_total = self.snapshot.observation_inserted_total;
|
||||||
|
let mut observation_already_present_total = self.snapshot.observation_already_present_total;
|
||||||
|
match outcome.entity() {
|
||||||
|
crate::RawTransactionIngestEntityPersistence::Inserted => {
|
||||||
|
entity_inserted_total = match checked_counter(entity_inserted_total, "entity_inserted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::RawTransactionIngestEntityPersistence::AlreadyPresent => {
|
||||||
|
entity_already_present_total = match checked_counter(entity_already_present_total, "entity_already_present_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::RawTransactionIngestEntityPersistence::SkippedPurged => {
|
||||||
|
entity_skipped_purged_total = match checked_counter(entity_skipped_purged_total, "entity_skipped_purged_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
}
|
||||||
|
match outcome.observation() {
|
||||||
|
crate::RawTransactionIngestObservationPersistence::Inserted => {
|
||||||
|
observation_inserted_total = match checked_counter(observation_inserted_total, "observation_inserted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::RawTransactionIngestObservationPersistence::AlreadyPresent => {
|
||||||
|
observation_already_present_total = match checked_counter(observation_already_present_total, "observation_already_present_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::RawTransactionIngestObservationPersistence::NotRecorded => {},
|
||||||
|
}
|
||||||
|
self.snapshot.persisted_total = persisted_total;
|
||||||
|
self.snapshot.entity_inserted_total = entity_inserted_total;
|
||||||
|
self.snapshot.entity_already_present_total = entity_already_present_total;
|
||||||
|
self.snapshot.entity_skipped_purged_total = entity_skipped_purged_total;
|
||||||
|
self.snapshot.observation_inserted_total = observation_inserted_total;
|
||||||
|
self.snapshot.observation_already_present_total = observation_already_present_total;
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one classified persistence fault counter and publishes the resulting latest value.
|
||||||
|
pub(crate) fn record_persistence_fault(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
code: ksp_core_lib::ErrorCode,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
if code == crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT {
|
||||||
|
let next = match checked_counter(self.snapshot.content_conflict_total, "content_conflict_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.content_conflict_total = next;
|
||||||
|
} else if code == crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED {
|
||||||
|
let next = match checked_counter(self.snapshot.store_failure_total, "store_failure_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.store_failure_total = next;
|
||||||
|
}
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn publish(&mut self, state: ksp_worker_api::WorkerState, admission_queue_depth: usize, in_flight_persistence: usize) -> ksp_core_lib::Result<()> {
|
||||||
|
let sequence = match self.snapshot.worker.sequence().next() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::counter_exhausted_error("snapshot_sequence")),
|
||||||
|
};
|
||||||
|
let worker = ksp_worker_api::WorkerSnapshot::new(
|
||||||
|
self.snapshot.worker.id().clone(),
|
||||||
|
self.snapshot.worker.kind().clone(),
|
||||||
|
sequence,
|
||||||
|
state,
|
||||||
|
health_for_state(state, self.snapshot.worker.health()),
|
||||||
|
activity_for_state(state, admission_queue_depth, in_flight_persistence),
|
||||||
|
);
|
||||||
|
self.snapshot.worker = worker;
|
||||||
|
self.snapshot.admission_queue_depth = admission_queue_depth;
|
||||||
|
self.snapshot.in_flight_persistence = in_flight_persistence;
|
||||||
|
self.sender.send_replace(self.snapshot.clone());
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn activity_for_state(state: ksp_worker_api::WorkerState, admission_queue_depth: usize, in_flight_persistence: usize) -> ksp_worker_api::WorkerActivity {
|
||||||
|
if admission_queue_depth > 0 || in_flight_persistence > 0 {
|
||||||
|
return ksp_worker_api::WorkerActivity::Active;
|
||||||
|
}
|
||||||
|
return match state {
|
||||||
|
ksp_worker_api::WorkerState::Running
|
||||||
|
| ksp_worker_api::WorkerState::Stopping
|
||||||
|
| ksp_worker_api::WorkerState::Stopped
|
||||||
|
| ksp_worker_api::WorkerState::Faulted(_) => ksp_worker_api::WorkerActivity::Idle,
|
||||||
|
_ => ksp_worker_api::WorkerActivity::Unknown,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn checked_counter(current: u64, field: &'static str) -> ksp_core_lib::Result<u64> {
|
||||||
|
return match current.checked_add(1) {
|
||||||
|
std::option::Option::Some(value) => std::result::Result::Ok(value),
|
||||||
|
std::option::Option::None => std::result::Result::Err(crate::counter_exhausted_error(field)),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn checked_optional_counter(current: u64, increment: bool, field: &'static str) -> ksp_core_lib::Result<u64> {
|
||||||
|
if !increment {
|
||||||
|
return std::result::Result::Ok(current);
|
||||||
|
}
|
||||||
|
return checked_counter(current, field);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn health_for_state(state: ksp_worker_api::WorkerState, previous: ksp_worker_api::WorkerHealth) -> ksp_worker_api::WorkerHealth {
|
||||||
|
return match state {
|
||||||
|
ksp_worker_api::WorkerState::Running => ksp_worker_api::WorkerHealth::Healthy,
|
||||||
|
ksp_worker_api::WorkerState::Stopping | ksp_worker_api::WorkerState::Stopped => previous,
|
||||||
|
ksp_worker_api::WorkerState::Faulted(_) => ksp_worker_api::WorkerHealth::Unhealthy,
|
||||||
|
_ => ksp_worker_api::WorkerHealth::Unknown,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/snapshot.rs"]
|
||||||
|
mod tests;
|
||||||
@@ -0,0 +1,147 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
// version: 8
|
||||||
|
|
||||||
|
//! Dependency firewall canaries for the RAW transaction ingest Worker foundation.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_manifest_dependency_surface_is_exact() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let dependencies = dependency_section(manifest);
|
||||||
|
let names = manifest_dependency_names(dependencies);
|
||||||
|
assert_eq!(names, vec!["ksp-core-lib", "ksp-logging-lib", "ksp-raw-transaction-lib", "ksp-store-lib", "ksp-worker-api", "sha2", "tokio",],);
|
||||||
|
for required in [
|
||||||
|
"ksp-core-lib = { path = \"../ksp-core-lib\" }",
|
||||||
|
"ksp-logging-lib = { path = \"../ksp-logging-lib\" }",
|
||||||
|
"ksp-raw-transaction-lib = { path = \"../ksp-raw-transaction-lib\" }",
|
||||||
|
"ksp-store-lib = { path = \"../ksp-store-lib\", default-features = false }",
|
||||||
|
"ksp-worker-api = { path = \"../ksp-worker-api\" }",
|
||||||
|
"sha2 = { workspace = true }",
|
||||||
|
"tokio = { workspace = true, features = [\"macros\", \"rt\", \"sync\", \"time\"] }",
|
||||||
|
] {
|
||||||
|
assert!(dependencies.contains(required), "required Worker dependency missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_manifest_keeps_forbidden_layers_and_live_sources_out() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let dependencies = dependency_section(manifest);
|
||||||
|
for forbidden in [
|
||||||
|
"futures-util",
|
||||||
|
"ksp-config-lib",
|
||||||
|
"ksp-interface-lib",
|
||||||
|
"ksp-job-api",
|
||||||
|
"ksp-job-backfill-lib",
|
||||||
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-program-api",
|
||||||
|
"ksp-store-api",
|
||||||
|
"ksp-store-postgres-lib",
|
||||||
|
"ksp-wallet-lib",
|
||||||
|
"reqwest",
|
||||||
|
"solana-",
|
||||||
|
"tokio-tungstenite",
|
||||||
|
"tonic",
|
||||||
|
"yellowstone-grpc-proto",
|
||||||
|
] {
|
||||||
|
assert!(!dependencies.contains(forbidden), "forbidden Worker dependency present: {forbidden}");
|
||||||
|
}
|
||||||
|
assert!(!manifest.contains("[dev-dependencies]"), "foundation requires no test-only dependency");
|
||||||
|
assert!(!manifest.contains("[build-dependencies]"), "foundation requires no build dependency");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_009_source_surface_hardens_shutdown_and_faults_without_backend_or_live_source() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let admission = include_str!("../src/admission.rs");
|
||||||
|
let persistence = include_str!("../src/persistence.rs");
|
||||||
|
let snapshot = include_str!("../src/snapshot.rs");
|
||||||
|
for required in [
|
||||||
|
"tokio::sync::mpsc::channel",
|
||||||
|
"canonicalize_raw_transaction",
|
||||||
|
"assemble_raw_transaction_acquisition",
|
||||||
|
"RawTransactionIngestPersistencePort",
|
||||||
|
"RawTransactionAcquisitionMode::Normal",
|
||||||
|
"RawTransactionIngestSnapshot",
|
||||||
|
"RawTransactionIngestSnapshotSource",
|
||||||
|
"WorkerSnapshotSource",
|
||||||
|
"tokio::sync::watch::channel",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED",
|
||||||
|
"record_source_failure",
|
||||||
|
"tokio::time::timeout",
|
||||||
|
"abort_all",
|
||||||
|
"backpressure_wait_total",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
root.contains(required)
|
||||||
|
|| runtime.contains(required)
|
||||||
|
|| admission.contains(required)
|
||||||
|
|| persistence.contains(required)
|
||||||
|
|| snapshot.contains(required),
|
||||||
|
"required pre.009 hardening contract missing: {required}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"pub use self::admission::RawTransactionAdmission",
|
||||||
|
"pub use self::admission::RawTransactionIngress",
|
||||||
|
"pub use self::persistence::RawTransactionIngestPersistencePort",
|
||||||
|
"unbounded_channel",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ksp_onchain_transport_lib::",
|
||||||
|
"ForceRehydrate",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!root.contains(forbidden)
|
||||||
|
&& !runtime.contains(forbidden)
|
||||||
|
&& !admission.contains(forbidden)
|
||||||
|
&& !persistence.contains(forbidden)
|
||||||
|
&& !snapshot.contains(forbidden),
|
||||||
|
"pre.009 crossed a forbidden runtime boundary: {forbidden}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
assert_eq!(runtime.matches("tokio::sync::watch::channel").count(), 2, "runtime retains only the external-control and private-source stop watches");
|
||||||
|
assert_eq!(snapshot.matches("tokio::sync::watch::channel").count(), 1, "exactly one shared concrete snapshot watch is allowed");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn dependency_section(manifest: &str) -> &str {
|
||||||
|
let after_header = match manifest.split_once("[dependencies]") {
|
||||||
|
std::option::Option::Some((_, section)) => section,
|
||||||
|
std::option::Option::None => {
|
||||||
|
return "";
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let end = match after_header.find("\n[") {
|
||||||
|
std::option::Option::Some(index) => index,
|
||||||
|
std::option::Option::None => after_header.len(),
|
||||||
|
};
|
||||||
|
return &after_header[..end];
|
||||||
|
}
|
||||||
|
|
||||||
|
fn manifest_dependency_names(section: &str) -> std::vec::Vec<&str> {
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for line in section.lines() {
|
||||||
|
let content = match line.split('#').next() {
|
||||||
|
std::option::Option::Some(value) => value.trim(),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
if content.is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match content.split('=').next() {
|
||||||
|
std::option::Option::Some(value) => value.trim().trim_end_matches(".workspace"),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
if !name.is_empty() {
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
names.sort_unstable();
|
||||||
|
return names;
|
||||||
|
}
|
||||||
281
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
Normal file
281
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
Normal file
@@ -0,0 +1,281 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
//! External public, security, redaction and release-boundary hardening canaries for `pre.010`.
|
||||||
|
|
||||||
|
fn network(value: &'static str) -> std::option::Option<ksp_store_lib::RawNetworkId> {
|
||||||
|
let result = ksp_store_lib::RawNetworkId::new(value);
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
assert_eq!(error.code(), ksp_store_lib::ERROR_CODE_RAW_MODEL_INVALID);
|
||||||
|
std::option::Option::None
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn worker_id(value: &'static str) -> std::option::Option<ksp_worker_api::WorkerId> {
|
||||||
|
let result = ksp_worker_api::WorkerId::new(value);
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
assert_eq!(error.code(), ksp_worker_api::ERROR_CODE_WORKER_ID_INVALID);
|
||||||
|
std::option::Option::None
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_external_error_codes_are_stable_unique_and_domain_scoped() {
|
||||||
|
let codes = [
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT, "content_conflict"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED, "counter_exhausted"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT, "drain_timeout"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID, "runtime_invalid"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID, "settings_invalid"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED, "source_failed"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED, "store_failed"),
|
||||||
|
];
|
||||||
|
for (index, (code, expected)) in codes.iter().enumerate() {
|
||||||
|
assert_eq!(code.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(code.code(), *expected);
|
||||||
|
for (other_index, (other, _)) in codes.iter().enumerate() {
|
||||||
|
if index != other_index {
|
||||||
|
assert_ne!(code, other);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_debug_and_settings_errors_redact_worker_identity_and_invalid_values() {
|
||||||
|
let raw_worker_id = "raw-ingest-secret-marker-001";
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let parsed_worker_id = match worker_id(raw_worker_id) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert_eq!(std::format!("{parsed_worker_id:?}"), "WorkerId(..)");
|
||||||
|
let settings = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::new(
|
||||||
|
network.clone(),
|
||||||
|
parsed_worker_id,
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
std::time::Duration::from_millis(100),
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let settings_debug = std::format!("{settings:?}");
|
||||||
|
assert!(!settings_debug.contains(raw_worker_id));
|
||||||
|
assert!(settings_debug.contains("WorkerId(..)"));
|
||||||
|
let invalid_worker_id = match worker_id("raw-ingest-secret-marker-002") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let invalid =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::new(network, invalid_worker_id, 65_537, 1, std::time::Duration::from_millis(100));
|
||||||
|
let error = match invalid {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(error) => error,
|
||||||
|
};
|
||||||
|
let error_debug = std::format!("{error:?}");
|
||||||
|
assert_eq!(error.code(), ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID);
|
||||||
|
assert!(!error_debug.contains("raw-ingest-secret-marker-002"));
|
||||||
|
assert!(!error_debug.contains("65537"));
|
||||||
|
assert!(error.context().iter().any(|context| return context.key() == "field" && context.value() == "admission_queue_capacity"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_manifest_dependency_surface_remains_exact_source_neutral_and_backend_neutral() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let mut section = "";
|
||||||
|
let mut normal = std::collections::BTreeSet::new();
|
||||||
|
let mut dev = std::collections::BTreeSet::new();
|
||||||
|
let mut build = std::collections::BTreeSet::new();
|
||||||
|
for line in manifest.lines() {
|
||||||
|
let trimmed = line.trim();
|
||||||
|
if trimmed.starts_with('[') && trimmed.ends_with(']') {
|
||||||
|
section = trimmed;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if trimmed.is_empty() || trimmed.starts_with('#') {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match trimmed.split_once('=') {
|
||||||
|
std::option::Option::Some((name, _)) => name.trim(),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
if section == "[dependencies]" {
|
||||||
|
normal.insert(name);
|
||||||
|
} else if section == "[dev-dependencies]" {
|
||||||
|
dev.insert(name);
|
||||||
|
} else if section == "[build-dependencies]" {
|
||||||
|
build.insert(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
normal,
|
||||||
|
std::collections::BTreeSet::from(["ksp-core-lib", "ksp-logging-lib", "ksp-raw-transaction-lib", "ksp-store-lib", "ksp-worker-api", "sha2", "tokio",])
|
||||||
|
);
|
||||||
|
assert!(dev.is_empty());
|
||||||
|
assert!(build.is_empty());
|
||||||
|
assert!(manifest.contains("ksp-store-lib = { path = \"../ksp-store-lib\", default-features = false }"));
|
||||||
|
assert!(manifest.contains("tokio = { workspace = true, features = [\"macros\", \"rt\", \"sync\", \"time\"] }"));
|
||||||
|
for forbidden in [
|
||||||
|
"ksp-config-lib",
|
||||||
|
"ksp-job-api",
|
||||||
|
"ksp-job-backfill-lib",
|
||||||
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-store-api",
|
||||||
|
"ksp-store-postgres-lib",
|
||||||
|
"reqwest",
|
||||||
|
"tokio-tungstenite",
|
||||||
|
"tonic",
|
||||||
|
"yellowstone-grpc-proto",
|
||||||
|
] {
|
||||||
|
assert!(!manifest.contains(forbidden), "forbidden Worker manifest dependency present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_source_visibility_contract_uses_crate_root_for_shared_items() {
|
||||||
|
let source_contracts: [(&str, &[&str]); 4] = [
|
||||||
|
(include_str!("../src/settings.rs"), &["RawTransactionIngestSettings"]),
|
||||||
|
(include_str!("../src/runtime.rs"), &["RawTransactionIngestHandle", "RawTransactionIngestWorker"]),
|
||||||
|
(include_str!("../src/snapshot.rs"), &["RawTransactionIngestSnapshot", "RawTransactionIngestSnapshotSource"]),
|
||||||
|
(include_str!("../src/persistence.rs"), &["RawTransactionIngestPersistenceOutcome", "RawTransactionIngestPersistencePort"]),
|
||||||
|
];
|
||||||
|
for (source, symbols) in source_contracts {
|
||||||
|
for symbol in symbols {
|
||||||
|
let required = std::format!("impl crate::{symbol}");
|
||||||
|
assert!(source.contains(required.as_str()), "shared item must use crate-root impl path: {symbol}");
|
||||||
|
let forbidden = std::format!("impl {symbol}");
|
||||||
|
assert!(!source.contains(forbidden.as_str()), "shared item must not use bare impl path: {symbol}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (module, source) in [
|
||||||
|
("admission", include_str!("../src/admission.rs")),
|
||||||
|
("error", include_str!("../src/error.rs")),
|
||||||
|
("identity", include_str!("../src/identity.rs")),
|
||||||
|
("persistence", include_str!("../src/persistence.rs")),
|
||||||
|
("runtime", include_str!("../src/runtime.rs")),
|
||||||
|
("settings", include_str!("../src/settings.rs")),
|
||||||
|
("snapshot", include_str!("../src/snapshot.rs")),
|
||||||
|
] {
|
||||||
|
let forbidden = std::format!("crate::{module}::");
|
||||||
|
assert!(!source.contains(forbidden.as_str()), "internal module path bypasses crate-root façade: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_production_surface_has_no_historical_backfill_or_retriever_contract() {
|
||||||
|
let sources = [
|
||||||
|
include_str!("../src/admission.rs"),
|
||||||
|
include_str!("../src/error.rs"),
|
||||||
|
include_str!("../src/identity.rs"),
|
||||||
|
include_str!("../src/lib.rs"),
|
||||||
|
include_str!("../src/persistence.rs"),
|
||||||
|
include_str!("../src/runtime.rs"),
|
||||||
|
include_str!("../src/settings.rs"),
|
||||||
|
include_str!("../src/snapshot.rs"),
|
||||||
|
];
|
||||||
|
for source in sources {
|
||||||
|
for forbidden in [
|
||||||
|
"ksp-worker-raw-retriever",
|
||||||
|
"raw_transaction_retriever",
|
||||||
|
"RawTransactionRetriever",
|
||||||
|
"Backfill",
|
||||||
|
"backfill",
|
||||||
|
"Checkpoint",
|
||||||
|
"checkpoint",
|
||||||
|
"Discovery",
|
||||||
|
"discovery",
|
||||||
|
"Hydration",
|
||||||
|
"hydration",
|
||||||
|
"historical",
|
||||||
|
] {
|
||||||
|
assert!(!source.contains(forbidden), "historical/retriever surface leaked into Worker production source: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_production_sources_scan_clean_for_config_secrets_backend_and_transport() {
|
||||||
|
let sources = [
|
||||||
|
include_str!("../src/admission.rs"),
|
||||||
|
include_str!("../src/error.rs"),
|
||||||
|
include_str!("../src/identity.rs"),
|
||||||
|
include_str!("../src/lib.rs"),
|
||||||
|
include_str!("../src/persistence.rs"),
|
||||||
|
include_str!("../src/runtime.rs"),
|
||||||
|
include_str!("../src/settings.rs"),
|
||||||
|
include_str!("../src/snapshot.rs"),
|
||||||
|
];
|
||||||
|
for source in sources {
|
||||||
|
let lower = source.to_ascii_lowercase();
|
||||||
|
for forbidden in ["api_key", "api-key", "authorization", "bearer ", "password", "credential", "secret"] {
|
||||||
|
assert!(!lower.contains(forbidden), "secret-like material leaked into Worker production source: {forbidden}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ksp_onchain_transport_lib::",
|
||||||
|
"ksp_offchain_transport_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"postgresql://",
|
||||||
|
"postgres://",
|
||||||
|
] {
|
||||||
|
assert!(!source.contains(forbidden), "Config/backend/Transport implementation leaked into Worker production source: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_lower_layers_have_no_dependency_return_to_concrete_worker() {
|
||||||
|
for manifest in [
|
||||||
|
include_str!("../../ksp-core-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-logging-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-raw-transaction-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-store-api/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-store-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-store-postgres-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-worker-api/Cargo.toml"),
|
||||||
|
] {
|
||||||
|
assert!(!manifest.contains("ksp-worker-raw-transaction-ingest-lib"));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_public_root_exposes_no_runtime_backend_or_live_source_implementation_types() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"pub mod ",
|
||||||
|
"tokio::",
|
||||||
|
"JoinSet",
|
||||||
|
"watch::Receiver",
|
||||||
|
"watch::Sender",
|
||||||
|
"mpsc::Sender",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ksp_onchain_transport_lib::",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tonic::",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "implementation/backend/live-source detail leaked into public root: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
137
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
Normal file
137
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
Normal file
@@ -0,0 +1,137 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
// version: 8
|
||||||
|
|
||||||
|
//! External public-surface proofs for the RAW transaction ingest Worker foundation.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_kind_code_and_settings_are_consumable_from_crate_root() {
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE, "raw_transaction_ingest");
|
||||||
|
let kind_result = ksp_worker_api::WorkerKindCode::new(ksp_worker_raw_transaction_ingest_lib::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE);
|
||||||
|
assert!(kind_result.is_ok());
|
||||||
|
let kind = match kind_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(kind.as_str(), "raw_transaction_ingest");
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-mainnet-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let settings = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::with_defaults(network, worker_id);
|
||||||
|
assert_eq!(settings.network().as_str(), "mainnet");
|
||||||
|
assert_eq!(settings.worker_id().as_str(), "raw-ingest-mainnet-001");
|
||||||
|
assert_eq!(settings.admission_queue_capacity(), ksp_worker_raw_transaction_ingest_lib::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY);
|
||||||
|
assert_eq!(settings.persistence_concurrency(), ksp_worker_raw_transaction_ingest_lib::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY);
|
||||||
|
assert_eq!(settings.shutdown_drain_timeout(), ksp_worker_raw_transaction_ingest_lib::DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_invalid_network_and_worker_identity_remain_owned_by_lower_contracts() {
|
||||||
|
let network = ksp_store_lib::RawNetworkId::new("");
|
||||||
|
let worker_id = ksp_worker_api::WorkerId::new("invalid worker id");
|
||||||
|
assert!(network.is_err());
|
||||||
|
assert!(worker_id.is_err());
|
||||||
|
let network_error = match network {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(network_error.code(), ksp_store_lib::ERROR_CODE_RAW_MODEL_INVALID);
|
||||||
|
let worker_error = match worker_id {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(worker_error.code(), ksp_worker_api::ERROR_CODE_WORKER_ID_INVALID);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_runtime_bound_errors_are_stable_and_do_not_echo_values() {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-mainnet-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let result = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::new(network, worker_id, 0, 8, std::time::Duration::from_secs(5));
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID);
|
||||||
|
assert_eq!(error.message(), "invalid RAW transaction ingest Worker settings");
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].value(), "admission_queue_capacity");
|
||||||
|
assert!(!std::format!("{error:?}").contains("raw-ingest-mainnet-001"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle() {
|
||||||
|
let _start: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings,
|
||||||
|
std::sync::Arc<ksp_store_lib::Store>,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWorker::start;
|
||||||
|
let _request_stop: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle) -> bool =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle::request_stop;
|
||||||
|
let _wait_terminal: for<'a> fn(
|
||||||
|
&'a ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestTerminalFuture<'a> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle::wait_terminal;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
assert!(!root.contains("JoinHandle"));
|
||||||
|
assert!(!root.contains("JoinSet"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_007_store_and_content_conflict_error_codes_are_public_and_stable() {
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT.code(), "content_conflict");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED.code(), "store_failed");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_snapshot_surface_and_common_projection_are_public_and_stable() {
|
||||||
|
let _snapshot_source: fn(
|
||||||
|
&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle::snapshot_source;
|
||||||
|
let _worker_snapshot_source: fn(
|
||||||
|
&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle::worker_snapshot_source;
|
||||||
|
let _current: fn(
|
||||||
|
&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource::current;
|
||||||
|
let _wait: for<'a> fn(
|
||||||
|
&'a ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource,
|
||||||
|
ksp_worker_api::WorkerSnapshotSequence,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotFuture<'a> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource::wait_for_change;
|
||||||
|
fn require_worker_source<T: ksp_worker_api::WorkerSnapshotSource + std::marker::Send + std::marker::Sync>() {}
|
||||||
|
require_worker_source::<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource>();
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED.code(), "counter_exhausted");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_009_source_and_drain_timeout_error_codes_are_public_and_stable() {
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT.code(), "drain_timeout");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED.code(), "source_failed");
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,113 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
//! Release-completeness canaries through the `pre.010` public/release/security hardening tranche.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_production_module_inventory_is_exact() -> std::io::Result<()> {
|
||||||
|
let source_root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src");
|
||||||
|
let entries = match std::fs::read_dir(source_root) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for entry in entries {
|
||||||
|
let entry = match entry {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let file_type = match entry.file_type() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if !file_type.is_file() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match entry.file_name().into_string() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => continue,
|
||||||
|
};
|
||||||
|
if name.ends_with(".rs") {
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
names.sort_unstable();
|
||||||
|
assert_eq!(names, std::vec!["admission.rs", "error.rs", "identity.rs", "lib.rs", "persistence.rs", "runtime.rs", "settings.rs", "snapshot.rs",]);
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_public_root_export_inventory_is_exact() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let mut exports = std::vec::Vec::new();
|
||||||
|
for line in root.lines() {
|
||||||
|
let line = line.trim();
|
||||||
|
if !line.starts_with("pub use ") {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let path = line.trim_start_matches("pub use ").trim_end_matches(';');
|
||||||
|
let name = match path.rsplit("::").next() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
exports.push(name);
|
||||||
|
}
|
||||||
|
exports.sort_unstable();
|
||||||
|
assert_eq!(
|
||||||
|
exports,
|
||||||
|
std::vec![
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY",
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY",
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT",
|
||||||
|
"RAW_TRANSACTION_INGEST_WORKER_KIND_CODE",
|
||||||
|
"RawTransactionIngestHandle",
|
||||||
|
"RawTransactionIngestSettings",
|
||||||
|
"RawTransactionIngestSnapshot",
|
||||||
|
"RawTransactionIngestSnapshotFuture",
|
||||||
|
"RawTransactionIngestSnapshotSource",
|
||||||
|
"RawTransactionIngestTerminalFuture",
|
||||||
|
"RawTransactionIngestWorker",
|
||||||
|
]
|
||||||
|
);
|
||||||
|
assert!(!root.contains("pub mod "));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_external_hardening_suite_is_present_and_scoped() {
|
||||||
|
let hardening = include_str!("hardening.rs");
|
||||||
|
for required in [
|
||||||
|
"pre_010_external_error_codes_are_stable_unique_and_domain_scoped",
|
||||||
|
"pre_010_debug_and_settings_errors_redact_worker_identity_and_invalid_values",
|
||||||
|
"pre_010_manifest_dependency_surface_remains_exact_source_neutral_and_backend_neutral",
|
||||||
|
"pre_010_source_visibility_contract_uses_crate_root_for_shared_items",
|
||||||
|
"pre_010_production_surface_has_no_historical_backfill_or_retriever_contract",
|
||||||
|
"pre_010_production_sources_scan_clean_for_config_secrets_backend_and_transport",
|
||||||
|
"pre_010_lower_layers_have_no_dependency_return_to_concrete_worker",
|
||||||
|
"pre_010_public_root_exposes_no_runtime_backend_or_live_source_implementation_types",
|
||||||
|
] {
|
||||||
|
assert!(hardening.contains(required), "required pre.010 hardening canary missing: {required}");
|
||||||
|
}
|
||||||
|
let dependency_boundary = include_str!("dependency_boundary.rs");
|
||||||
|
assert!(dependency_boundary.contains("pre_002_manifest_dependency_surface_is_exact"));
|
||||||
|
assert!(dependency_boundary.contains("pre_009_source_surface_hardens_shutdown_and_faults_without_backend_or_live_source"));
|
||||||
|
let public_api = include_str!("public_api.rs");
|
||||||
|
assert!(public_api.contains("pre_003_kind_code_and_settings_are_consumable_from_crate_root"));
|
||||||
|
assert!(public_api.contains("pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle"));
|
||||||
|
assert!(public_api.contains("pre_008_snapshot_surface_and_common_projection_are_public_and_stable"));
|
||||||
|
assert!(public_api.contains("pre_009_source_and_drain_timeout_error_codes_are_public_and_stable"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,264 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
// version: 4
|
||||||
|
|
||||||
|
fn material(
|
||||||
|
network: &str,
|
||||||
|
signature_byte: u8,
|
||||||
|
transaction_data: &str,
|
||||||
|
) -> std::option::Option<(ksp_store_lib::RawNetworkId, ksp_raw_transaction_lib::RawTransactionMaterial)> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new(network) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let signature = ksp_store_lib::RawTransactionSignature::new([signature_byte; 64]);
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
network.clone(),
|
||||||
|
signature,
|
||||||
|
42,
|
||||||
|
std::option::Option::Some(1_700_000_000),
|
||||||
|
transaction_data,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
return std::option::Option::Some((network, material));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn provenance() -> std::option::Option<ksp_store_lib::RawAcquisitionProvenance> {
|
||||||
|
let provider = match ksp_store_lib::RawProvenanceCode::new("deterministic-harness") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let protocol = match ksp_store_lib::RawProvenanceCode::new("internal") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let method = match ksp_store_lib::RawProvenanceCode::new("admission") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_lib::RawTimestamp::from_unix_millis(1_700_000_000_000) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(ksp_store_lib::RawAcquisitionProvenance::new(
|
||||||
|
provider,
|
||||||
|
protocol,
|
||||||
|
method,
|
||||||
|
ksp_store_lib::RawAcquisitionOrigin::Live,
|
||||||
|
received_at,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_006_bounded_admission_applies_async_backpressure_until_one_slot_is_consumed() {
|
||||||
|
let (network, first_material) = match material("mainnet", 1, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (_, second_material) = match material("mainnet", 2, "BAUG") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
|
||||||
|
let first_ingress = crate::RawTransactionIngress {
|
||||||
|
material: first_material,
|
||||||
|
network: network.clone(),
|
||||||
|
provenance: first_provenance,
|
||||||
|
source_key: [1; 32],
|
||||||
|
};
|
||||||
|
assert!(sender.send(first_ingress).await.is_ok());
|
||||||
|
let second_sender = sender.clone();
|
||||||
|
let second_network = network.clone();
|
||||||
|
let completed = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
|
||||||
|
let completed_task = std::sync::Arc::clone(&completed);
|
||||||
|
let second_task = tokio::spawn(async move {
|
||||||
|
let second_ingress = crate::RawTransactionIngress {
|
||||||
|
material: second_material,
|
||||||
|
network: second_network,
|
||||||
|
provenance: second_provenance,
|
||||||
|
source_key: [2; 32],
|
||||||
|
};
|
||||||
|
let admitted = second_sender.send(second_ingress).await.is_ok();
|
||||||
|
completed_task.store(admitted, std::sync::atomic::Ordering::Release);
|
||||||
|
return admitted;
|
||||||
|
});
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
assert!(!completed.load(std::sync::atomic::Ordering::Acquire));
|
||||||
|
let first = admission.receive(&network).await;
|
||||||
|
assert!(matches!(first, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
for _ in 0..64 {
|
||||||
|
if completed.load(std::sync::atomic::Ordering::Acquire) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert!(completed.load(std::sync::atomic::Ordering::Acquire));
|
||||||
|
let joined = second_task.await;
|
||||||
|
assert!(matches!(joined, std::result::Result::Ok(true)));
|
||||||
|
let second = admission.receive(&network).await;
|
||||||
|
assert!(matches!(second, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_006_stop_preempts_a_blocked_sender_without_silent_post_stop_admission() {
|
||||||
|
let (network, first_material) = match material("mainnet", 3, "BwgJ") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (_, second_material) = match material("mainnet", 4, "CgsM") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (stop_sender, stop_receiver) = tokio::sync::watch::channel(false);
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
|
||||||
|
let first_ingress = crate::RawTransactionIngress {
|
||||||
|
material: first_material,
|
||||||
|
network: network.clone(),
|
||||||
|
provenance: first_provenance,
|
||||||
|
source_key: [3; 32],
|
||||||
|
};
|
||||||
|
assert!(sender.send(first_ingress).await.is_ok());
|
||||||
|
let blocked_sender = sender.clone();
|
||||||
|
let blocked_network = network.clone();
|
||||||
|
let blocked_task = tokio::spawn(async move {
|
||||||
|
let mut blocked_stop = stop_receiver;
|
||||||
|
let blocked_ingress = crate::RawTransactionIngress {
|
||||||
|
material: second_material,
|
||||||
|
network: blocked_network,
|
||||||
|
provenance: second_provenance,
|
||||||
|
source_key: [4; 32],
|
||||||
|
};
|
||||||
|
return tokio::select! {
|
||||||
|
biased;
|
||||||
|
_changed = blocked_stop.changed() => false,
|
||||||
|
result = blocked_sender.send(blocked_ingress) => result.is_ok(),
|
||||||
|
};
|
||||||
|
});
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
stop_sender.send_replace(true);
|
||||||
|
let blocked = blocked_task.await;
|
||||||
|
assert!(matches!(blocked, std::result::Result::Ok(false)));
|
||||||
|
let first = admission.receive(&network).await;
|
||||||
|
assert!(matches!(first, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
admission.close();
|
||||||
|
let end = admission.receive(&network).await;
|
||||||
|
assert!(matches!(end, std::result::Result::Ok(std::option::Option::None)));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_observation_key_domain_and_common_raw_assembly_golden_are_exact() {
|
||||||
|
let (network, material) = match material("mainnet", 7, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let ingress = crate::RawTransactionIngress { material, network: network.clone(), provenance: provenance.clone(), source_key: [9; 32] };
|
||||||
|
let acquisition = super::canonicalize_ingress(&network, ingress);
|
||||||
|
assert!(acquisition.is_ok());
|
||||||
|
let acquisition = match acquisition {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(acquisition.transaction().reference().network(), &network);
|
||||||
|
assert_eq!(acquisition.transaction().reference().signature().as_bytes(), &[7; 64]);
|
||||||
|
assert_eq!(acquisition.transaction().slot(), 42);
|
||||||
|
assert_eq!(acquisition.transaction().payload().format_id().as_str(), ksp_raw_transaction_lib::RAW_TRANSACTION_FORMAT_ID);
|
||||||
|
assert_eq!(acquisition.transaction().payload().format_version(), ksp_raw_transaction_lib::RAW_TRANSACTION_FORMAT_VERSION);
|
||||||
|
assert_eq!(acquisition.transaction().payload().bytes(), br#"{"transaction":["AQID","base64"]}"#);
|
||||||
|
assert_eq!(acquisition.observation().provenance(), &provenance);
|
||||||
|
assert_eq!(acquisition.observation().transaction(), acquisition.transaction().reference());
|
||||||
|
assert_eq!(
|
||||||
|
acquisition.observation().observation_key().as_bytes(),
|
||||||
|
&[
|
||||||
|
248, 209, 97, 220, 168, 168, 167, 42, 13, 161, 16, 252, 135, 136, 46, 231, 19, 104, 72, 70, 28, 106, 163, 43, 9, 227, 10, 97, 136, 185, 93, 204,
|
||||||
|
],
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_network_guards_reject_ingress_or_material_mismatch_without_echoing_values() {
|
||||||
|
let (mainnet, mainnet_material) = match material("mainnet", 8, "DQ4P") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (devnet, devnet_material) = match material("devnet", 9, "EBES") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let wrong_ingress = crate::RawTransactionIngress { material: mainnet_material, network: devnet, provenance: first_provenance, source_key: [10; 32] };
|
||||||
|
let wrong_ingress = super::canonicalize_ingress(&mainnet, wrong_ingress);
|
||||||
|
assert!(wrong_ingress.is_err());
|
||||||
|
let wrong_material = crate::RawTransactionIngress {
|
||||||
|
material: devnet_material,
|
||||||
|
network: mainnet.clone(),
|
||||||
|
provenance: second_provenance,
|
||||||
|
source_key: [11; 32],
|
||||||
|
};
|
||||||
|
let wrong_material = super::canonicalize_ingress(&mainnet, wrong_material);
|
||||||
|
assert!(wrong_material.is_err());
|
||||||
|
for result in [wrong_ingress, wrong_material] {
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => continue,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
let rendered = std::format!("{error:?}");
|
||||||
|
assert!(!rendered.contains("mainnet"));
|
||||||
|
assert!(!rendered.contains("devnet"));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_full_queue_dequeue_marks_source_neutral_backpressure_observation() {
|
||||||
|
let (network, material) = match material("mainnet", 13, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
|
||||||
|
let ingress = crate::RawTransactionIngress { material, network: network.clone(), provenance, source_key: [13; 32] };
|
||||||
|
assert!(sender.send(ingress).await.is_ok());
|
||||||
|
let first = admission.receive(&network).await;
|
||||||
|
assert!(matches!(first, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
assert!(admission.take_backpressure_wait_observed());
|
||||||
|
admission.close();
|
||||||
|
let second = admission.receive(&network).await;
|
||||||
|
assert!(matches!(second, std::result::Result::Ok(std::option::Option::None)));
|
||||||
|
assert!(!admission.take_backpressure_wait_observed());
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,253 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/persistence.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
#[derive(Clone, Copy)]
|
||||||
|
enum PortResponse {
|
||||||
|
Outcome(ksp_store_lib::RawEntityWriteOutcome, ksp_store_lib::RawObservationWriteOutcome),
|
||||||
|
Conflict,
|
||||||
|
StoreFailure,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct FakePersistencePort {
|
||||||
|
calls: std::sync::atomic::AtomicUsize,
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
normal_mode_seen: std::sync::atomic::AtomicBool,
|
||||||
|
response: PortResponse,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FakePersistencePort {
|
||||||
|
fn new(network: ksp_store_lib::RawNetworkId, response: PortResponse) -> Self {
|
||||||
|
return Self {
|
||||||
|
calls: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
network,
|
||||||
|
normal_mode_seen: std::sync::atomic::AtomicBool::new(false),
|
||||||
|
response,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistencePort for FakePersistencePort {
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool {
|
||||||
|
return &self.network == network;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
_transaction: ksp_store_lib::RawTransaction,
|
||||||
|
_observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>> {
|
||||||
|
self.calls.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
self.normal_mode_seen.store(mode == ksp_store_lib::RawTransactionAcquisitionMode::Normal, std::sync::atomic::Ordering::Release);
|
||||||
|
let response = self.response;
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
return match response {
|
||||||
|
PortResponse::Outcome(entity, observation) => std::result::Result::Ok(ksp_store_lib::RawAcquisitionWriteOutcome::new(entity, observation)),
|
||||||
|
PortResponse::Conflict => std::result::Result::Err(ksp_core_lib::Error::new(ksp_store_lib::ERROR_CODE_RAW_CONFLICT, "synthetic conflict")),
|
||||||
|
PortResponse::StoreFailure => std::result::Result::Err(ksp_core_lib::Error::new(
|
||||||
|
ksp_core_lib::ErrorCode::new("synthetic_store", "write_failed"),
|
||||||
|
"synthetic remote failure text that must not escape",
|
||||||
|
)),
|
||||||
|
};
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn acquisition(network_name: &str, signature_byte: u8) -> std::option::Option<ksp_raw_transaction_lib::RawTransactionAcquisition> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new(network_name) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let signature = ksp_store_lib::RawTransactionSignature::new([signature_byte; 64]);
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
network,
|
||||||
|
signature,
|
||||||
|
42,
|
||||||
|
std::option::Option::Some(1_700_000_000),
|
||||||
|
"AQID",
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
let transaction = match ksp_raw_transaction_lib::canonicalize_raw_transaction(material) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let provider = match ksp_store_lib::RawProvenanceCode::new("deterministic-harness") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let protocol = match ksp_store_lib::RawProvenanceCode::new("internal") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let method = match ksp_store_lib::RawProvenanceCode::new("persistence") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_lib::RawTimestamp::from_unix_millis(1_700_000_000_000) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let provenance = ksp_store_lib::RawAcquisitionProvenance::new(provider, protocol, method, ksp_store_lib::RawAcquisitionOrigin::Live, received_at);
|
||||||
|
return std::option::Option::Some(ksp_raw_transaction_lib::assemble_raw_transaction_acquisition(
|
||||||
|
transaction,
|
||||||
|
ksp_store_lib::RawObservationKey::new([signature_byte; 32]),
|
||||||
|
provenance,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn network(name: &str) -> std::option::Option<ksp_store_lib::RawNetworkId> {
|
||||||
|
return match ksp_store_lib::RawNetworkId::new(name) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_normal_mode_classifies_new_idempotent_and_purged_outcomes() {
|
||||||
|
let cases = [
|
||||||
|
(
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
crate::RawTransactionIngestEntityPersistence::Inserted,
|
||||||
|
crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent, ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent),
|
||||||
|
crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
crate::RawTransactionIngestObservationPersistence::AlreadyPresent,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::SkippedPurged, ksp_store_lib::RawObservationWriteOutcome::NotRecorded),
|
||||||
|
crate::RawTransactionIngestEntityPersistence::SkippedPurged,
|
||||||
|
crate::RawTransactionIngestObservationPersistence::NotRecorded,
|
||||||
|
),
|
||||||
|
];
|
||||||
|
for (index, (response, expected_entity, expected_observation)) in cases.into_iter().enumerate() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, response);
|
||||||
|
let acquisition = match acquisition("mainnet", index as u8 + 1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
let outcome = match result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(outcome.entity(), expected_entity);
|
||||||
|
assert_eq!(outcome.observation(), expected_observation);
|
||||||
|
assert!(port.normal_mode_seen.load(std::sync::atomic::Ordering::Acquire));
|
||||||
|
assert_eq!(port.calls.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_content_conflict_maps_to_terminal_worker_code_without_remote_text() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, PortResponse::Conflict);
|
||||||
|
let acquisition = match acquisition("mainnet", 9) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT);
|
||||||
|
assert!(error.context().is_empty());
|
||||||
|
assert!(!std::format!("{error:?}").contains("synthetic conflict"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_store_failure_keeps_only_safe_lower_error_code() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, PortResponse::StoreFailure);
|
||||||
|
let acquisition = match acquisition("mainnet", 10) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED);
|
||||||
|
assert_eq!(error.context().len(), 2);
|
||||||
|
assert_eq!(error.context()[0].value(), "synthetic_store");
|
||||||
|
assert_eq!(error.context()[1].value(), "write_failed");
|
||||||
|
assert!(!std::format!("{error:?}").contains("synthetic remote failure text"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_rehydrated_or_impossible_store_outcomes_are_runtime_invalid() {
|
||||||
|
let cases = [
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Rehydrated, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent),
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::SkippedPurged, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
];
|
||||||
|
for (index, response) in cases.into_iter().enumerate() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, response);
|
||||||
|
let acquisition = match acquisition("mainnet", index as u8 + 20) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => continue,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].value(), "persistence.store_outcome_invalid");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_store_network_mismatch_is_rejected_before_write() {
|
||||||
|
let network = match network("devnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(
|
||||||
|
network,
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
);
|
||||||
|
let acquisition = match acquisition("mainnet", 30) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
assert_eq!(error.context()[0].value(), "persistence.store_network_mismatch");
|
||||||
|
assert_eq!(port.calls.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,871 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
// version: 9
|
||||||
|
|
||||||
|
struct ActiveTaskGuard {
|
||||||
|
active: std::sync::Arc<std::sync::atomic::AtomicUsize>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ActiveTaskGuard {
|
||||||
|
fn new(active: std::sync::Arc<std::sync::atomic::AtomicUsize>) -> Self {
|
||||||
|
active.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return Self { active };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::ops::Drop for ActiveTaskGuard {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
self.active.fetch_sub(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn settings(network: &str) -> std::option::Option<crate::RawTransactionIngestSettings> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new(network) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-runtime-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(crate::RawTransactionIngestSettings::with_defaults(network, worker_id));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_for_active_count(active: &std::sync::Arc<std::sync::atomic::AtomicUsize>, expected: usize) -> bool {
|
||||||
|
for _ in 0..64 {
|
||||||
|
if active.load(std::sync::atomic::Ordering::Acquire) == expected {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
return active.load(std::sync::atomic::Ordering::Acquire) == expected;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_current_runtime_is_required_before_spawn() {
|
||||||
|
let result = super::current_runtime_handle();
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].value(), "start.runtime_unavailable");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_store_network_mismatch_is_rejected_without_echoing_network_values() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let other = match ksp_store_lib::RawNetworkId::new("devnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let result = super::validate_store_network(&settings, &other);
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
let rendered = std::format!("{error:?}");
|
||||||
|
assert!(!rendered.contains("mainnet"));
|
||||||
|
assert!(!rendered.contains("devnet"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_004_immediate_stop_is_idempotent_and_reaches_stopped_terminal() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation(settings, tokio::runtime::Handle::current(), std::option::Option::None) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(handle.snapshot_source().current().worker_snapshot().state(), ksp_worker_api::WorkerState::Starting);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
assert!(!handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
assert!(terminal.is_ok());
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert!(terminal.is_terminal());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_004_running_lifecycle_stops_through_cloned_handle() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation(settings, tokio::runtime::Handle::current(), std::option::Option::None) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
assert_eq!(handle.snapshot_source().current().worker_snapshot().state(), ksp_worker_api::WorkerState::Running);
|
||||||
|
let clone = handle.clone();
|
||||||
|
assert!(clone.request_stop());
|
||||||
|
assert!(!handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_004_dropping_last_control_handle_causes_private_runtime_exit() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation(settings, tokio::runtime::Handle::current(), std::option::Option::None) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let observer = handle.snapshot_source();
|
||||||
|
std::mem::drop(handle);
|
||||||
|
loop {
|
||||||
|
let current = observer.current();
|
||||||
|
if current.worker_snapshot().state().is_terminal() {
|
||||||
|
assert_eq!(current.worker_snapshot().state(), ksp_worker_api::WorkerState::Stopped);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let _changed = observer.wait_for_change(current.worker_snapshot().sequence()).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_005_supervisor_joins_all_cooperative_source_tasks_before_terminal() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_active = std::sync::Arc::clone(&active);
|
||||||
|
let handle = match super::start_foundation_with_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::None,
|
||||||
|
move |children, stop_receiver, _admission_sender| {
|
||||||
|
for _ in 0..3 {
|
||||||
|
let active = std::sync::Arc::clone(&source_active);
|
||||||
|
let mut child_stop = stop_receiver.clone();
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let _guard = ActiveTaskGuard::new(active);
|
||||||
|
loop {
|
||||||
|
let changed = child_stop.changed().await;
|
||||||
|
if changed.is_err() || *child_stop.borrow() {
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_active_count(&active, 3).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_005_supervisor_reaps_completed_source_task_and_still_joins_live_child_on_stop() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let completed = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_active = std::sync::Arc::clone(&active);
|
||||||
|
let source_completed = std::sync::Arc::clone(&completed);
|
||||||
|
let handle = match super::start_foundation_with_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::None,
|
||||||
|
move |children, stop_receiver, _admission_sender| {
|
||||||
|
let completed = std::sync::Arc::clone(&source_completed);
|
||||||
|
let _completed_abort_handle = children.spawn(async move {
|
||||||
|
completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
let active = std::sync::Arc::clone(&source_active);
|
||||||
|
let mut child_stop = stop_receiver.clone();
|
||||||
|
let _active_abort_handle = children.spawn(async move {
|
||||||
|
let _guard = ActiveTaskGuard::new(active);
|
||||||
|
loop {
|
||||||
|
let changed = child_stop.changed().await;
|
||||||
|
if changed.is_err() || *child_stop.borrow() {
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_active_count(&active, 1).await);
|
||||||
|
for _ in 0..64 {
|
||||||
|
if completed.load(std::sync::atomic::Ordering::Acquire) == 1 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert_eq!(completed.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Copy)]
|
||||||
|
enum RuntimePortResponse {
|
||||||
|
Conflict,
|
||||||
|
StoreFailure,
|
||||||
|
StoreFailureBlocked,
|
||||||
|
SuccessBlocked,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct RuntimePortActiveGuard<'a> {
|
||||||
|
active: &'a std::sync::atomic::AtomicUsize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::ops::Drop for RuntimePortActiveGuard<'_> {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
self.active.fetch_sub(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
struct RuntimePersistencePort {
|
||||||
|
active: std::sync::atomic::AtomicUsize,
|
||||||
|
completed: std::sync::atomic::AtomicUsize,
|
||||||
|
max_active: std::sync::atomic::AtomicUsize,
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
normal_mode_seen: std::sync::atomic::AtomicBool,
|
||||||
|
notify: tokio::sync::Notify,
|
||||||
|
released: std::sync::atomic::AtomicBool,
|
||||||
|
response: RuntimePortResponse,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RuntimePersistencePort {
|
||||||
|
fn new(network: ksp_store_lib::RawNetworkId, response: RuntimePortResponse, released: bool) -> Self {
|
||||||
|
return Self {
|
||||||
|
active: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
completed: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
max_active: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
network,
|
||||||
|
normal_mode_seen: std::sync::atomic::AtomicBool::new(false),
|
||||||
|
notify: tokio::sync::Notify::new(),
|
||||||
|
released: std::sync::atomic::AtomicBool::new(released),
|
||||||
|
response,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn release(&self) {
|
||||||
|
self.released.store(true, std::sync::atomic::Ordering::Release);
|
||||||
|
self.notify.notify_waiters();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn update_max_active(&self, current: usize) {
|
||||||
|
let mut observed = self.max_active.load(std::sync::atomic::Ordering::Acquire);
|
||||||
|
while current > observed {
|
||||||
|
match self.max_active.compare_exchange_weak(observed, current, std::sync::atomic::Ordering::AcqRel, std::sync::atomic::Ordering::Acquire) {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => observed = value,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistencePort for RuntimePersistencePort {
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool {
|
||||||
|
return &self.network == network;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
_transaction: ksp_store_lib::RawTransaction,
|
||||||
|
_observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>> {
|
||||||
|
self.normal_mode_seen.store(mode == ksp_store_lib::RawTransactionAcquisitionMode::Normal, std::sync::atomic::Ordering::Release);
|
||||||
|
let response = self.response;
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
if matches!(response, RuntimePortResponse::StoreFailureBlocked | RuntimePortResponse::SuccessBlocked) {
|
||||||
|
let current = self.active.fetch_add(1, std::sync::atomic::Ordering::AcqRel) + 1;
|
||||||
|
let _active_guard = RuntimePortActiveGuard { active: &self.active };
|
||||||
|
self.update_max_active(current);
|
||||||
|
while !self.released.load(std::sync::atomic::Ordering::Acquire) {
|
||||||
|
self.notify.notified().await;
|
||||||
|
}
|
||||||
|
if matches!(response, RuntimePortResponse::SuccessBlocked) {
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Ok(ksp_store_lib::RawAcquisitionWriteOutcome::new(
|
||||||
|
ksp_store_lib::RawEntityWriteOutcome::Inserted,
|
||||||
|
ksp_store_lib::RawObservationWriteOutcome::Inserted,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if matches!(response, RuntimePortResponse::Conflict) {
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(ksp_store_lib::ERROR_CODE_RAW_CONFLICT, "synthetic conflict"));
|
||||||
|
}
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(
|
||||||
|
ksp_core_lib::ErrorCode::new("synthetic_store", "write_failed"),
|
||||||
|
"synthetic store failure",
|
||||||
|
));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn runtime_ingress(network: &ksp_store_lib::RawNetworkId, signature_byte: u8) -> std::option::Option<crate::RawTransactionIngress> {
|
||||||
|
let signature = ksp_store_lib::RawTransactionSignature::new([signature_byte; 64]);
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
network.clone(),
|
||||||
|
signature,
|
||||||
|
42,
|
||||||
|
std::option::Option::Some(1_700_000_000),
|
||||||
|
"AQID",
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
let provider = match ksp_store_lib::RawProvenanceCode::new("deterministic-harness") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let protocol = match ksp_store_lib::RawProvenanceCode::new("internal") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let method = match ksp_store_lib::RawProvenanceCode::new("runtime") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_lib::RawTimestamp::from_unix_millis(1_700_000_000_000) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let provenance = ksp_store_lib::RawAcquisitionProvenance::new(provider, protocol, method, ksp_store_lib::RawAcquisitionOrigin::Live, received_at);
|
||||||
|
return std::option::Option::Some(crate::RawTransactionIngress { material, network: network.clone(), provenance, source_key: [signature_byte; 32] });
|
||||||
|
}
|
||||||
|
|
||||||
|
fn settings_with_persistence_concurrency(concurrency: usize) -> std::option::Option<crate::RawTransactionIngestSettings> {
|
||||||
|
return settings_with_runtime_limits(8, concurrency, std::time::Duration::from_secs(5));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn settings_with_runtime_limits(
|
||||||
|
admission_queue_capacity: usize,
|
||||||
|
persistence_concurrency: usize,
|
||||||
|
shutdown_drain_timeout: std::time::Duration,
|
||||||
|
) -> std::option::Option<crate::RawTransactionIngestSettings> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-runtime-persistence-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return match crate::RawTransactionIngestSettings::new(network, worker_id, admission_queue_capacity, persistence_concurrency, shutdown_drain_timeout) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_for_completed(port: &std::sync::Arc<RuntimePersistencePort>, expected: usize) -> bool {
|
||||||
|
for _ in 0..128 {
|
||||||
|
if port.completed.load(std::sync::atomic::Ordering::Acquire) == expected {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
return port.completed.load(std::sync::atomic::Ordering::Acquire) == expected;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_for_max_active(port: &std::sync::Arc<RuntimePersistencePort>, expected: usize) -> bool {
|
||||||
|
for _ in 0..128 {
|
||||||
|
if port.max_active.load(std::sync::atomic::Ordering::Acquire) == expected {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
return port.max_active.load(std::sync::atomic::Ordering::Acquire) == expected;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_runtime_bounds_in_flight_store_persistence_to_configured_concurrency() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
for signature_byte in 1..=4 {
|
||||||
|
let ingress = match runtime_ingress(&network, signature_byte) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 2).await);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 2);
|
||||||
|
assert_eq!(port.max_active.load(std::sync::atomic::Ordering::Acquire), 2);
|
||||||
|
port.release();
|
||||||
|
assert!(wait_for_completed(&port, 4).await);
|
||||||
|
assert!(port.normal_mode_seen.load(std::sync::atomic::Ordering::Acquire));
|
||||||
|
assert_eq!(port.max_active.load(std::sync::atomic::Ordering::Acquire), 2);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_content_conflict_becomes_terminal_after_private_drain() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::Conflict, true));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 11) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT));
|
||||||
|
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
let snapshot = source.current();
|
||||||
|
assert_eq!(snapshot.content_conflict_total(), 1);
|
||||||
|
assert_eq!(snapshot.store_failure_total(), 0);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_store_failure_becomes_terminal_without_exposing_remote_error_text() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::StoreFailure, true));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 12) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED));
|
||||||
|
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert!(!std::format!("{handle:?}").contains("synthetic store failure"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_008_runtime_snapshots_count_successful_pipeline_and_retain_terminal() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, true));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
for signature_byte in 21..=23 {
|
||||||
|
let ingress = match runtime_ingress(&network, signature_byte) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let concrete_source = handle.snapshot_source();
|
||||||
|
let common_source = handle.worker_snapshot_source();
|
||||||
|
assert!(wait_for_completed(&port, 3).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
let snapshot = concrete_source.current();
|
||||||
|
assert_eq!(snapshot.worker_snapshot().state(), ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
assert_eq!(snapshot.admitted_total(), 3);
|
||||||
|
assert_eq!(snapshot.canonicalized_total(), 3);
|
||||||
|
assert_eq!(snapshot.persisted_total(), 3);
|
||||||
|
assert_eq!(snapshot.entity_inserted_total(), 3);
|
||||||
|
assert_eq!(snapshot.entity_already_present_total(), 0);
|
||||||
|
assert_eq!(snapshot.entity_skipped_purged_total(), 0);
|
||||||
|
assert_eq!(snapshot.observation_inserted_total(), 3);
|
||||||
|
assert_eq!(snapshot.observation_already_present_total(), 0);
|
||||||
|
assert_eq!(snapshot.content_conflict_total(), 0);
|
||||||
|
assert_eq!(snapshot.store_failure_total(), 0);
|
||||||
|
assert_eq!(snapshot.source_failure_total(), 0);
|
||||||
|
assert_eq!(snapshot.backpressure_wait_total(), 0);
|
||||||
|
assert_eq!(snapshot.admission_queue_depth(), 0);
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::current(&common_source);
|
||||||
|
assert_eq!(&common, snapshot.worker_snapshot());
|
||||||
|
let retained = concrete_source.wait_for_change(snapshot.worker_snapshot().sequence()).await;
|
||||||
|
assert_eq!(retained, snapshot);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_008_fault_snapshots_count_classified_store_failures_and_project_unhealthy() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::StoreFailure, true));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 24) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED));
|
||||||
|
let snapshot = source.current();
|
||||||
|
assert_eq!(snapshot.worker_snapshot().state(), terminal);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
assert_eq!(snapshot.admitted_total(), 1);
|
||||||
|
assert_eq!(snapshot.canonicalized_total(), 1);
|
||||||
|
assert_eq!(snapshot.persisted_total(), 0);
|
||||||
|
assert_eq!(snapshot.store_failure_total(), 1);
|
||||||
|
assert_eq!(snapshot.content_conflict_total(), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_source_failure_is_counted_and_late_stop_cannot_replace_terminal_fault() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation_with_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::None,
|
||||||
|
move |children: &mut tokio::task::JoinSet<ksp_core_lib::Result<()>>, _stop_receiver, _admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED));
|
||||||
|
let snapshot = source.current();
|
||||||
|
assert_eq!(snapshot.source_failure_total(), 1);
|
||||||
|
assert_eq!(snapshot.backpressure_wait_total(), 0);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
assert!(!handle.request_stop());
|
||||||
|
let retained = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(retained, terminal);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_stop_does_not_hide_store_failure_observed_during_bounded_drain() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::StoreFailureBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 41) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.ingress_invalid")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 1).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
port.release();
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED));
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
assert_eq!(snapshot.store_failure_total(), 1);
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_saturation_is_observable_without_drop_or_unbounded_admission() {
|
||||||
|
let settings = match settings_with_runtime_limits(1, 1, std::time::Duration::from_secs(5)) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let source_stage = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_stage_for_task = source_stage.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
for signature_byte in 51..=53 {
|
||||||
|
let ingress = match runtime_ingress(&network, signature_byte) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.ingress_invalid")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
source_stage_for_task.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 1).await);
|
||||||
|
for _ in 0..128 {
|
||||||
|
if source_stage.load(std::sync::atomic::Ordering::Acquire) == 2 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert_eq!(source_stage.load(std::sync::atomic::Ordering::Acquire), 2);
|
||||||
|
port.release();
|
||||||
|
assert!(wait_for_completed(&port, 3).await);
|
||||||
|
for _ in 0..128 {
|
||||||
|
if source_stage.load(std::sync::atomic::Ordering::Acquire) == 3 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert_eq!(source_stage.load(std::sync::atomic::Ordering::Acquire), 3);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
assert_eq!(snapshot.persisted_total(), 3);
|
||||||
|
assert_eq!(snapshot.source_failure_total(), 0);
|
||||||
|
assert!(snapshot.backpressure_wait_total() >= 1);
|
||||||
|
assert_eq!(snapshot.admission_queue_depth(), 0);
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_drain_timeout_aborts_and_joins_all_owned_source_and_persistence_tasks() {
|
||||||
|
let settings = match settings_with_runtime_limits(1, 1, crate::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let source_active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_active_for_task = source_active.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 61) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.ingress_invalid")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
let _guard = ActiveTaskGuard::new(source_active_for_task);
|
||||||
|
return std::future::pending::<ksp_core_lib::Result<()>>().await;
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 1).await);
|
||||||
|
assert!(wait_for_active_count(&source_active, 1).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT));
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
assert_eq!(snapshot.admission_queue_depth(), 0);
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
assert_eq!(source_active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/settings.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
fn identities() -> std::option::Option<(ksp_store_lib::RawNetworkId, ksp_worker_api::WorkerId)> {
|
||||||
|
let network_result = ksp_store_lib::RawNetworkId::new("mainnet");
|
||||||
|
assert!(network_result.is_ok());
|
||||||
|
let network = match network_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let worker_id_result = ksp_worker_api::WorkerId::new("raw-ingest-mainnet-001");
|
||||||
|
assert!(worker_id_result.is_ok());
|
||||||
|
let worker_id = match worker_id_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some((network, worker_id));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_defaults_are_exact_and_preserve_typed_identity() {
|
||||||
|
let (network, worker_id) = match identities() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let settings = crate::RawTransactionIngestSettings::with_defaults(network.clone(), worker_id.clone());
|
||||||
|
assert_eq!(settings.network(), &network);
|
||||||
|
assert_eq!(settings.worker_id(), &worker_id);
|
||||||
|
assert_eq!(settings.admission_queue_capacity(), 256);
|
||||||
|
assert_eq!(settings.persistence_concurrency(), 8);
|
||||||
|
assert_eq!(settings.shutdown_drain_timeout(), std::time::Duration::from_secs(5));
|
||||||
|
assert_eq!(crate::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY, 256);
|
||||||
|
assert_eq!(crate::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY, 8);
|
||||||
|
assert_eq!(crate::DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT, std::time::Duration::from_secs(5));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_inclusive_bounds_accept_exact_minimum_and_maximum() {
|
||||||
|
for (queue, concurrency, drain) in [
|
||||||
|
(
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT,
|
||||||
|
),
|
||||||
|
] {
|
||||||
|
let (network, worker_id) = match identities() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let settings = crate::RawTransactionIngestSettings::new(network, worker_id, queue, concurrency, drain);
|
||||||
|
assert!(settings.is_ok());
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_invalid_runtime_bounds_return_stable_field_scoped_error() {
|
||||||
|
let invalid = [
|
||||||
|
(0, 8, std::time::Duration::from_secs(5), "admission_queue_capacity"),
|
||||||
|
(65_537, 8, std::time::Duration::from_secs(5), "admission_queue_capacity"),
|
||||||
|
(256, 0, std::time::Duration::from_secs(5), "persistence_concurrency"),
|
||||||
|
(256, 65, std::time::Duration::from_secs(5), "persistence_concurrency"),
|
||||||
|
(256, 8, std::time::Duration::from_millis(99), "shutdown_drain_timeout"),
|
||||||
|
(256, 8, std::time::Duration::from_millis(30_001), "shutdown_drain_timeout"),
|
||||||
|
];
|
||||||
|
for (queue, concurrency, drain, field) in invalid {
|
||||||
|
let (network, worker_id) = match identities() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::RawTransactionIngestSettings::new(network, worker_id, queue, concurrency, drain);
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => continue,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID);
|
||||||
|
assert_eq!(error.code().domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(error.code().code(), "settings_invalid");
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].key(), "field");
|
||||||
|
assert_eq!(error.context()[0].value(), field);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_debug_keeps_worker_identity_redacted() {
|
||||||
|
let (network, worker_id) = match identities() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let settings = crate::RawTransactionIngestSettings::with_defaults(network, worker_id);
|
||||||
|
let rendered = std::format!("{settings:?}");
|
||||||
|
assert!(rendered.contains("mainnet"));
|
||||||
|
assert!(rendered.contains("WorkerId(..)"));
|
||||||
|
assert!(!rendered.contains("raw-ingest-mainnet-001"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,136 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/snapshot.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
fn snapshot_foundation() -> std::option::Option<(crate::RawTransactionIngestSnapshotPublisher, crate::RawTransactionIngestSnapshotSource)> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-snapshot-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let settings = crate::RawTransactionIngestSettings::with_defaults(network, worker_id.clone());
|
||||||
|
let kind = match ksp_worker_api::WorkerKindCode::new(crate::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let mut lifecycle = ksp_worker_api::WorkerLifecycle::new(worker_id, kind);
|
||||||
|
if lifecycle.start().is_err() {
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
return std::option::Option::Some(crate::RawTransactionIngestSnapshotPublisher::new(&settings, &lifecycle));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_initial_snapshot_and_common_projection_are_exact() {
|
||||||
|
let (publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let _publisher = publisher;
|
||||||
|
let concrete = source.current();
|
||||||
|
assert_eq!(concrete.worker_snapshot().sequence().value(), 0);
|
||||||
|
assert_eq!(concrete.worker_snapshot().state(), ksp_worker_api::WorkerState::Starting);
|
||||||
|
assert_eq!(concrete.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unknown);
|
||||||
|
assert_eq!(concrete.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Unknown);
|
||||||
|
assert_eq!(concrete.admission_queue_capacity(), crate::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY);
|
||||||
|
assert_eq!(concrete.admission_queue_depth(), 0);
|
||||||
|
assert_eq!(concrete.persistence_concurrency(), crate::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY);
|
||||||
|
assert_eq!(concrete.in_flight_persistence(), 0);
|
||||||
|
assert_eq!(concrete.admitted_total(), 0);
|
||||||
|
assert_eq!(concrete.canonicalized_total(), 0);
|
||||||
|
assert_eq!(concrete.persisted_total(), 0);
|
||||||
|
assert_eq!(concrete.entity_inserted_total(), 0);
|
||||||
|
assert_eq!(concrete.entity_already_present_total(), 0);
|
||||||
|
assert_eq!(concrete.entity_skipped_purged_total(), 0);
|
||||||
|
assert_eq!(concrete.observation_inserted_total(), 0);
|
||||||
|
assert_eq!(concrete.observation_already_present_total(), 0);
|
||||||
|
assert_eq!(concrete.content_conflict_total(), 0);
|
||||||
|
assert_eq!(concrete.store_failure_total(), 0);
|
||||||
|
assert_eq!(concrete.source_failure_total(), 0);
|
||||||
|
assert_eq!(concrete.backpressure_wait_total(), 0);
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::current(&source);
|
||||||
|
assert_eq!(&common, concrete.worker_snapshot());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_checked_counter_exhaustion_is_stable_and_never_wraps() {
|
||||||
|
let result = super::checked_counter(u64::MAX, "admitted_total");
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED);
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].key(), "field");
|
||||||
|
assert_eq!(error.context()[0].value(), "admitted_total");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_008_slow_concrete_and_common_listeners_coalesce_to_same_latest_sequence() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let observed = source.current().worker_snapshot().sequence();
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Running, 0, 0).is_ok());
|
||||||
|
assert!(publisher.record_admission_success(ksp_worker_api::WorkerState::Running, 1, 1, false).is_ok());
|
||||||
|
let concrete = source.wait_for_change(observed).await;
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::wait_for_change(&source, observed).await;
|
||||||
|
assert_eq!(concrete.worker_snapshot().sequence(), common.sequence());
|
||||||
|
assert_eq!(concrete.worker_snapshot().sequence().value(), 2);
|
||||||
|
assert_eq!(concrete.worker_snapshot().state(), ksp_worker_api::WorkerState::Running);
|
||||||
|
assert_eq!(concrete.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(concrete.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Active);
|
||||||
|
assert_eq!(concrete.admission_queue_depth(), 1);
|
||||||
|
assert_eq!(concrete.in_flight_persistence(), 1);
|
||||||
|
assert_eq!(concrete.admitted_total(), 1);
|
||||||
|
assert_eq!(concrete.canonicalized_total(), 1);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_008_terminal_latest_value_is_retained_after_publisher_drop() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Running, 0, 0).is_ok());
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Stopping, 0, 0).is_ok());
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Stopped, 0, 0).is_ok());
|
||||||
|
std::mem::drop(publisher);
|
||||||
|
let retained = source.current();
|
||||||
|
assert_eq!(retained.worker_snapshot().state(), ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(retained.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(retained.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
let same_terminal = source.wait_for_change(retained.worker_snapshot().sequence()).await;
|
||||||
|
assert_eq!(same_terminal, retained);
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::wait_for_change(&source, retained.worker_snapshot().sequence()).await;
|
||||||
|
assert_eq!(common, retained.worker_snapshot().clone());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_009_source_failure_and_backpressure_counters_advance_without_changing_projection_contract() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Running, 0, 0).is_ok());
|
||||||
|
assert!(publisher.record_admission_success(ksp_worker_api::WorkerState::Running, 0, 1, true).is_ok());
|
||||||
|
assert!(publisher.record_source_failure(ksp_worker_api::WorkerState::Running, 0, 1).is_ok());
|
||||||
|
let concrete = source.current();
|
||||||
|
assert_eq!(concrete.admitted_total(), 1);
|
||||||
|
assert_eq!(concrete.canonicalized_total(), 1);
|
||||||
|
assert_eq!(concrete.backpressure_wait_total(), 1);
|
||||||
|
assert_eq!(concrete.source_failure_total(), 1);
|
||||||
|
assert_eq!(concrete.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(concrete.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Active);
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::current(&source);
|
||||||
|
assert_eq!(&common, concrete.worker_snapshot());
|
||||||
|
return;
|
||||||
|
}
|
||||||
393
deltas/0.3.11/pre.001.md
Normal file
393
deltas/0.3.11/pre.001.md
Normal file
@@ -0,0 +1,393 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.001.md -->
|
||||||
|
<!-- version: 2 -->
|
||||||
|
|
||||||
|
# Delta 0.3.11-pre.001 — audit/sizing fondation Worker RawTransaction ingest
|
||||||
|
|
||||||
|
## Base
|
||||||
|
|
||||||
|
```text
|
||||||
|
archive : khadhroony-solana-project-v0.3.10.zip
|
||||||
|
SHA-256 : befafea61304aaea15c94db7c8b3525c58b7c1622bf880a9110373bcc0ae6ac0
|
||||||
|
ZIP bytes : 7993096
|
||||||
|
ZIP entries : 1901
|
||||||
|
unzip -t : PASS
|
||||||
|
workspace members : 20
|
||||||
|
workspace.package.version base : 0.3.10
|
||||||
|
stable delta : deltas/0.3.10/rel.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Le ZIP a une racine unique et a été contrôlé sans entrée absolue, path traversal, symlink, `.git/`, `target/`, `node_modules/`, `Cargo.lock` ou `.env`.
|
||||||
|
|
||||||
|
Préconditions du prompt `030` confirmées :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-raw-transaction-lib présent et documenté
|
||||||
|
ksp-worker-api présent
|
||||||
|
ksp-job-backfill-lib présent et consommateur de ksp-raw-transaction-lib
|
||||||
|
ksp-worker-raw-transaction-ingest-lib absent
|
||||||
|
```
|
||||||
|
|
||||||
|
## Type de livraison
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-general-0.3.11-pre.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le type `general` est requis par le bump `Cargo.toml` de toute prerelease non-fix (`VER-ID-009`). L'archive d'échange contient uniquement les fichiers ajoutés/modifiés par cette tranche et son delta.
|
||||||
|
|
||||||
|
## Baseline exécutée avant modification
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 772 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
L'audit mécanique était vert mais la lecture humaine des règles a trouvé deux divergences normatives actives non détectées par les scripts.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Exécuter intégralement le gate d'ouverture demandé par `prompts/030-V0_3_11_START_PROMPT.md` : lecture/audit de la stable `v0.3.10`, contrôle renforcé des règles, inventaire Worker/Common RAW/Store/Backfill, fraîcheur des dépendances runtime, brainstorming public/private et supervisor, définition du harness déterministe, threat model, sizing et planification, sans commencer l'implémentation lourde du Worker.
|
||||||
|
|
||||||
|
## Réconciliation normative
|
||||||
|
|
||||||
|
### D1–D4
|
||||||
|
|
||||||
|
Défauts hérités trouvés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
KSP-DURABLE-002 : D2 encore nommé Core canonique
|
||||||
|
KSP-DURABLE-006 : instructions encore qualifiées de faits Core
|
||||||
|
DEP-STORE-004 : D2 encore nommé Core canonique
|
||||||
|
```
|
||||||
|
|
||||||
|
Les règles plus récentes et l'architecture `008` imposent déjà :
|
||||||
|
|
||||||
|
```text
|
||||||
|
D1 RAW
|
||||||
|
D2 STRUCTURAL
|
||||||
|
D3 DECODED
|
||||||
|
D4 DOMAIN
|
||||||
|
```
|
||||||
|
|
||||||
|
Correction : normalisation des trois règles actives. Les archives/deltas/plans historiques ne sont pas réécrits.
|
||||||
|
|
||||||
|
### Nom du Worker RAW
|
||||||
|
|
||||||
|
Les règles `KSP-WORKER-005/006/007/010` utilisaient encore :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-retriever
|
||||||
|
```
|
||||||
|
|
||||||
|
Le nom durable retenu par les architectures, ROADMAP, CHANGELOG, handoff `0.3.10` et prompt `030` est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Correction : normalisation du nom dans ces quatre règles, sans changement de leur responsabilité fonctionnelle.
|
||||||
|
|
||||||
|
Un scan des règles actives après correction ne trouve plus `D2 Core canonique`, `faits Core distincts` ni `ksp-worker-raw-retriever`.
|
||||||
|
|
||||||
|
## Inventaire et décisions de fondation
|
||||||
|
|
||||||
|
### Frontières préservées
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-api reste runtime-neutral et générique
|
||||||
|
ksp-raw-transaction-lib reste source-neutral sans runtime/Transport/Config/Worker/Job/backend
|
||||||
|
Worker persistera via ksp-store-lib default-features=false
|
||||||
|
Store reste vérité de convergence (network, signature)
|
||||||
|
Backfill et Worker restent producteurs indépendants
|
||||||
|
aucun paramètre historique métier au start Worker
|
||||||
|
aucun edge Worker -> Config/Job/backend physique
|
||||||
|
```
|
||||||
|
|
||||||
|
### Surface publique minimale retenue
|
||||||
|
|
||||||
|
```text
|
||||||
|
RAW_TRANSACTION_INGEST_WORKER_KIND_CODE
|
||||||
|
RawTransactionIngestSettings
|
||||||
|
RawTransactionIngestWorker
|
||||||
|
RawTransactionIngestHandle
|
||||||
|
RawTransactionIngestSnapshot
|
||||||
|
RawTransactionIngestSnapshotSource / future associée
|
||||||
|
ErrorCodes statiques Worker
|
||||||
|
```
|
||||||
|
|
||||||
|
Settings de fondation :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network
|
||||||
|
worker_id
|
||||||
|
admission_queue_capacity
|
||||||
|
persistence_concurrency
|
||||||
|
shutdown_drain_timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
Bornes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission queue : 1..=65_536, default 256
|
||||||
|
persistence concurrency : 1..=64, default 8
|
||||||
|
shutdown drain : 100 ms..=30 s, default 5 s
|
||||||
|
```
|
||||||
|
|
||||||
|
### Types reportés
|
||||||
|
|
||||||
|
Ne sont pas figés publiquement avant une vraie source :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionSourceId
|
||||||
|
SourceSettings
|
||||||
|
SourceCapability
|
||||||
|
SourceRole
|
||||||
|
ContinuityState
|
||||||
|
run/source frontiers
|
||||||
|
provider-specific settings
|
||||||
|
retry/replay/gap repair
|
||||||
|
```
|
||||||
|
|
||||||
|
### Runtime
|
||||||
|
|
||||||
|
```text
|
||||||
|
caller-owned active Tokio runtime
|
||||||
|
Worker-owned private supervisor/tasks
|
||||||
|
JoinHandle/JoinSet privés
|
||||||
|
start(settings, Arc<Store>) -> Handle
|
||||||
|
request_stop idempotent
|
||||||
|
wait_terminal async sans Tokio nominal public
|
||||||
|
mpsc admission borné
|
||||||
|
watch stop privé
|
||||||
|
watch snapshot latest-value
|
||||||
|
aucun unbounded channel
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pipeline
|
||||||
|
|
||||||
|
```text
|
||||||
|
private deterministic ingress
|
||||||
|
-> ksp-raw-transaction-lib canonicalization
|
||||||
|
-> Worker-owned deterministic observation key/provenance
|
||||||
|
-> common RawTransactionAcquisition
|
||||||
|
-> ksp-store-lib atomic Normal persistence
|
||||||
|
-> explicit outcomes/counters
|
||||||
|
-> concrete/common latest-value snapshot
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun cache mémoire n'est utilisé comme vérité de correction ; Store décide new/idempotent/conflict.
|
||||||
|
|
||||||
|
### Fault/shutdown
|
||||||
|
|
||||||
|
```text
|
||||||
|
content conflict -> terminal fault, no overwrite, no winner
|
||||||
|
Store error -> terminal safe fault
|
||||||
|
source harness failure -> terminal safe fault
|
||||||
|
stop -> close new admissions + bounded drain admitted work
|
||||||
|
expiry drain -> abort private remaining tasks + join + terminal drain_timeout
|
||||||
|
terminal publié seulement après absence de tâche privée survivante
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fraîcheur externe
|
||||||
|
|
||||||
|
Vérifié le 8 septembre 2026 :
|
||||||
|
|
||||||
|
```text
|
||||||
|
tokio latest stable = 1.53.1 ; workspace = ^1.53
|
||||||
|
futures-util latest stable = 0.3.34 ; workspace = ^0.3
|
||||||
|
sha2 latest stable = 0.11.0 ; workspace = ^0.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun bump workspace n'est requis.
|
||||||
|
|
||||||
|
Dépendances prévues pour la crate foundation lorsqu'elle est créée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-core-lib
|
||||||
|
ksp-worker-api
|
||||||
|
ksp-raw-transaction-lib
|
||||||
|
ksp-store-lib default-features=false
|
||||||
|
ksp-logging-lib
|
||||||
|
sha2
|
||||||
|
tokio features macros,rt,sync,time
|
||||||
|
```
|
||||||
|
|
||||||
|
Non retenus en fondation :
|
||||||
|
|
||||||
|
```text
|
||||||
|
futures-util : pas d'usage nécessaire après choix JoinSet/mpsc/watch/select!
|
||||||
|
ksp-onchain-transport-lib : aucune source live en 0.3.11
|
||||||
|
ksp-config-lib : edge interdit
|
||||||
|
SDK provider : interdit/inutile
|
||||||
|
```
|
||||||
|
|
||||||
|
## Harness et threat model
|
||||||
|
|
||||||
|
Le plan `032` fixe un harness privé sans réseau couvrant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
start/stop
|
||||||
|
source failure
|
||||||
|
queue saturation/backpressure
|
||||||
|
bounded persistence concurrency
|
||||||
|
new/idempotent/new observation
|
||||||
|
content conflict
|
||||||
|
Store failure
|
||||||
|
stop pendant admission/persistence
|
||||||
|
drain timeout
|
||||||
|
no orphan tasks
|
||||||
|
slow/no snapshot listener
|
||||||
|
latest-value concrete/common
|
||||||
|
redaction et absence de surface Job/Config/provider
|
||||||
|
```
|
||||||
|
|
||||||
|
## Sizing recalibré
|
||||||
|
|
||||||
|
Le forecast initial est scindé pour respecter le budget de 15–20 minutes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
pre.002 crate + dependency firewall
|
||||||
|
pre.003 identity/settings
|
||||||
|
pre.004 lifecycle/start/handle/terminal
|
||||||
|
pre.005 supervisor/task ownership + harness
|
||||||
|
pre.006 bounded admission + common RAW
|
||||||
|
pre.007 Store persistence/idempotence/conflict
|
||||||
|
pre.008 snapshots + Worker API projection
|
||||||
|
pre.009 shutdown/backpressure/fault hardening
|
||||||
|
pre.010 public/release/security hardening
|
||||||
|
pre.011 gate technique final
|
||||||
|
pre.012 réconciliation documentaire
|
||||||
|
pre.013 préparation publication
|
||||||
|
rel.001
|
||||||
|
```
|
||||||
|
|
||||||
|
Décision : `0.3.11` reste clôturable dans une seule session **à condition** de ne pas introduire source live, Transport productif, Config, migration Store ou continuity/gap repair. Toute tranche réellement trop grosse est scindée avant exécution.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
deltas/0.3.11/pre.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/rules/RULES_KSP.md
|
||||||
|
docs/rules/RULES_DEPENDENCIES.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Versions :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml : 505 -> 506
|
||||||
|
workspace.package.version : 0.3.10 -> 0.3.11-pre.1
|
||||||
|
RULES_KSP.md : 40 -> 41
|
||||||
|
RULES_DEPENDENCIES.md : 18 -> 19
|
||||||
|
```
|
||||||
|
|
||||||
|
Le bump Cargo est requis par `VER-ID-009` pour une prerelease non-fix, même sans modification Rust.
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation après modification
|
||||||
|
|
||||||
|
Audits statiques réellement exécutés sur le delta complet :
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 775 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
Un scan sémantique complémentaire des règles actives confirme l'absence des trois formulations obsolètes corrigées. La comparaison byte-à-byte contre la stable confirme exactement 3 fichiers ajoutés, 3 fichiers modifiés et aucun fichier supprimé.
|
||||||
|
|
||||||
|
## Preuve opérateur stable distincte
|
||||||
|
|
||||||
|
Le journal opérateur fourni pour `0.3.10` montre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check : sans erreur
|
||||||
|
cargo check --workspace : sans erreur
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings : sans erreur
|
||||||
|
ksp-raw-transaction-lib : 26 tests verts
|
||||||
|
ksp-job-backfill-lib : 77 tests verts
|
||||||
|
cargo tree normal des deux crates : fourni
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette preuve concerne la stable d'entrée ; elle ne devient pas une exécution locale de `0.3.11-pre.001`.
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test --workspace --all-targets --all-features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
Motif : `cargo` n'est pas installé dans l'environnement d'assemblage. `pre.001` ne contient aucun Rust Worker, mais l'opérateur doit rejouer les gates applicables après application du delta.
|
||||||
|
|
||||||
|
## Questions fermées
|
||||||
|
|
||||||
|
```text
|
||||||
|
settings techniques et bornes V1
|
||||||
|
public/private surface de fondation
|
||||||
|
absence de SourceCapability/Role publics avant 0.3.12
|
||||||
|
start + Arc<Store> + handle
|
||||||
|
terminal future runtime-neutral
|
||||||
|
supervisor/task ownership privé
|
||||||
|
mpsc/watch exacts
|
||||||
|
harness déterministe sans réseau
|
||||||
|
absence de frontier/timestamp public en fondation
|
||||||
|
Store conflict/failure policy
|
||||||
|
Tokio direct + features minimales retenues
|
||||||
|
absence de futures-util/Transport en foundation
|
||||||
|
sizing et maintien de la release
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions reportées
|
||||||
|
|
||||||
|
```text
|
||||||
|
premier contrat public SourceId/capability/role
|
||||||
|
Yellowstone source settings/adapters
|
||||||
|
source required/optional policy
|
||||||
|
retry/reconnect
|
||||||
|
run frontier/continuity
|
||||||
|
gap repair
|
||||||
|
hydration productive
|
||||||
|
WS/Helius/HTTP live
|
||||||
|
multi-provider coalescence
|
||||||
|
hot reconfiguration desired/effective
|
||||||
|
provider smokes
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune question reportée ne bloque `pre.002`.
|
||||||
|
|
||||||
|
## Non inclus
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune nouvelle crate Worker créée
|
||||||
|
aucun Rust modifié
|
||||||
|
aucun Transport/Config/Store code modifié
|
||||||
|
aucune migration
|
||||||
|
aucune source live
|
||||||
|
aucun CHANGELOG/ROADMAP modifié
|
||||||
|
aucun prompt 0.3.12 produit
|
||||||
|
```
|
||||||
164
deltas/0.3.11/pre.002.md
Normal file
164
deltas/0.3.11/pre.002.md
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.002` — squelette Worker RAW + dependency firewall
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.001
|
||||||
|
workspace.package.version = 0.3.11-pre.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour `pre.001` est vert sur `cargo fmt --all`, les audits Rust/Markdown, `cargo check --workspace` et Clippy workspace strict `--all-targets --all-features -- -D warnings`. Aucun `cargo test` supplémentaire n'a été communiqué pour ce gate et aucun n'est inventé.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Créer uniquement le squelette de `ksp-worker-raw-transaction-ingest-lib`, l'inscrire au workspace et verrouiller son graphe de dépendances. Aucun settings, identity, `start`, task, channel, Store write, canonicalisation ou snapshot runtime n'est matérialisé dans cette tranche.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Cette tranche est une prerelease non-fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.11-pre.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le `Cargo.toml` racine passe du header `506` au header `507` et le workspace passe de 20 à 21 membres.
|
||||||
|
|
||||||
|
## Graphe de production exact
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
-> ksp-core-lib
|
||||||
|
-> ksp-logging-lib
|
||||||
|
-> ksp-raw-transaction-lib
|
||||||
|
-> ksp-store-lib default-features=false
|
||||||
|
-> ksp-worker-api
|
||||||
|
-> sha2
|
||||||
|
-> tokio features=macros,rt,sync,time
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces edges matérialisent uniquement la frontière décidée en `pre.001`. Le code runtime qui les consommera reste réservé à ses tranches dédiées.
|
||||||
|
|
||||||
|
## Firewall explicite
|
||||||
|
|
||||||
|
Absents du manifest de production :
|
||||||
|
|
||||||
|
```text
|
||||||
|
futures-util
|
||||||
|
ksp-config-lib
|
||||||
|
ksp-interface-lib
|
||||||
|
ksp-job-api
|
||||||
|
ksp-job-backfill-lib
|
||||||
|
ksp-onchain-transport-lib
|
||||||
|
ksp-program-api
|
||||||
|
ksp-store-api direct
|
||||||
|
ksp-store-postgres-lib
|
||||||
|
ksp-wallet-lib
|
||||||
|
reqwest
|
||||||
|
Solana protocol SDK direct
|
||||||
|
tokio-tungstenite
|
||||||
|
tonic
|
||||||
|
yellowstone-grpc-proto
|
||||||
|
```
|
||||||
|
|
||||||
|
Il n'existe ni `[dev-dependencies]`, ni `[build-dependencies]` dans cette tranche.
|
||||||
|
|
||||||
|
## Crate-root volontairement vide de comportement
|
||||||
|
|
||||||
|
`src/lib.rs` contient uniquement les lints workspace et la rustdoc de responsabilité. Il n'expose encore :
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun pub mod
|
||||||
|
aucun type public
|
||||||
|
aucune fonction publique
|
||||||
|
aucun start/spawn
|
||||||
|
aucun appel Tokio
|
||||||
|
aucun appel Store/common RAW/Worker API
|
||||||
|
```
|
||||||
|
|
||||||
|
Le test externe `dependency_boundary.rs` verrouille le manifest exact, les edges interdits et l'absence de comportement runtime prématuré au crate-root.
|
||||||
|
|
||||||
|
## Documentation volontairement différée
|
||||||
|
|
||||||
|
`README.md` et `USAGE.md` ne sont pas créés à ce stade car la crate ne possède encore aucune API consommable. Leur réconciliation version-neutral reste réservée à `pre.012`, après freeze de la fondation runtime.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
deltas/0.3.11/pre.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 776 file(s))
|
||||||
|
|
||||||
|
contrôle TOML/statique : workspace 0.3.11-pre.2, 21 membres, manifest Worker exact, Store sans default features, Tokio macros/rt/sync/time
|
||||||
|
comparaison byte-à-byte avec pre.001 : 4 fichiers ajoutés, 3 modifiés, 0 supprimé
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions conservées
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune source live dans 0.3.11
|
||||||
|
aucun Transport productif avant 0.3.12
|
||||||
|
Store uniquement via ksp-store-lib default-features=false
|
||||||
|
aucun backend Store direct
|
||||||
|
aucun Job/Backfill/Config dans le Worker
|
||||||
|
futures-util absent tant qu'un usage réel n'est pas démontré
|
||||||
|
Tokio direct parce que le Worker concret possédera ses tâches runtime
|
||||||
|
aucune API publique avant la tranche qui en possède le contrat
|
||||||
|
```
|
||||||
|
|
||||||
|
## Suite
|
||||||
|
|
||||||
|
Après gate opérateur vert, `pre.003` introduit uniquement l'identité Worker concrète et `RawTransactionIngestSettings` avec defaults, bornes et validation. Aucun spawn n'y est autorisé.
|
||||||
100
deltas/0.3.11/pre.003-fix.001.md
Normal file
100
deltas/0.3.11/pre.003-fix.001.md
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.003-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.003-fix.001` — propagation d'erreur explicite des settings
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.003
|
||||||
|
workspace.package.version = 0.3.11-pre.3
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent, mais `cargo clippy --workspace --all-targets --all-features -- -D warnings` rejette deux usages de l'opérateur `?` dans `RawTransactionIngestSettings::new`.
|
||||||
|
|
||||||
|
Diagnostics :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs:42:9
|
||||||
|
clippy::question_mark_used
|
||||||
|
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs:48:9
|
||||||
|
clippy::question_mark_used
|
||||||
|
```
|
||||||
|
|
||||||
|
La règle normative `RUST-ERR-002` interdit également l'opérateur `?` en code de production. Les tests de la crate communiqués dans le même journal restent verts : 4 tests unitaires, 3 tests dependency-boundary et 3 tests public API, sans échec.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger strictement `pre.003` sans modifier son contrat fonctionnel : remplacer les deux propagations `?` par un contrôle de flux explicite retournant exactement la même erreur. Aucun type, borne, default, code d'erreur, dépendance, test, lifecycle ou comportement runtime n'est ajouté.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de production. Conformément à `VER-ID-007` et `VER-ID-010` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.003-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.3.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Dans `RawTransactionIngestSettings::new`, les validations de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission_queue_capacity
|
||||||
|
persistence_concurrency
|
||||||
|
```
|
||||||
|
|
||||||
|
utilisent désormais `if let Result::Err(error)` puis `return Err(error)`. La validation de `shutdown_drain_timeout` était déjà explicite et reste inchangée.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.003-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : aucun opérateur `?` restant dans settings.rs
|
||||||
|
scan statique : aucune modification de surface publique/settings hors propagation d'erreur
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.004` reste bloquée jusqu'à validation opérateur verte de ce fix. Aucun périmètre de `pre.003` n'est redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
185
deltas/0.3.11/pre.003.md
Normal file
185
deltas/0.3.11/pre.003.md
Normal file
@@ -0,0 +1,185 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.003.md -->
|
||||||
|
<!-- version: 3 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.003` — identité Worker + settings bornés
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.002
|
||||||
|
workspace.package.version = 0.3.11-pre.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour `pre.002` est entièrement vert : `cargo fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, tests de la crate et arbres Cargo normal/features. Les 3 tests `pre.002` passent et le firewall de dépendances reste conforme.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement l'identité de la verticale Worker RAW transaction ingest et ses settings runtime techniques bornés. Aucun lifecycle, `start`, spawn, task, channel, supervisor, persistence Store, canonicalisation ou source live n'est introduit.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Cette tranche est une prerelease non-fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.11-pre.3
|
||||||
|
```
|
||||||
|
|
||||||
|
## Identité Worker
|
||||||
|
|
||||||
|
La verticale possède le code stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RAW_TRANSACTION_INGEST_WORKER_KIND_CODE = "raw_transaction_ingest"
|
||||||
|
```
|
||||||
|
|
||||||
|
La transformation de ce texte en `WorkerKindCode` reste sous validation de `ksp-worker-api`; aucun second validateur d'identité n'est créé dans la crate concrète.
|
||||||
|
|
||||||
|
## `RawTransactionIngestSettings`
|
||||||
|
|
||||||
|
Surface exacte :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network: ksp_store_lib::RawNetworkId
|
||||||
|
worker_id: ksp_worker_api::WorkerId
|
||||||
|
admission_queue_capacity: usize
|
||||||
|
persistence_concurrency: usize
|
||||||
|
shutdown_drain_timeout: Duration
|
||||||
|
```
|
||||||
|
|
||||||
|
Les champs sont privés et exposés par getters. Deux constructions existent : paramètres explicites bornés et defaults V1 sur des identités déjà validées.
|
||||||
|
|
||||||
|
Bornes exactes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission queue : 1..=65_536, default 256
|
||||||
|
persistence concurrency : 1..=64, default 8
|
||||||
|
shutdown drain : 100 ms..=30 s, default 5 s
|
||||||
|
```
|
||||||
|
|
||||||
|
Les bornes sont inclusives. Aucun champ historique, provider-specific, retry, replay, gap repair, source capability ou frontier n'est ajouté.
|
||||||
|
|
||||||
|
## Erreur settings
|
||||||
|
|
||||||
|
La crate ouvre un seul code propre à cette tranche :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.settings_invalid
|
||||||
|
```
|
||||||
|
|
||||||
|
Il couvre uniquement les limites techniques possédées par le Worker concret. Le contexte d'erreur contient seulement le nom statique du champ invalide :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission_queue_capacity
|
||||||
|
persistence_concurrency
|
||||||
|
shutdown_drain_timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune valeur caller-supplied n'est copiée dans le diagnostic.
|
||||||
|
|
||||||
|
Les erreurs de `RawNetworkId`, `WorkerId` et `WorkerKindCode` restent celles de leurs crates propriétaires.
|
||||||
|
|
||||||
|
## Redaction
|
||||||
|
|
||||||
|
`Debug` du settings rend le network logique mais conserve le `WorkerId(..)` redacted fourni par `ksp-worker-api`; le texte réel du Worker ID n'est pas rendu.
|
||||||
|
|
||||||
|
## Frontière runtime maintenue
|
||||||
|
|
||||||
|
Toujours absents :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestWorker
|
||||||
|
start/spawn
|
||||||
|
JoinHandle/JoinSet
|
||||||
|
mpsc/watch
|
||||||
|
supervisor
|
||||||
|
Store write
|
||||||
|
common RAW conversion
|
||||||
|
Transport
|
||||||
|
source live/provider SDK
|
||||||
|
```
|
||||||
|
|
||||||
|
Le manifest de production est inchangé depuis `pre.002`.
|
||||||
|
|
||||||
|
## Tests ajoutés/étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
unit_tests/settings.rs
|
||||||
|
defaults exacts
|
||||||
|
bornes min/max inclusives
|
||||||
|
six cas hors bornes
|
||||||
|
code/contexte d'erreur stable
|
||||||
|
redaction Debug du WorkerId
|
||||||
|
|
||||||
|
tests/public_api.rs
|
||||||
|
kind + settings consommables depuis crate root
|
||||||
|
validation network/Worker identity reste lower-layer-owned
|
||||||
|
erreur de borne publique stable et sans echo d'identité
|
||||||
|
|
||||||
|
tests/dependency_boundary.rs
|
||||||
|
firewall pre.002 conservé
|
||||||
|
crate-root pre.003 limité à identity/settings
|
||||||
|
absence de comportement runtime prématuré
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/identity.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/settings.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
deltas/0.3.11/pre.003.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Premier passage : quatre écarts de forme détectés par l'audit Rust (rustdoc `pub(crate)`, ordre de constantes, ligne vide de fonction), puis un espacement Markdown documentaire détecté après ajout de la validation. Ils ont tous été corrigés avant livraison.
|
||||||
|
|
||||||
|
Gate final local :
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 777 file(s))
|
||||||
|
|
||||||
|
contrôle statique runtime-boundary : clean
|
||||||
|
comparaison byte-à-byte avec pre.002 : 6 fichiers ajoutés, 5 modifiés, 0 supprimé
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Suite
|
||||||
|
|
||||||
|
Après gate opérateur vert, `pre.004` introduit le `start` caller-runtime-owned, le handle concret, le lifecycle/terminal et le stop idempotent. Aucun supervisor complet ni persistence réelle ne doit être anticipé avant leurs tranches dédiées.
|
||||||
135
deltas/0.3.11/pre.004.md
Normal file
135
deltas/0.3.11/pre.004.md
Normal file
@@ -0,0 +1,135 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.004.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.004` — start, handle, lifecycle et terminal Worker
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.003-fix.001
|
||||||
|
workspace.package.version = 0.3.11-pre.3.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert après le fix `pre.003-fix.001` sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict et les dix tests de la crate Worker.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.004` du plan `032` : start synchrone sur runtime Tokio caller-owned, handle clonable, stop idempotent, lifecycle générique `ksp-worker-api` et future terminal boxed, sans ouvrir le supervisor de tâches enfants, l'admission, la canonicalisation, la persistence, les snapshots concrets ou une source live.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.004
|
||||||
|
workspace.package.version : 0.3.11-pre.4
|
||||||
|
```
|
||||||
|
|
||||||
|
## Contrat public ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID
|
||||||
|
RawTransactionIngestWorker
|
||||||
|
RawTransactionIngestHandle
|
||||||
|
RawTransactionIngestTerminalFuture
|
||||||
|
RawTransactionIngestWorker::start(settings, Arc<Store>)
|
||||||
|
RawTransactionIngestHandle::request_stop()
|
||||||
|
RawTransactionIngestHandle::wait_terminal()
|
||||||
|
```
|
||||||
|
|
||||||
|
`start` exige un runtime Tokio courant, vérifie le réseau du Store avant spawn et conserve le Store par `Arc` sans le fermer ni l'utiliser encore pour une écriture.
|
||||||
|
|
||||||
|
`request_stop()` partage `WorkerStopToken` et retourne `true` uniquement lors de la première demande acceptée. Le task racine est réveillé par un `watch<bool>` privé ; la disparition de tous les handles ferme ce channel et provoque également une terminaison coopérative.
|
||||||
|
|
||||||
|
`wait_terminal()` retourne une future boxed `Send`. Elle attend un `WorkerState` terminal puis la fermeture du sender d'état par le task racine ; aucun `JoinHandle` public n'est nécessaire.
|
||||||
|
|
||||||
|
## Lifecycle de fondation
|
||||||
|
|
||||||
|
```text
|
||||||
|
Starting -> Running -> Stopping -> Stopped
|
||||||
|
Starting -> Stopping -> Stopped si stop immédiat
|
||||||
|
```
|
||||||
|
|
||||||
|
Les transitions utilisent `WorkerLifecycle`. Un invariant impossible devient `Faulted(worker_raw_transaction_ingest.runtime_invalid)` sans payload, worker id, réseau ou erreur externe dans le diagnostic.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
JoinSet / tasks enfants
|
||||||
|
mpsc / admission
|
||||||
|
canonicalisation common RAW
|
||||||
|
observation key
|
||||||
|
persistence Store
|
||||||
|
snapshots concrets / WorkerSnapshotSource
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
```
|
||||||
|
|
||||||
|
Les méthodes de snapshot prévues par la surface finale restent différées à `pre.008`. `pre.005` introduit le supervisor et l'ownership des tâches enfants autour du wake-up de stop déjà minimalement nécessaire ici.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
Les tests couvrent statiquement ou via harness runtime privé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
runtime Tokio courant obligatoire
|
||||||
|
network Store mismatch sûr
|
||||||
|
stop immédiat idempotent
|
||||||
|
Running -> stop via clone de handle
|
||||||
|
drop du dernier handle -> sortie du task racine
|
||||||
|
surface publique start/handle/terminal future
|
||||||
|
absence de JoinHandle public et du scope pre.005+
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
deltas/0.3.11/pre.004.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique des interdits/scope runtime
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.004` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.005` reste bloquée jusqu'à validation opérateur verte de cette tranche.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Le supervisor/tasks enfants, l'admission, la persistence et les snapshots restent dans leurs tranches planifiées.
|
||||||
125
deltas/0.3.11/pre.005.md
Normal file
125
deltas/0.3.11/pre.005.md
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.005.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.005` — supervisor privé et ownership des tâches
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.004
|
||||||
|
workspace.package.version = 0.3.11-pre.4
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, les 16 tests de la crate Worker, doc-tests et les arbres Cargo normal/features.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.005` du plan `032` : supervisor privé, `JoinSet` privé, ownership et join des tâches enfants, intégration du stop au seam source déterministe et preuve qu'aucune tâche enfant coopérative ne survit au terminal. Aucun admission `mpsc`, aucune canonicalisation RAW et aucune persistence Store réelle.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.005
|
||||||
|
workspace.package.version : 0.3.11-pre.5
|
||||||
|
```
|
||||||
|
|
||||||
|
## Runtime privé ajouté
|
||||||
|
|
||||||
|
Le task racine devient le supervisor du run. Il possède un `tokio::task::JoinSet<()>` privé et un seam privé de spawn de tâches source.
|
||||||
|
|
||||||
|
La production appelle ce seam sans source. Les tests injectent des tâches déterministes qui reçoivent un clone du `watch<bool>` de stop.
|
||||||
|
|
||||||
|
Le supervisor :
|
||||||
|
|
||||||
|
```text
|
||||||
|
reap des enfants terminés via join_next()
|
||||||
|
stop prioritaire via select! biased
|
||||||
|
join de tous les enfants restants après stop/fermeture du contrôle
|
||||||
|
Stopped publié uniquement après join complet
|
||||||
|
JoinError -> supervision non clean ; Faulted à la fermeture de supervision
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun `JoinHandle` ou `JoinSet` n'est ajouté à la surface publique.
|
||||||
|
|
||||||
|
## Arrêt coopératif
|
||||||
|
|
||||||
|
Le timeout de drain configuré existe déjà dans les settings mais n'est pas encore appliqué ici. `pre.005` attend coopérativement les enfants ; le timeout, `abort_all` et les races stop/fault sont réservés au hardening `pre.009`.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
mpsc / admission
|
||||||
|
PrivateRawTransactionIngress
|
||||||
|
canonicalisation common RAW
|
||||||
|
observation key
|
||||||
|
persistence Store
|
||||||
|
snapshots concrets / WorkerSnapshotSource
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
abort/timeout de drain
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
supervisor joint trois tâches source coopératives avant Stopped
|
||||||
|
compteur d'enfants actifs == 0 au terminal
|
||||||
|
reap d'une source terminée pendant Running
|
||||||
|
join d'une source encore active lors du stop
|
||||||
|
canary statique : JoinSet présent mais privé
|
||||||
|
canary statique : aucun scope pre.006+ ouvert
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.005.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique des interdits/scope runtime
|
||||||
|
Markdown table audit: clean (340 table(s), 780 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.005` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.006` reste bloquée jusqu'à validation opérateur verte de cette tranche.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. L'admission bornée et la canonicalisation common RAW restent dans `pre.006`, la persistence dans `pre.007`, les snapshots dans `pre.008` et le drain forcé/races dans `pre.009`.
|
||||||
97
deltas/0.3.11/pre.006-fix.001.md
Normal file
97
deltas/0.3.11/pre.006-fix.001.md
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.006-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.006-fix.001` — correction syntaxique du test `tokio::select!`
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.006
|
||||||
|
workspace.package.version = 0.3.11-pre.6
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent, mais que `cargo clippy --workspace --all-targets --all-features -- -D warnings` et `cargo test -p ksp-worker-raw-transaction-ingest-lib` ne peuvent compiler le test `unit_tests/admission.rs`.
|
||||||
|
|
||||||
|
Diagnostic exact :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/../unit_tests/admission.rs:152:13
|
||||||
|
error: no rules expected `result`
|
||||||
|
```
|
||||||
|
|
||||||
|
La cause est syntaxique : deux branches adjacentes du `tokio::select!` du test de préemption du sender bloqué n'étaient pas séparées par des virgules.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger strictement le test de `pre.006` sans modifier le contrat fonctionnel, le runtime de production, l'admission, la canonicalisation Common RAW, l'observation key, le golden ou les dépendances.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.006-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.6.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Le `tokio::select!` du test `pre_006_bounded_sender_is_preempted_by_stop_while_queue_is_full` sépare désormais explicitement ses branches :
|
||||||
|
|
||||||
|
```text
|
||||||
|
_changed = blocked_stop.changed() => false,
|
||||||
|
result = blocked_sender.send(blocked_ingress) => result.is_ok(),
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun comportement de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.006-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : le `tokio::select!` fautif contient deux branches séparées par des virgules
|
||||||
|
scan statique : aucun fichier de production de la crate Worker n'est modifié
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.007` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre de `pre.006` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
180
deltas/0.3.11/pre.006.md
Normal file
180
deltas/0.3.11/pre.006.md
Normal file
@@ -0,0 +1,180 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.006.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.006` — admission bornée et canonicalisation Common RAW
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.005
|
||||||
|
workspace.package.version = 0.3.11-pre.5
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, les 18 tests de la crate Worker, doc-tests et les arbres Cargo normal/features.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.006` du plan `032` : admission `mpsc` centrale bornée, backpressure async, ingress privé, canonicalisation via `ksp-raw-transaction-lib`, observation key Worker déterministe et assembly du `RawTransactionAcquisition`. Aucune persistence Store et aucune source réseau.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.006
|
||||||
|
workspace.package.version : 0.3.11-pre.6
|
||||||
|
```
|
||||||
|
|
||||||
|
## Admission privée bornée
|
||||||
|
|
||||||
|
Le supervisor crée un seul channel central dimensionné par `RawTransactionIngestSettings::admission_queue_capacity()` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
tokio::sync::mpsc::channel(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le seam source reçoit le sender `mpsc` borné et le signal de stop. Le contrat prouvé est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
send().await pour le backpressure
|
||||||
|
select! biased : stop avant send dans le harness source
|
||||||
|
aucun unbounded_channel
|
||||||
|
receiver close au shutdown
|
||||||
|
```
|
||||||
|
|
||||||
|
Au shutdown, le receiver ferme les nouvelles admissions et consume les entrées déjà en queue avant le join final des tâches enfants.
|
||||||
|
|
||||||
|
## Ingress et Common RAW
|
||||||
|
|
||||||
|
L'ingress crate-private, absent de l'API publique, transporte :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionMaterial
|
||||||
|
RawNetworkId attendu
|
||||||
|
RawAcquisitionProvenance
|
||||||
|
source key opaque [u8; 32]
|
||||||
|
```
|
||||||
|
|
||||||
|
Le pipeline est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network guard
|
||||||
|
ksp_raw_transaction_lib::canonicalize_raw_transaction(...)
|
||||||
|
material network guard
|
||||||
|
observation key Worker
|
||||||
|
ksp_raw_transaction_lib::assemble_raw_transaction_acquisition(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune canonicalisation RAW v1 n'est réimplémentée dans le Worker.
|
||||||
|
|
||||||
|
## Observation key V1
|
||||||
|
|
||||||
|
Domaine exact :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp.raw_transaction_ingest.observation.v1\0
|
||||||
|
```
|
||||||
|
|
||||||
|
Le SHA-256 encode avec longueur `u64` big-endian :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network
|
||||||
|
signature
|
||||||
|
source key
|
||||||
|
```
|
||||||
|
|
||||||
|
Golden harness :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network : mainnet
|
||||||
|
signature : 64 x 0x07
|
||||||
|
source key : 32 x 0x09
|
||||||
|
SHA-256 : f8d161dca8a8a72a0da110fc87882ee7136848461c6aa32b09e30a6188b95dcc
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun timestamp local, run id ou ordre d'admission ne participe à la clé.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
capacity=1 : second send bloqué jusqu'à consommation d'un slot
|
||||||
|
stop : sender bloqué rejeté sans admission post-stop
|
||||||
|
payload common RAW canonique exact
|
||||||
|
format id/version exacts
|
||||||
|
observation-key golden exact
|
||||||
|
assembly transaction/provenance/reference exact
|
||||||
|
mismatch network ingress/material rejetés sans fuite de valeurs
|
||||||
|
dependency boundary : mpsc/common RAW requis, persistence/Transport/snapshot toujours interdits
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
deltas/0.3.11/pre.006.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionWrite / persistence Store
|
||||||
|
RawTransactionAcquisitionMode::Normal
|
||||||
|
persistence concurrency
|
||||||
|
Store outcome mapping / idempotence / conflict
|
||||||
|
snapshots concrets
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
retry / reconnect / gap repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 table(s), 781 file(s))
|
||||||
|
|
||||||
|
scan production : aucun ?, unwrap, expect, panic, unbounded_channel
|
||||||
|
scan scope : aucune persistence Store, aucun Transport, aucun snapshot concret
|
||||||
|
pipeline : exactement 1 mpsc::channel, 1 canonicalize_raw_transaction, 1 assemble_raw_transaction_acquisition
|
||||||
|
observation golden indépendant : f8d161dca8a8a72a0da110fc87882ee7136848461c6aa32b09e30a6188b95dcc
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.006` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.007` reste bloquée jusqu'à validation opérateur verte de l'admission bornée, du backpressure, du golden observation et de l'assembly Common RAW.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. La persistence réelle et les outcomes Store restent dans `pre.007`, les snapshots dans `pre.008`, et le hardening fault/drain dans `pre.009`.
|
||||||
161
deltas/0.3.11/pre.007.md
Normal file
161
deltas/0.3.11/pre.007.md
Normal file
@@ -0,0 +1,161 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.007.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.007` — persistence Store bornée et outcomes déterministes
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.006-fix.001
|
||||||
|
workspace.package.version = 0.3.11-pre.6.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, les 22 tests de la crate Worker et les doc-tests. Aucun `cargo tree` n'a été rejoué après le fix, celui-ci n'ayant modifié aucune dépendance ni feature.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.007` du plan `032` : persistence atomique via la façade Store, mode `Normal`, concurrence bornée, classification entity/observation, idempotence/purge et terminaux dédiés content conflict/Store failure. Aucun snapshot concret et aucune source réseau.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.007
|
||||||
|
workspace.package.version : 0.3.11-pre.7
|
||||||
|
```
|
||||||
|
|
||||||
|
## Port Store privé
|
||||||
|
|
||||||
|
La crate ajoute un port `RawTransactionIngestPersistencePort` crate-private, implémenté par `ksp_store_lib::Store` et utilisé exclusivement avec :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionWrite::persist_raw_transaction_acquisition(..., RawTransactionAcquisitionMode::Normal)
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun backend Store concret n'entre dans le graphe ou dans le code Worker.
|
||||||
|
|
||||||
|
## Outcomes
|
||||||
|
|
||||||
|
Classification exacte :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Inserted + Inserted -> Inserted / Inserted
|
||||||
|
AlreadyPresent + Inserted -> AlreadyPresent / Inserted
|
||||||
|
AlreadyPresent + AlreadyPresent -> AlreadyPresent / AlreadyPresent
|
||||||
|
SkippedPurged + NotRecorded -> SkippedPurged / NotRecorded
|
||||||
|
```
|
||||||
|
|
||||||
|
`Rehydrated` et les couples impossibles deviennent `runtime_invalid`.
|
||||||
|
|
||||||
|
## Faults Store
|
||||||
|
|
||||||
|
Deux codes publics deviennent actifs :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.content_conflict
|
||||||
|
worker_raw_transaction_ingest.store_failed
|
||||||
|
```
|
||||||
|
|
||||||
|
Le conflit Store canonique est projeté sur `content_conflict`. Toute autre erreur Store est réduite à `store_failed` en conservant uniquement son domain/code sûr, jamais son texte arbitraire.
|
||||||
|
|
||||||
|
## Concurrence bornée
|
||||||
|
|
||||||
|
Le supervisor possède un `JoinSet` privé de persistences et n'admet une nouvelle acquisition que lorsque :
|
||||||
|
|
||||||
|
```text
|
||||||
|
in_flight_persistence < persistence_concurrency
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune queue secondaire ou persistence non bornée n'est introduite. Le harness bloque quatre writes avec une limite de deux et exige `max_active == 2`.
|
||||||
|
|
||||||
|
## Drain
|
||||||
|
|
||||||
|
Sur stop ou fault : nouvelles admissions fermées, acquisitions déjà admises drainées/persistées, persistences rejointes, sources rejointes, puis terminal. Le timeout/abort et la priorité complète des races terminales restent dans `pre.009`.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
mode Normal exact
|
||||||
|
new/idempotent/purged + observation distincte
|
||||||
|
Rehydrated/couples impossibles rejetés
|
||||||
|
network mismatch avant write
|
||||||
|
content conflict redacted
|
||||||
|
Store failure redacted
|
||||||
|
persistence concurrency réellement bornée
|
||||||
|
terminaux runtime content_conflict/store_failed
|
||||||
|
ErrorCode publics stables
|
||||||
|
dependency boundary persistence privée / backend direct interdit
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/persistence.rs
|
||||||
|
deltas/0.3.11/pre.007.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
snapshots concrets / compteurs publics
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
backend Store direct
|
||||||
|
ForceRehydrate
|
||||||
|
source_failed / counter_exhausted
|
||||||
|
drain timeout / abort forcé
|
||||||
|
retry / reconnect / gap repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean
|
||||||
|
|
||||||
|
scan production : aucun unwrap, expect, panic, unbounded_channel
|
||||||
|
scan scope : aucun backend Store direct, aucun Transport, aucun snapshot concret
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.007` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'a changé dans cette tranche ; aucun `cargo tree` intermédiaire n'est donc requis :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.008` reste bloquée jusqu'à validation opérateur verte de la persistence `Normal`, des outcomes/idempotences, de la concurrence bornée et des terminaux Store/conflict.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Les snapshots/counters restent dans `pre.008`; le hardening fault/shutdown reste dans `pre.009`.
|
||||||
100
deltas/0.3.11/pre.008-fix.001.md
Normal file
100
deltas/0.3.11/pre.008-fix.001.md
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.008-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.008-fix.001` — correction Clippy du traitement de publication snapshot
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.008
|
||||||
|
workspace.package.version = 0.3.11-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown, `cargo check --workspace` et tous les tests de `ksp-worker-raw-transaction-ingest-lib` passent, mais que `cargo clippy --workspace --all-targets --all-features -- -D warnings` rejette un `if` imbriqué dans `src/runtime.rs`.
|
||||||
|
|
||||||
|
Diagnostic exact :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs:176:17
|
||||||
|
clippy::collapsible_if
|
||||||
|
```
|
||||||
|
|
||||||
|
Le code traitait une erreur de publication snapshot puis testait séparément `fault.is_none()`. Clippy exige la forme conditionnelle fusionnée.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger strictement `pre.008` sans modifier son comportement : même erreur, même priorité du premier fault, mêmes snapshots, mêmes compteurs, même lifecycle et même runtime. Aucune dépendance, feature ou surface publique n'est modifiée.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de production. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.008-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.8.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Le traitement fautif devient la forme équivalente acceptée par Clippy :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
if let Result::Err(error) = published
|
||||||
|
&& fault.is_none()
|
||||||
|
{
|
||||||
|
fault = Option::Some(error.code());
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
La sémantique reste identique : une erreur de publication snapshot ne remplace jamais un fault déjà fixé.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.008-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : le nested-if signalé n'existe plus
|
||||||
|
scan statique : aucune dépendance/feature ni surface publique modifiée
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'étant modifiée, aucun `cargo tree` n'est requis pour ce fix.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.009` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre de `pre.008` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
167
deltas/0.3.11/pre.008.md
Normal file
167
deltas/0.3.11/pre.008.md
Normal file
@@ -0,0 +1,167 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.008.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.008` — snapshots latest-value et projection Worker API
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.007
|
||||||
|
workspace.package.version = 0.3.11-pre.7
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 31 tests de la crate Worker et les doc-tests. Aucun `cargo tree` intermédiaire n'était requis : `pre.007` n'avait modifié aucune dépendance ni feature.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.008` du plan `032` : snapshot concret latest-value, compteurs checked, projection directe vers `ksp-worker-api::WorkerSnapshotSource`, coalescence des listeners et terminal retenu. Aucun hardening timeout/abort et aucune source réseau.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.008
|
||||||
|
workspace.package.version : 0.3.11-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
## Snapshot concret
|
||||||
|
|
||||||
|
La crate expose :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestSnapshot
|
||||||
|
RawTransactionIngestSnapshotFuture
|
||||||
|
RawTransactionIngestSnapshotSource
|
||||||
|
```
|
||||||
|
|
||||||
|
Le snapshot contient uniquement la projection `WorkerSnapshot`, les capacités/profondeurs techniques bornées et les compteurs sûrs prévus par le plan.
|
||||||
|
|
||||||
|
## Un seul flux latest-value
|
||||||
|
|
||||||
|
Un unique :
|
||||||
|
|
||||||
|
```text
|
||||||
|
tokio::sync::watch::channel<RawTransactionIngestSnapshot>
|
||||||
|
```
|
||||||
|
|
||||||
|
porte l'état observable du run. Le watch terminal spécialisé des tranches précédentes disparaît : `wait_terminal()`, `snapshot_source()` et `worker_snapshot_source()` lisent tous ce même flux.
|
||||||
|
|
||||||
|
`RawTransactionIngestSnapshotSource` implémente directement `ksp_worker_api::WorkerSnapshotSource`. La sequence `WorkerSnapshotSequence` est donc identique pour la vue concrète et la projection commune.
|
||||||
|
|
||||||
|
## Compteurs checked
|
||||||
|
|
||||||
|
Les compteurs `u64` effectivement incrémentés dans cette tranche utilisent `checked_add(1)`. Aucun wrap silencieux n'est admis. Le nouveau code public est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.counter_exhausted
|
||||||
|
```
|
||||||
|
|
||||||
|
Les compteurs sont mis à jour au point d'ownership du supervisor : dequeue/canonicalisation et récolte de persistence. `admission_queue_depth` provient directement du receiver `mpsc` borné ; `in_flight_persistence` provient du `JoinSet` privé existant.
|
||||||
|
|
||||||
|
`source_failure_total` et `backpressure_wait_total` font partie du snapshot sûr mais restent à zéro en `pre.008`; leur comportement effectif appartient à `pre.009`.
|
||||||
|
|
||||||
|
## Projection common Worker
|
||||||
|
|
||||||
|
Mapping de fondation :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Starting -> health Unknown
|
||||||
|
Running -> health Healthy
|
||||||
|
Stopping / Stopped -> conserve le dernier health non terminal
|
||||||
|
Faulted -> health Unhealthy
|
||||||
|
queue > 0 ou persistence > 0 -> activity Active
|
||||||
|
sinon Running/Stopping/terminal -> activity Idle
|
||||||
|
```
|
||||||
|
|
||||||
|
La state machine reste celle de `ksp-worker-api::WorkerLifecycle`.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
snapshot initial exact
|
||||||
|
projection common/concrete même sequence
|
||||||
|
counter exhaustion stable sans wrap
|
||||||
|
listener lent coalescé vers la dernière valeur
|
||||||
|
terminal latest-value retenu après fermeture publisher
|
||||||
|
surface publique snapshot_source/worker_snapshot_source
|
||||||
|
WorkerSnapshotSource Send + Sync
|
||||||
|
pipeline runtime réussi -> counters exacts
|
||||||
|
Store failure -> health Unhealthy + store_failure_total
|
||||||
|
absence de watch terminal séparé
|
||||||
|
absence de nouveau dependency edge
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/snapshot.rs
|
||||||
|
deltas/0.3.11/pre.008.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
backend Store direct
|
||||||
|
source_failed public
|
||||||
|
source failure behavior
|
||||||
|
backpressure wait instrumentation productive
|
||||||
|
drain timeout / abort forcé
|
||||||
|
fault/stop precedence finalisée
|
||||||
|
retry / reconnect / gap repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean
|
||||||
|
|
||||||
|
scan production : aucun unwrap, expect, panic, unbounded_channel
|
||||||
|
scan scope : aucun backend Store direct, aucun Transport, aucune nouvelle dépendance
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.008` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'a changé dans cette tranche ; aucun `cargo tree` intermédiaire n'est requis :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.009` reste bloquée jusqu'à validation opérateur verte des snapshots common/concrete, des compteurs checked et du terminal latest-value retenu.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Le hardening shutdown/backpressure/fault races reste dans `pre.009`.
|
||||||
100
deltas/0.3.11/pre.009-fix.001.md
Normal file
100
deltas/0.3.11/pre.009-fix.001.md
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.009-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.009-fix.001` — correction de compilation des tests de hardening
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.009
|
||||||
|
workspace.package.version = 0.3.11-pre.9
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent, mais que `cargo clippy --workspace --all-targets --all-features -- -D warnings` et `cargo test -p ksp-worker-raw-transaction-ingest-lib` ne peuvent compiler deux tests ajoutés par `pre.009`.
|
||||||
|
|
||||||
|
Diagnostics exacts :
|
||||||
|
|
||||||
|
```text
|
||||||
|
unit_tests/admission.rs : appel à RawTransactionAdmission::receive sans expected_network et traitement Option au lieu de Result<Option<_>>
|
||||||
|
unit_tests/runtime.rs : type du paramètre children non inférable dans le closure source-failure
|
||||||
|
```
|
||||||
|
|
||||||
|
Le code de production compile ; les défauts sont strictement localisés dans les tests de `pre.009`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement les tests de hardening de `pre.009`, sans modifier le runtime de production, l'admission, la persistence, les snapshots, les terminaux, les compteurs, les dépendances ou les features.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche des fichiers Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.009-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.9.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modifications
|
||||||
|
|
||||||
|
Le test `pre_009_full_queue_dequeue_marks_source_neutral_backpressure_observation` conserve désormais une copie du `RawNetworkId`, appelle explicitement `receive(&network)` et vérifie la forme `Result<Option<_>>` avec `matches!`, sans `unwrap`, `expect` ni `?`.
|
||||||
|
|
||||||
|
Le test `pre_009_source_failure_is_counted_and_late_stop_cannot_replace_terminal_fault` annote explicitement le paramètre `children` du closure avec le type privé attendu par le harness :
|
||||||
|
|
||||||
|
```text
|
||||||
|
&mut tokio::task::JoinSet<ksp_core_lib::Result<()>>
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun fichier Rust de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.009-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : les deux appels receive utilisent expected_network et traitent Result<Option<_>>
|
||||||
|
scan statique : le closure source-failure annote explicitement JoinSet<ksp_core_lib::Result<()>>
|
||||||
|
scan statique : aucun fichier src/ de production ni manifest de crate Worker modifié
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'étant modifiée, aucun `cargo tree` n'est requis pour ce fix.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.010` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre fonctionnel de `pre.009` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
97
deltas/0.3.11/pre.009-fix.002.md
Normal file
97
deltas/0.3.11/pre.009-fix.002.md
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.009-fix.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.009-fix.002` — complément de compilation du test source-failure
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.009-fix.001
|
||||||
|
workspace.package.version = 0.3.11-pre.9.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent, mais que `cargo clippy --workspace --all-targets --all-features -- -D warnings` et `cargo test -p ksp-worker-raw-transaction-ingest-lib` ne peuvent compiler un test de hardening `pre.009`.
|
||||||
|
|
||||||
|
Diagnostic exact :
|
||||||
|
|
||||||
|
```text
|
||||||
|
unit_tests/runtime.rs : appel de start_foundation_with_source_spawner avec 3 arguments alors que le harness en exige 4 ; store_guard manque
|
||||||
|
```
|
||||||
|
|
||||||
|
Le code de production compile ; le défaut est strictement localisé dans le test source-failure.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement l'appel du harness dans ce test, sans modifier le runtime de production, l'admission, la persistence, les snapshots, les terminaux, les compteurs, les dépendances ou les features.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.009-fix.002
|
||||||
|
workspace.package.version : 0.3.11-pre.9.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Le test `pre_009_source_failure_is_counted_and_late_stop_cannot_replace_terminal_fault` fournit désormais explicitement le troisième argument du harness :
|
||||||
|
|
||||||
|
```text
|
||||||
|
std::option::Option::None
|
||||||
|
```
|
||||||
|
|
||||||
|
Ce test ne possède pas de Store réel et ne teste aucune persistence ; l'absence de `store_guard` est donc explicite et conforme au rôle du harness.
|
||||||
|
|
||||||
|
Aucun fichier Rust de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.009-fix.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : le test source-failure fournit explicitement store_guard = None
|
||||||
|
scan statique : aucun fichier src/ de production ni manifest de crate Worker modifié
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'étant modifiée, aucun `cargo tree` n'est requis pour ce fix.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.010` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre fonctionnel de `pre.009` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
96
deltas/0.3.11/pre.009-fix.003.md
Normal file
96
deltas/0.3.11/pre.009-fix.003.md
Normal file
@@ -0,0 +1,96 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.009-fix.003.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.009-fix.003` — conformité Clippy du test source-failure
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.009-fix.002
|
||||||
|
workspace.package.version = 0.3.11-pre.9.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown, `cargo check --workspace` et les 45 tests de `ksp-worker-raw-transaction-ingest-lib` passent. Le seul échec restant est le gate Clippy strict :
|
||||||
|
|
||||||
|
```text
|
||||||
|
clippy::implicit_return
|
||||||
|
unit_tests/runtime.rs : le bloc async de la source faultée retourne implicitement Result::Err(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le code de production et le comportement du Worker ne sont pas en cause.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Rendre le test source-failure conforme à la règle d'explicitation des retours, sans modifier le runtime de production, l'admission, la persistence, les snapshots, les terminaux, les compteurs, les dépendances ou les features.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.009-fix.003
|
||||||
|
workspace.package.version : 0.3.11-pre.9.fix.3
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Dans `pre_009_source_failure_is_counted_and_late_stop_cannot_replace_terminal_fault`, le future injecté dans le `JoinSet` utilise désormais un retour explicite :
|
||||||
|
|
||||||
|
```text
|
||||||
|
return Result::Err(...);
|
||||||
|
```
|
||||||
|
|
||||||
|
La valeur retournée, le code d'erreur injecté et toute la sémantique du test restent inchangés.
|
||||||
|
|
||||||
|
Aucun fichier Rust de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.009-fix.003.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : le bloc async source-failure contient un return explicite
|
||||||
|
scan statique : aucun fichier src/ de production ni manifest de crate Worker modifié
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'étant modifiée, aucun `cargo tree` n'est requis pour ce fix.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.010` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre fonctionnel de `pre.009` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
135
deltas/0.3.11/pre.009.md
Normal file
135
deltas/0.3.11/pre.009.md
Normal file
@@ -0,0 +1,135 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.009.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.009` — hardening shutdown, backpressure et races fault/stop
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.008-fix.001
|
||||||
|
workspace.package.version = 0.3.11-pre.8.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 38 tests de la crate Worker et les doc-tests. Aucune dépendance/feature n'ayant changé, aucun `cargo tree` intermédiaire n'était requis.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Fermer uniquement le hardening `pre.009` du plan `032` : faults source sûrs, saturation observable, ordering stop/fault déterministe, deadline de drain, abort+join au timeout et absence de tâches orphelines. Aucune source live n'est introduite.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.009
|
||||||
|
workspace.package.version : 0.3.11-pre.9
|
||||||
|
```
|
||||||
|
|
||||||
|
## Faults source
|
||||||
|
|
||||||
|
Les tâches source privées retournent `ksp_core_lib::Result<()>`. Une erreur source ou un `JoinError` devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.source_failed
|
||||||
|
```
|
||||||
|
|
||||||
|
Le détail distant/provider n'est pas propagé et `source_failure_total` est incrémenté via checked-add.
|
||||||
|
|
||||||
|
## Shutdown borné
|
||||||
|
|
||||||
|
Le drain complet est enveloppé par `shutdown_drain_timeout`. En cas de timeout :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission.close()
|
||||||
|
persistence.abort_all()
|
||||||
|
sources.abort_all()
|
||||||
|
join complet des persistences
|
||||||
|
join complet des sources
|
||||||
|
Faulted(worker_raw_transaction_ingest.drain_timeout)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le terminal n'est donc jamais publié avant la récolte de toutes les tâches possédées.
|
||||||
|
|
||||||
|
## Ordering stop/fault
|
||||||
|
|
||||||
|
Le stop reste prioritaire dans le `select!`, mais un Store fault découvert pendant le drain n'est pas masqué. Un fault terminal déjà décidé n'est pas remplacé par un stop tardif. Le timeout de drain prévaut lorsqu'il empêche la clôture coopérative.
|
||||||
|
|
||||||
|
## Backpressure source-neutral
|
||||||
|
|
||||||
|
L'admission conserve une unique queue `mpsc` bornée. Le dequeue constate si cette queue était pleine à sa capacité configurée et incrémente `backpressure_wait_total`. Aucun drop, aucune queue secondaire et aucun `unbounded_channel` ne sont ajoutés.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
source failure terminal + compteur
|
||||||
|
late stop ne remplace pas source_failed
|
||||||
|
stop puis Store fault pendant drain
|
||||||
|
saturation capacity=1 sans drop
|
||||||
|
backpressure_wait_total observable
|
||||||
|
drain timeout + abort/join des sources et persistences
|
||||||
|
aucune tâche active après terminal timeout
|
||||||
|
projection snapshot source/backpressure
|
||||||
|
error codes source_failed/drain_timeout publics
|
||||||
|
dependency firewall inchangé
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.009.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/snapshot.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'est ajoutée, supprimée ou modifiée. Le `Cargo.toml` de la crate Worker reste identique à la base.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique des frontières runtime/dependencies
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.009` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'ayant changé, aucun `cargo tree` intermédiaire n'est requis :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.010` reste bloquée jusqu'à validation opérateur verte du hardening shutdown/backpressure/fault races.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Les sources live restent explicitement hors scope de `0.3.11`.
|
||||||
78
deltas/0.3.11/pre.010-fix.001.md
Normal file
78
deltas/0.3.11/pre.010-fix.001.md
Normal file
@@ -0,0 +1,78 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.010-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.010-fix.001` — correction du shadowing dans la canarie hardening
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.010
|
||||||
|
workspace.package.version = 0.3.11-pre.10
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent. Le gate Clippy/compilation des tests externes échoue sur `tests/hardening.rs` avec `E0618` : la variable locale `worker_id` masque la fonction helper `worker_id(...)`, puis le second appel tente d'appeler la valeur `WorkerId` comme une fonction.
|
||||||
|
|
||||||
|
Le code de production, les dépendances et le comportement du Worker ne sont pas en cause.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement cette collision de nom dans la canarie `pre.010`, sans modifier la surface publique, le runtime, les dépendances, les features ou les règles de hardening.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.010-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.10.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Dans `pre_010_debug_and_settings_errors_redact_worker_identity_and_invalid_values`, la variable locale issue du premier parse est renommée de `worker_id` vers `parsed_worker_id`. La fonction helper `worker_id(...)` reste ainsi appelable pour le second identifiant de test.
|
||||||
|
|
||||||
|
Aucun fichier Rust de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.010-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'est ajoutée, supprimée ou modifiée. Le `Cargo.toml` de la crate Worker reste identique à `pre.010`.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucun `cargo tree` n'est requis pour ce fix :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.011` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre fonctionnel de `pre.010` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
126
deltas/0.3.11/pre.010.md
Normal file
126
deltas/0.3.11/pre.010.md
Normal file
@@ -0,0 +1,126 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.010.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.010` — hardening public, release et sécurité
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.009-fix.003
|
||||||
|
workspace.package.version = 0.3.11-pre.9.fix.3
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 45 tests de la crate Worker et les doc-tests. Aucune dépendance/feature n'ayant changé, aucun `cargo tree` intermédiaire n'était requis.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Fermer uniquement le hardening externe `pre.010` du plan `032` : surface publique crate-root exacte, dépendances exactes, absence de surface historique, redaction, codes d'erreur, inventaire des modules et scans Config/secret/backend/Transport. Aucun comportement runtime n'est ajouté.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.010
|
||||||
|
workspace.package.version : 0.3.11-pre.10
|
||||||
|
```
|
||||||
|
|
||||||
|
## Hardening externe
|
||||||
|
|
||||||
|
Nouvelle suite `tests/hardening.rs` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
7 ErrorCode publics uniques et domain-scoped
|
||||||
|
WorkerId redacted dans Debug
|
||||||
|
settings invalides sans echo de valeur ni WorkerId
|
||||||
|
manifest productif exact
|
||||||
|
aucun dev/build dependency
|
||||||
|
crate::Item pour les impl partagées
|
||||||
|
aucune surface Backfill/historical/checkpoint/discovery/hydration/retriever
|
||||||
|
aucun Config productif
|
||||||
|
aucun secret-like material productif
|
||||||
|
aucun backend Store direct
|
||||||
|
aucun Transport productif
|
||||||
|
aucune dépendance retour depuis les couches basses
|
||||||
|
aucun type runtime/backend/live-source exposé par la crate-root
|
||||||
|
```
|
||||||
|
|
||||||
|
## Release completeness
|
||||||
|
|
||||||
|
Nouvelle suite `tests/release_completeness.rs` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
8 modules de production exacts
|
||||||
|
24 exports publics crate-root exacts
|
||||||
|
aucun pub mod
|
||||||
|
présence des canaries dependency/public/hardening requises
|
||||||
|
```
|
||||||
|
|
||||||
|
Le corpus attendu après cette tranche est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
35 unit tests
|
||||||
|
3 dependency-boundary tests
|
||||||
|
7 public API tests
|
||||||
|
8 hardening tests
|
||||||
|
3 release-completeness tests
|
||||||
|
56 tests Worker au total
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
deltas/0.3.11/pre.010.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Code de production
|
||||||
|
|
||||||
|
Aucun fichier `crates/ksp-worker-raw-transaction-ingest-lib/src/*.rs` n'est modifié.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'est ajoutée, supprimée ou modifiée. Le `Cargo.toml` de la crate Worker reste identique à la base.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scans statiques API/dependency/historical/Config/secret/backend/Transport
|
||||||
|
comparaison byte-for-byte des sources productives et du manifest Worker
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.010` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'ayant changé, aucun `cargo tree` intermédiaire n'est requis :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.011` reste bloquée jusqu'à validation opérateur verte des canaries public/release/security.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Les sources live restent hors scope de `0.3.11`.
|
||||||
45
deltas/0.3.11/pre.011.md
Normal file
45
deltas/0.3.11/pre.011.md
Normal file
@@ -0,0 +1,45 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.011.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.011`
|
||||||
|
|
||||||
|
## Objet
|
||||||
|
|
||||||
|
Ouvrir le gate technique final de `0.3.11` sans ajouter de scope fonctionnel.
|
||||||
|
|
||||||
|
## Changements
|
||||||
|
|
||||||
|
- passage de `workspace.package.version` à `0.3.11-pre.11` ;
|
||||||
|
- clôture documentaire du gate `pre.010-fix.001` ;
|
||||||
|
- matérialisation du gate final workspace, all-targets/all-features ;
|
||||||
|
- réintroduction volontaire des arbres Cargo Worker normal/features et de `cargo tree --duplicates` pour la validation finale ;
|
||||||
|
- aucun changement de code productif, test, dépendance ou feature.
|
||||||
|
|
||||||
|
## Non-changements
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune source live
|
||||||
|
aucun Transport
|
||||||
|
aucun Config
|
||||||
|
aucun backend Store direct
|
||||||
|
aucune API publique nouvelle
|
||||||
|
aucun comportement runtime nouveau
|
||||||
|
aucune dépendance/feature nouvelle
|
||||||
|
aucune réconciliation README/USAGE
|
||||||
|
aucune publication metadata
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate requis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo test --workspace --all-targets --all-features
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
169
deltas/0.3.11/pre.012.md
Normal file
169
deltas/0.3.11/pre.012.md
Normal file
@@ -0,0 +1,169 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.012.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.012` — réconciliation documentaire finale
|
||||||
|
|
||||||
|
## 1. Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.011
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate technique final opérateur du 8 septembre 2026 est fermé avant cette tranche : rustfmt check, audits, `cargo check`, Clippy strict, 56 tests Worker, tests workspace all-targets/all-features, arbres Cargo Worker normal/features et `cargo tree --duplicates` ont été exécutés avec succès sur `0.3.11-pre.11`.
|
||||||
|
|
||||||
|
## 2. Objectif
|
||||||
|
|
||||||
|
`pre.012` est le couloir de réconciliation documentaire finale de `0.3.11`.
|
||||||
|
|
||||||
|
La tranche :
|
||||||
|
|
||||||
|
- transforme les README/USAGE du Worker en références durables ;
|
||||||
|
- remplace les formulations où le Worker était encore décrit comme futur ;
|
||||||
|
- distingue explicitement la fondation source-neutral matérialisée des sources live `0.3.12+` ;
|
||||||
|
- réaligne les graphes d'architecture sur l'absence réelle de Transport dans la fondation ;
|
||||||
|
- ferme plan et validation sur les preuves réellement exécutées ;
|
||||||
|
- ne finalise ni `CHANGELOG.md`, ni `ROADMAP.md`, ni le prompt `0.3.12`.
|
||||||
|
|
||||||
|
## 3. Version workspace
|
||||||
|
|
||||||
|
`pre.012` est documentaire mais n'est pas un fix. Le contrat Cargo KSP impose donc le bump prerelease mécanique :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.11
|
||||||
|
->
|
||||||
|
0.3.11-pre.12
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun `Cargo.toml` de crate, edge de dépendance, feature, build setting, runtime ou configuration n'est modifié.
|
||||||
|
|
||||||
|
## 4. Contrat Worker documenté
|
||||||
|
|
||||||
|
La documentation durable fixe l'état actuel suivant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Worker concret ksp-worker-raw-transaction-ingest-lib
|
||||||
|
consumer de ksp-worker-api
|
||||||
|
settings réseau/Worker bornés
|
||||||
|
start/stop sur runtime Tokio caller-owned
|
||||||
|
supervision privée des tâches
|
||||||
|
mpsc central borné et backpressure
|
||||||
|
canonicalisation/assembly via ksp-raw-transaction-lib
|
||||||
|
observation key déterministe
|
||||||
|
persistence atomique via ksp-store-lib, mode Normal
|
||||||
|
concurrence Store bornée
|
||||||
|
snapshots latest-value concrets + WorkerSnapshotSource
|
||||||
|
classification settings/runtime/store/content/source/counter/drain
|
||||||
|
shutdown deadline + abort/join avant terminal timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
La même documentation rend explicite ce qui n'appartient pas encore à la fondation :
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune source HTTP/WS/Yellowstone productive
|
||||||
|
aucun edge ksp-onchain-transport-lib
|
||||||
|
aucune lecture Config
|
||||||
|
aucun backend Store direct
|
||||||
|
aucune API publique enqueue/source registration
|
||||||
|
aucun replay/gap-repair live
|
||||||
|
aucune campagne historique/backfill
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. Graphe de dépendances réconcilié
|
||||||
|
|
||||||
|
Le graphe normal Worker confirmé par `pre.011` est documenté comme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
-> ksp-core-lib
|
||||||
|
-> ksp-logging-lib
|
||||||
|
-> ksp-raw-transaction-lib
|
||||||
|
-> ksp-store-lib # default-features = false
|
||||||
|
-> ksp-worker-api
|
||||||
|
-> sha2
|
||||||
|
-> tokio
|
||||||
|
```
|
||||||
|
|
||||||
|
L'edge Worker -> `ksp-onchain-transport-lib` reste réservé aux adapters live ultérieurs et n'est plus présenté comme une dépendance déjà matérialisée.
|
||||||
|
|
||||||
|
## 6. Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/README.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
|
||||||
|
deltas/0.3.11/pre.012.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 7. Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
README.md
|
||||||
|
docs/architecture/004-COMPONENT_INVENTORY.md
|
||||||
|
docs/architecture/005-DEPENDENCY_GRAPH.md
|
||||||
|
docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md
|
||||||
|
docs/architecture/010-APPS_SERVICES_SCENARIOS_AND_CONTROL.md
|
||||||
|
docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md
|
||||||
|
docs/plans/002-FUNCTIONAL_RELEASE_SEQUENCE.md
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 8. Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## 9. Surfaces volontairement inchangées
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/**
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/**
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/**
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
prompts/**
|
||||||
|
config/**
|
||||||
|
```
|
||||||
|
|
||||||
|
## 10. Validations exécutées pendant l'assemblage
|
||||||
|
|
||||||
|
Sur l'état exact destiné à l'archive :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 table(s), 795 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
La revue de diff confirme également qu'aucun fichier `src/`, test, manifest de crate, Config, `CHANGELOG.md`, `ROADMAP.md` ou prompt n'est modifié.
|
||||||
|
|
||||||
|
## 11. Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne dispose pas de Cargo/Rustc. Les validations Cargo propres au delta restent donc opérateur :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate technique lourd (`clippy`, workspace all-targets/all-features, arbres Cargo) n'est pas rejoué car il appartient à `pre.011` et aucun code/test/dépendance/feature n'est modifié.
|
||||||
|
|
||||||
|
## 12. Décisions prises
|
||||||
|
|
||||||
|
- la fondation `0.3.11` est documentée comme Worker concret utilisable pour lifecycle/observabilité, mais sans source réseau productive ;
|
||||||
|
- l'admission reste privée : aucune API publique d'enqueue n'est inventée pour anticiper `0.3.12` ;
|
||||||
|
- les graphes séparent l'état matériel `0.3.11` du futur edge Transport ;
|
||||||
|
- le Job Backfill reste indépendant et historique ;
|
||||||
|
- `pre.013` reste seule propriétaire du prompt `0.3.12`, de `CHANGELOG.md` et de `ROADMAP.md`.
|
||||||
|
|
||||||
|
## 13. Questions ouvertes
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune question bloquante pour la clôture documentaire 0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Les choix précis de source/adapters live restent des décisions de `0.3.12+`, sans être figés par cette tranche documentaire.
|
||||||
164
deltas/0.3.11/pre.013.md
Normal file
164
deltas/0.3.11/pre.013.md
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.013.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.013` — préparation de publication
|
||||||
|
|
||||||
|
## 1. Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.012
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.012` est vert sur rustfmt check, audits Rust/Markdown, `cargo check --workspace`, 56 tests Worker et doc-tests.
|
||||||
|
|
||||||
|
## 2. Objectif
|
||||||
|
|
||||||
|
`pre.013` est la dernière prerelease de préparation de publication de `0.3.11`.
|
||||||
|
|
||||||
|
Conformément au couloir KSP de fermeture, elle modifie uniquement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompt de démarrage 0.3.12
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
Cargo.toml racine mécanique
|
||||||
|
delta pre.013
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun README, USAGE, plan, validation, architecture, code, test, Config ou manifest de crate n'est modifié.
|
||||||
|
|
||||||
|
## 3. Version workspace
|
||||||
|
|
||||||
|
La prerelease non-fix synchronise Cargo :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.12
|
||||||
|
->
|
||||||
|
0.3.11-pre.13
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. Préparation de la release suivante
|
||||||
|
|
||||||
|
Le nouveau prompt :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/031-V0_3_12_START_PROMPT.md
|
||||||
|
```
|
||||||
|
|
||||||
|
ouvre `0.3.12` exclusivement depuis le futur stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Sa mission est bornée à Yellowstone + hydration HTTP + continuité propre au run du Worker. Son `pre.001` impose avant codage :
|
||||||
|
|
||||||
|
```text
|
||||||
|
audit stable réel 0.3.11
|
||||||
|
audit Yellowstone/Solana/provider courant
|
||||||
|
matrice Transaction/Status/Block/BlockMeta/Slot/from_slot/ReplayInfo
|
||||||
|
architecture d'injection Transport sans Worker -> Config
|
||||||
|
stratégie hydration et provenance
|
||||||
|
frontier/reconnect/replay semantics
|
||||||
|
threat model live
|
||||||
|
smokes réellement accessibles
|
||||||
|
graphe Cargo cible
|
||||||
|
sizing et rescission éventuelle
|
||||||
|
plan 033 + validation 029
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. CHANGELOG
|
||||||
|
|
||||||
|
`CHANGELOG.md` reçoit l'entrée de synthèse `0.3.11` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
fondation Worker source-neutral
|
||||||
|
admission/persistence/snapshots/shutdown hardening
|
||||||
|
dependency firewall
|
||||||
|
absence volontaire de source live dans 0.3.11
|
||||||
|
gate technique final réel
|
||||||
|
handoff 0.3.12-0.3.14
|
||||||
|
prompt 031
|
||||||
|
```
|
||||||
|
|
||||||
|
## 6. ROADMAP
|
||||||
|
|
||||||
|
L'entrée `0.3.11` passe de `[ ]` à `[X]` et décrit l'état effectivement livré. Les entrées `0.3.12`–`0.3.14` restent ouvertes et conservent leur découpage live existant.
|
||||||
|
|
||||||
|
## 7. Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/031-V0_3_12_START_PROMPT.md
|
||||||
|
deltas/0.3.11/pre.013.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 8. Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 9. Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## 10. Surfaces explicitement inchangées
|
||||||
|
|
||||||
|
```text
|
||||||
|
README.md
|
||||||
|
docs/**
|
||||||
|
crates/**
|
||||||
|
config/**
|
||||||
|
tests/**
|
||||||
|
```
|
||||||
|
|
||||||
|
Hormis le nouveau prompt situé sous `prompts/`, aucun autre prompt n'est modifié.
|
||||||
|
|
||||||
|
## 11. Validations exécutées pendant l'assemblage
|
||||||
|
|
||||||
|
Sur l'état exact destiné à l'archive :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 table(s), 797 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
La revue de diff confirme également :
|
||||||
|
|
||||||
|
```text
|
||||||
|
2 fichiers ajoutés
|
||||||
|
3 fichiers modifiés
|
||||||
|
0 fichier supprimé
|
||||||
|
aucun changement README/docs/crates/config/tests
|
||||||
|
prompt 031 : 16 sections normatives présentes
|
||||||
|
```
|
||||||
|
|
||||||
|
## 12. Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne dispose pas de Cargo/Rustc. Les contrôles Cargo minimum de la prerelease restent opérateur :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun test ou Clippy lourd n'est requis par le contenu de cette tranche de publication, puisqu'aucun code/test/dépendance/feature n'est modifié.
|
||||||
|
|
||||||
|
## 13. Décisions prises
|
||||||
|
|
||||||
|
- `0.3.11` est prêt pour une publication stable mécanique après validation de `pre.013` ;
|
||||||
|
- `0.3.12` ne commence qu'après `0.3.11-rel.001` et le tag stable `v0.3.11` ;
|
||||||
|
- le prompt suivant n'autorise aucun codage Yellowstone avant son `pre.001` d'audit/sizing ;
|
||||||
|
- aucune correction d'un couloir antérieur n'est absorbée dans cette préparation de publication.
|
||||||
|
|
||||||
|
## 14. Questions ouvertes
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune question bloquante pour la préparation de publication 0.3.11
|
||||||
|
```
|
||||||
176
deltas/0.3.11/rel.001.md
Normal file
176
deltas/0.3.11/rel.001.md
Normal file
@@ -0,0 +1,176 @@
|
|||||||
|
<!-- file: deltas/0.3.11/rel.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-rel.001` — publication stable
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
Base directe attendue :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.013
|
||||||
|
workspace.package.version = 0.3.11-pre.13
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.013`, exécuté le **8 septembre 2026**, est propre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check: PASS
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 table(s), 797 file(s))
|
||||||
|
cargo check --workspace: PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
Les couloirs de fermeture antérieurs sont déjà fermés :
|
||||||
|
|
||||||
|
- `pre.011` : gate technique final sur `0.3.11-pre.11`, avec rustfmt check, audits Rust/Markdown, `cargo check --workspace`, Clippy workspace/all-targets/all-features `-D warnings`, 56 tests Worker, `cargo test --workspace --all-targets --all-features`, arbres Cargo Worker normal/features et `cargo tree --duplicates` ;
|
||||||
|
- `pre.012` : réconciliation documentaire finale avec README/USAGE durables du Worker et architecture/références réalignées sur l'absence volontaire de source live dans `0.3.11` ;
|
||||||
|
- `pre.013` : préparation minimale de publication avec `CHANGELOG.md`, `ROADMAP.md` et `prompts/031-V0_3_12_START_PROMPT.md`, dont les 16 sections normatives ont été vérifiées conformes à `docs/rules/PROMPT_STRUCTURE.md`.
|
||||||
|
|
||||||
|
## Objet
|
||||||
|
|
||||||
|
Publier mécaniquement la version stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Conformément à `VER-LIFECYCLE-012`, cette tranche ne corrige ni code, ni test, ni documentation durable, ni architecture, ni configuration, ni dépendance, ni prompt.
|
||||||
|
|
||||||
|
Tout défaut nouveau renvoie vers une prerelease appropriée ; `rel.001` n'est pas une tranche de rattrapage.
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
### `Cargo.toml`
|
||||||
|
|
||||||
|
Le header est incrémenté parce que le fichier est réellement modifié :
|
||||||
|
|
||||||
|
```text
|
||||||
|
517 -> 518
|
||||||
|
```
|
||||||
|
|
||||||
|
La version workspace devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.13 -> 0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune autre ligne du `Cargo.toml` racine n'est modifiée.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/rel.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Surfaces explicitement inchangées
|
||||||
|
|
||||||
|
```text
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
README.md
|
||||||
|
RULES.md
|
||||||
|
prompts/031-V0_3_12_START_PROMPT.md
|
||||||
|
crates/**
|
||||||
|
docs/**
|
||||||
|
config/**
|
||||||
|
```
|
||||||
|
|
||||||
|
La surface préparée en `pre.013` reste byte-identique hors mécanique Cargo et ajout du présent delta.
|
||||||
|
|
||||||
|
## Surface stable publiée
|
||||||
|
|
||||||
|
`0.3.11` stabilise `ksp-worker-raw-transaction-ingest-lib` comme fondation source-neutral du Worker continu `RawTransaction` :
|
||||||
|
|
||||||
|
- identité Worker stable `raw_transaction_ingest` et settings bornés ;
|
||||||
|
- lifecycle `start` / stop idempotent / terminal future sur runtime Tokio fourni par le caller ;
|
||||||
|
- supervisor privé avec ownership explicite des tâches source et persistence ;
|
||||||
|
- admission centrale `mpsc` bornée, backpressure observable et absence de drop silencieux ;
|
||||||
|
- canonicalisation/assembly via `ksp-raw-transaction-lib` et observation key déterministe ;
|
||||||
|
- persistence atomique via la seule façade `ksp-store-lib`, en mode `Normal`, avec insert/idempotence/tombstone et `content_conflict` explicite ;
|
||||||
|
- snapshots concrets latest-value projetés sur `WorkerSnapshotSource`, avec séquences et compteurs checked ;
|
||||||
|
- classification stable des faults Store/source/counter/drain ;
|
||||||
|
- hardening des races stop/fault, Store lent/fautif, saturation et shutdown borné ;
|
||||||
|
- expiration de `shutdown_drain_timeout` suivie d'un abort + join de toutes les tâches privées avant publication terminale ;
|
||||||
|
- canaris externes de dependency firewall, redaction, inventaire des modules et exports publics.
|
||||||
|
|
||||||
|
La release ferme volontairement **sans source réseau productive** :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Worker -X-> ksp-onchain-transport-lib
|
||||||
|
Worker -X-> ksp-config-lib
|
||||||
|
Worker -X-> ksp-job-backfill-lib
|
||||||
|
Worker -X-> backend Store physique
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Job Backfill historique paramétré et le Worker Ingest continu restent deux producteurs indépendants du même Store. `0.3.11` ne fournit ni adapter HTTP/WS/Yellowstone productif, ni API publique d'enqueue/source registration, ni replay/gap-repair live.
|
||||||
|
|
||||||
|
La trajectoire live stabilisée est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12 Yellowstone + hydration HTTP + continuité propre au run
|
||||||
|
0.3.13 WS standard / Helius / HTTP live + convergence multi-source
|
||||||
|
0.3.14 gap repair / hardening multi-source
|
||||||
|
0.3.15 ksp-app-raw-transaction-ingest-desk
|
||||||
|
0.3.16 Backfill multi-source / multi-stratégie
|
||||||
|
```
|
||||||
|
|
||||||
|
## Suite préparée
|
||||||
|
|
||||||
|
La session suivante part exclusivement de la base stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
et exécute :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/031-V0_3_12_START_PROMPT.md
|
||||||
|
```
|
||||||
|
|
||||||
|
pour ouvrir :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12 — Yellowstone + hydration HTTP + continuité de run du Worker RawTransaction
|
||||||
|
```
|
||||||
|
|
||||||
|
`0.3.12-pre.001` doit commencer par lecture, audit interne/externe, brainstorming, sizing et planification avant toute implémentation lourde. Il doit notamment réauditer Yellowstone/Solana/providers, l'injection Transport, l'hydration HTTP, le frontier, `from_slot`/ReplayInfo, les smokes réellement accessibles et le graphe Cargo cible avant d'ajouter un edge productif au Worker.
|
||||||
|
|
||||||
|
## Gate demandé
|
||||||
|
|
||||||
|
La tranche est purement mécanique. Aucune dépendance, feature, source runtime, configuration ou documentation durable n'est modifiée par `rel.001`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.11
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Après gate propre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
commit : v0.3.11-rel.001
|
||||||
|
tag : v0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun tag de prerelease ou `rel.001` n'est requis.
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: docs/architecture/004-COMPONENT_INVENTORY.md -->
|
<!-- file: docs/architecture/004-COMPONENT_INVENTORY.md -->
|
||||||
<!-- version: 37 -->
|
<!-- version: 38 -->
|
||||||
|
|
||||||
# Inventaire initial des composants KSP
|
# Inventaire initial des composants KSP
|
||||||
|
|
||||||
@@ -45,8 +45,8 @@ Ce document maintient l'inventaire synthétique des composants retenus ou presse
|
|||||||
| Backfill Desk | `ksp-app-backfill-desk` | app | Implémenté | `0.3.7`, extension `0.3.16` | contrôle du backfill RAW ; sélection des stratégies/sources ajoutée après le worker live |
|
| Backfill Desk | `ksp-app-backfill-desk` | app | Implémenté | `0.3.7`, extension `0.3.16` | contrôle du backfill RAW ; sélection des stratégies/sources ajoutée après le worker live |
|
||||||
| Store Desk | `ksp-app-store-desk` | app | Implémenté | `0.3.8` | inspection RAW read-only Transaction/Account/Observation via façade Store |
|
| Store Desk | `ksp-app-store-desk` | app | Implémenté | `0.3.8` | inspection RAW read-only Transaction/Account/Observation via façade Store |
|
||||||
| RAW transaction common | `ksp-raw-transaction-lib` | lib | Implémenté | `0.3.10` | canonicalisation/wire RAW Transaction v1 source-neutral partagé entre producteurs |
|
| RAW transaction common | `ksp-raw-transaction-lib` | lib | Implémenté | `0.3.10` | canonicalisation/wire RAW Transaction v1 source-neutral partagé entre producteurs |
|
||||||
| Worker lifecycle | `ksp-worker-api` | API | Retenu | `0.3.9` | lifecycle/health/progression génériques des services continus |
|
| Worker lifecycle | `ksp-worker-api` | API | Implémenté | `0.3.9` | lifecycle/health/progression génériques des services continus |
|
||||||
| RAW transaction worker | `ksp-worker-raw-transaction-ingest-lib` | worker/lib | Retenu | `0.3.11`–`0.3.14` | ingestion continue `RawTransaction` multi-source, déduplication/provenance/recovery |
|
| RAW transaction worker | `ksp-worker-raw-transaction-ingest-lib` | worker/lib | Implémenté | `0.3.11`, extensions `0.3.12+` | fondation source-neutral, puis sources live multi-source/recovery |
|
||||||
| RAW ingest Desk | `ksp-app-raw-transaction-ingest-desk` | app | Retenu | `0.3.15` | choix/supervision d’une ou plusieurs sources/méthodes sans réimplémenter le worker |
|
| RAW ingest Desk | `ksp-app-raw-transaction-ingest-desk` | app | Retenu | `0.3.15` | choix/supervision d’une ou plusieurs sources/méthodes sans réimplémenter le worker |
|
||||||
| STRUCTURAL job | nom à fixer | job/lib | Retenu | couche STRUCTURAL | normalisation Solana générique RAW -> STRUCTURAL bornée/rejouable |
|
| STRUCTURAL job | nom à fixer | job/lib | Retenu | couche STRUCTURAL | normalisation Solana générique RAW -> STRUCTURAL bornée/rejouable |
|
||||||
| STRUCTURAL worker | nom à fixer | worker/lib | Retenu | fin couche STRUCTURAL | backlog RAW -> STRUCTURAL continu |
|
| STRUCTURAL worker | nom à fixer | worker/lib | Retenu | fin couche STRUCTURAL | backlog RAW -> STRUCTURAL continu |
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: docs/architecture/005-DEPENDENCY_GRAPH.md -->
|
<!-- file: docs/architecture/005-DEPENDENCY_GRAPH.md -->
|
||||||
<!-- version: 26 -->
|
<!-- version: 27 -->
|
||||||
|
|
||||||
# Graphe de dépendances KSP
|
# Graphe de dépendances KSP
|
||||||
|
|
||||||
@@ -419,22 +419,22 @@ Il remplit RAW et ne décode aucun programme. Il ne dépend pas de Config : la c
|
|||||||
ksp-worker-api
|
ksp-worker-api
|
||||||
-> ksp-core-lib
|
-> ksp-core-lib
|
||||||
|
|
||||||
ksp-worker-raw-transaction-ingest-lib
|
ksp-worker-raw-transaction-ingest-lib # fondation source-neutral matérialisée
|
||||||
-> ksp-worker-api
|
-> ksp-core-lib
|
||||||
-> ksp-onchain-transport-lib
|
|
||||||
-> ksp-raw-transaction-lib
|
|
||||||
-> ksp-store-lib
|
|
||||||
-> ksp-logging-lib
|
-> ksp-logging-lib
|
||||||
# Config reste possédé par la composition supérieure ; aucune dépendance backend/provider physique
|
-> ksp-raw-transaction-lib
|
||||||
|
-> ksp-store-lib # default-features = false ; aucun backend imposé
|
||||||
|
-> ksp-worker-api
|
||||||
|
-> sha2 / tokio # observation key + runtime privé
|
||||||
|
|
||||||
ksp-worker-control-lib
|
ksp-worker-control-lib # composant retenu, non matérialisé ici
|
||||||
-> ksp-worker-api
|
-> ksp-worker-api
|
||||||
-> ksp-core-lib
|
-> ksp-core-lib
|
||||||
```
|
```
|
||||||
|
|
||||||
`ksp-worker-api` est ouvert en `0.3.9` comme contrat générique de services continus et ne connaît ni Solana, ni Transport, ni Store, ni Tauri. Son premier consumer concret est prévu en `0.3.11` avec `ksp-worker-raw-transaction-ingest-lib`, puis ses sources live sont complétées jusqu’en `0.3.14`.
|
`ksp-worker-api` est ouvert en `0.3.9` comme contrat générique de services continus et ne connaît ni Solana, ni Transport, ni Store, ni Tauri. `ksp-worker-raw-transaction-ingest-lib` est son premier consumer concret : sa fondation `0.3.11` est source-neutral, possède admission/persistence/snapshots/shutdown, et ne dépend pas encore de Transport. Les adapters live sont ajoutés ensuite sans déplacer Config ni backend physique dans le Worker.
|
||||||
|
|
||||||
Le worker RAW Transaction n'est pas défini comme « un worker WebSocket » ou « un worker gRPC ». Il reçoit une ou plusieurs stratégies d'acquisition construites au-dessus des façades KSP réellement disponibles ; celles-ci peuvent être alternatives, complémentaires (discovery + hydration), redondantes entre providers ou spécialisées live/catch-up/gap-repair. La transaction canonique reste identifiée indépendamment de la source et chaque acquisition utile conserve sa propre observation/provenance Store.
|
La cible live n'est pas « un worker WebSocket » ou « un worker gRPC ». Les tranches de sources peuvent ajouter `ksp-onchain-transport-lib` lorsque l'adapter productif le nécessite et composer des stratégies alternatives, complémentaires (discovery + hydration), redondantes entre providers ou spécialisées live/catch-up/gap-repair. La transaction canonique reste identifiée indépendamment de la source et chaque acquisition utile conserve sa propre observation/provenance Store.
|
||||||
|
|
||||||
RAW worker puis STRUCTURAL worker sont introduits à la fin de leur couche respective, lorsque persistence/backlog sont disponibles. Le traitement RAW -> STRUCTURAL borné est porté par un STRUCTURAL job distinct du service continu. Les workers DECODED/DOMAIN sont introduits avec les groupes Program concernés plutôt que tous anticipés en bloc.
|
RAW worker puis STRUCTURAL worker sont introduits à la fin de leur couche respective, lorsque persistence/backlog sont disponibles. Le traitement RAW -> STRUCTURAL borné est porté par un STRUCTURAL job distinct du service continu. Les workers DECODED/DOMAIN sont introduits avec les groupes Program concernés plutôt que tous anticipés en bloc.
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md -->
|
<!-- file: docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md -->
|
||||||
<!-- version: 17 -->
|
<!-- version: 18 -->
|
||||||
|
|
||||||
# Acquisition, workers, jobs et pipelines spécialisés
|
# Acquisition, workers, jobs et pipelines spécialisés
|
||||||
|
|
||||||
@@ -102,6 +102,8 @@ ksp-worker-raw-transaction-ingest-lib
|
|||||||
|
|
||||||
Sa responsabilité est l'acquisition **continue** de `RawTransaction` puis la persistance via `ksp-store-lib`. Il ne décode pas de Program, ne possède aucun SQL/backend physique et ne fait pas de la source réseau une partie de l'identité canonique de transaction.
|
Sa responsabilité est l'acquisition **continue** de `RawTransaction` puis la persistance via `ksp-store-lib`. Il ne décode pas de Program, ne possède aucun SQL/backend physique et ne fait pas de la source réseau une partie de l'identité canonique de transaction.
|
||||||
|
|
||||||
|
La fondation matérialisée possède déjà le lifecycle continu, les settings techniques bornés, la queue d'admission privée bornée, la canonicalisation Common RAW, la persistance atomique backend-neutral, les snapshots latest-value et le shutdown borné. Elle ne possède encore aucune source réseau productive et n'a donc aucun edge Transport ; les adapters live/catch-up complètent ensuite ce même Worker sans ouvrir de second runtime parallèle.
|
||||||
|
|
||||||
Le modèle cible n'est pas :
|
Le modèle cible n'est pas :
|
||||||
|
|
||||||
```text
|
```text
|
||||||
@@ -140,7 +142,7 @@ Le worker ne doit pas être réduit à un enum superficiel `Http | WebSocket | G
|
|||||||
|
|
||||||
#### Résultat de l'audit `0.3.9`
|
#### Résultat de l'audit `0.3.9`
|
||||||
|
|
||||||
L'audit exhaustif est synthétisé dans [`011-RAW_TRANSACTION_ACQUISITION.md`](011-RAW_TRANSACTION_ACQUISITION.md). Il confirme qu'un besoin d'acquisition Solana/provider ne remonte pas dans `ksp-worker-api` : le contrat générique reste fermé et le futur Worker concret porte ses propres capabilities de sources.
|
L'audit exhaustif est synthétisé dans [`011-RAW_TRANSACTION_ACQUISITION.md`](011-RAW_TRANSACTION_ACQUISITION.md). Il confirme qu'un besoin d'acquisition Solana/provider ne remonte pas dans `ksp-worker-api` : le contrat générique reste fermé et le Worker concret porte ses propres capabilities de sources, sans les ouvrir dans sa fondation source-neutral.
|
||||||
|
|
||||||
Les familles admises par la synthèse couvrent notamment :
|
Les familles admises par la synthèse couvrent notamment :
|
||||||
|
|
||||||
@@ -479,14 +481,24 @@ Le Job conserve seul ses scopes, campagnes, checkpoints et lifecycle.
|
|||||||
|
|
||||||
### RAW worker
|
### RAW worker
|
||||||
|
|
||||||
|
Fondation source-neutral matérialisée :
|
||||||
|
|
||||||
```text
|
```text
|
||||||
ksp-worker-raw-transaction-ingest-lib
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
-> ksp-core-lib
|
||||||
-> ksp-worker-api
|
-> ksp-worker-api
|
||||||
-> ksp-onchain-transport-lib
|
|
||||||
-> ksp-raw-transaction-lib
|
-> ksp-raw-transaction-lib
|
||||||
-> ksp-interface-lib # seulement si un fait passif partagé aide réellement la composition live
|
-> ksp-store-lib # façade Store ; default-features=false ; aucun backend physique direct
|
||||||
-> ksp-store-lib # façade Store ; aucun backend physique direct
|
|
||||||
-> ksp-logging-lib
|
-> ksp-logging-lib
|
||||||
|
-> sha2 / tokio # observation key + runtime privé
|
||||||
|
```
|
||||||
|
|
||||||
|
Extension live ultérieure :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
-> ksp-onchain-transport-lib # seulement lorsque les adapters sources productifs sont matérialisés
|
||||||
|
-> ksp-interface-lib # seulement si un fait passif partagé aide réellement la composition live
|
||||||
|
|
||||||
composition supérieure / future Desk
|
composition supérieure / future Desk
|
||||||
-> ksp-config-lib
|
-> ksp-config-lib
|
||||||
@@ -495,7 +507,7 @@ composition supérieure / future Desk
|
|||||||
-> ksp-store-lib
|
-> ksp-store-lib
|
||||||
```
|
```
|
||||||
|
|
||||||
Le Worker conserve seul son runtime continu, ses sources actives, sa continuité et son lifecycle. Il n'appelle ni ne pilote le Job Backfill.
|
Le Worker conserve seul son runtime continu, ses sources actives lorsqu'elles existent, sa continuité et son lifecycle. Il n'appelle ni ne pilote le Job Backfill. La fondation n'expose aucun ingress public : les sources internes futures alimentent la queue bornée et subissent sa backpressure.
|
||||||
|
|
||||||
Les événements Interface peuvent servir de signal provider-neutral à la composition live, mais ne constituent jamais le backlog durable. Après crash ou perte d'un événement, la reprise s'appuie sur Store et sur les primitives de replay/hydratation appropriées.
|
Les événements Interface peuvent servir de signal provider-neutral à la composition live, mais ne constituent jamais le backlog durable. Après crash ou perte d'un événement, la reprise s'appuie sur Store et sur les primitives de replay/hydratation appropriées.
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: docs/architecture/010-APPS_SERVICES_SCENARIOS_AND_CONTROL.md -->
|
<!-- file: docs/architecture/010-APPS_SERVICES_SCENARIOS_AND_CONTROL.md -->
|
||||||
<!-- version: 14 -->
|
<!-- version: 15 -->
|
||||||
|
|
||||||
# Applications, services, scenarios et control plane
|
# Applications, services, scenarios et control plane
|
||||||
|
|
||||||
@@ -138,7 +138,7 @@ Backfill Desk suit exactement cette règle : elle ouvre Store via `ksp-store-lib
|
|||||||
|
|
||||||
Store Desk suit la même frontière sans Transport : Config sélectionne Logging + Store, `ksp-store-lib` fournit health, reads et inspection backend-neutral, et le Store ouvert reste côté Rust. Les DataTables frontend consomment des summaries server-side et counts exacts ; les détails chargent une seule entité avec preview bornée. Aucun command d'écriture, SQL, cursor opaque, backend physique ou secret Store n'est exposé à Tauri/TypeScript.
|
Store Desk suit la même frontière sans Transport : Config sélectionne Logging + Store, `ksp-store-lib` fournit health, reads et inspection backend-neutral, et le Store ouvert reste côté Rust. Les DataTables frontend consomment des summaries server-side et counts exacts ; les détails chargent une seule entité avec preview bornée. Aucun command d'écriture, SQL, cursor opaque, backend physique ou secret Store n'est exposé à Tauri/TypeScript.
|
||||||
|
|
||||||
Raw Transaction Ingest Desk est la surface spécialisée de contrôle prévue après le premier worker concret. Elle peut sélectionner une ou plusieurs stratégies compatibles et afficher lifecycle/health/rates/backpressure/reconnect/recovery, mais la sélection d'une source ne déplace ni discovery, ni hydration, ni déduplication, ni provenance dans l'application. Les endpoints/credentials restent Config/Transport-owned et le détail des entités persistées reste Store Desk-owned.
|
Raw Transaction Ingest Desk est la surface spécialisée de contrôle prévue après la fondation du premier worker concret. Cette fondation expose déjà lifecycle, stop et snapshots source-neutral ; la Desk ne doit cependant sélectionner des sources ou afficher reconnect/recovery qu'après matérialisation des adapters live correspondants. La sélection d'une source ne déplace ni discovery, ni hydration, ni déduplication, ni provenance dans l'application. Les endpoints/credentials restent Config/Transport-owned et le détail des entités persistées reste Store Desk-owned.
|
||||||
|
|
||||||
Les couches N1–N4 expriment des responsabilités et une direction de dépendances ; elles n'imposent pas de traverser toutes les couches intermédiaires.
|
Les couches N1–N4 expriment des responsabilités et une direction de dépendances ; elles n'imposent pas de traverser toutes les couches intermédiaires.
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md -->
|
<!-- file: docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md -->
|
||||||
<!-- version: 11 -->
|
<!-- version: 12 -->
|
||||||
|
|
||||||
# Acquisition et alimentation `RawTransaction`
|
# Acquisition et alimentation `RawTransaction`
|
||||||
|
|
||||||
@@ -149,7 +149,7 @@ Il peut utiliser **HTTP, WS, Yellowstone gRPC, replay provider ou archive** si c
|
|||||||
|
|
||||||
### 3.2 Worker Raw Transaction Ingest : acquisition continue start/stop
|
### 3.2 Worker Raw Transaction Ingest : acquisition continue start/stop
|
||||||
|
|
||||||
`ksp-worker-raw-transaction-ingest-lib` a pour rôle de **remplir continuellement Store à partir du moment où il est démarré**.
|
`ksp-worker-raw-transaction-ingest-lib` a pour rôle cible de **remplir continuellement Store à partir du moment où il est démarré**. Sa fondation source-neutral matérialise déjà le runtime, l'admission, la canonicalisation Common RAW, la persistence et les snapshots, mais aucune source réseau productive n'est encore branchée.
|
||||||
|
|
||||||
Son contrôle métier V1 est volontairement court :
|
Son contrôle métier V1 est volontairement court :
|
||||||
|
|
||||||
@@ -161,17 +161,17 @@ snapshot / notifications
|
|||||||
|
|
||||||
Le caller ne lui fournit pas une signature, un `program_id`, une plage historique, une limite de campagne ou une requête de backfill. Les endpoints, réseaux, sources activées, capabilities et secrets proviennent de Config/composition, pas d'un payload métier de `start`.
|
Le caller ne lui fournit pas une signature, un `program_id`, une plage historique, une limite de campagne ou une requête de backfill. Les endpoints, réseaux, sources activées, capabilities et secrets proviennent de Config/composition, pas d'un payload métier de `start`.
|
||||||
|
|
||||||
À partir de son démarrage, le Worker :
|
La fondation actuelle, lorsqu'elle démarre sans adapter source productif, publie son lifecycle/snapshot puis reste contrôlable jusqu'au stop. Dès qu'une source interne est matérialisée, le pipeline cible devient :
|
||||||
|
|
||||||
```text
|
```text
|
||||||
acquiert les nouvelles transactions disponibles
|
acquiert les nouvelles transactions disponibles
|
||||||
hydrate les signaux live incomplets si nécessaire
|
hydrate les signaux live incomplets si nécessaire
|
||||||
normalise puis persiste RawTransaction + observations
|
normalise puis persiste RawTransaction + observations
|
||||||
publie ses notifications indépendamment de leurs lecteurs
|
publie ses snapshots latest-value indépendamment de leurs lecteurs
|
||||||
continue jusqu'à stop ou fault
|
continue jusqu'à stop ou fault
|
||||||
```
|
```
|
||||||
|
|
||||||
Le Worker peut utiliser **WS, Yellowstone gRPC, HTTP, block polling, provider streams ou sources EARLY** si ces capacités servent l'acquisition live.
|
Le Worker peut ensuite utiliser **WS, Yellowstone gRPC, HTTP, block polling, provider streams ou sources EARLY** si ces capacités servent l'acquisition live. Ces sources restent internes au Worker ; le caller ne pousse pas directement des ingress dans sa queue privée.
|
||||||
|
|
||||||
Il ne lance pas de campagne historique arbitraire.
|
Il ne lance pas de campagne historique arbitraire.
|
||||||
|
|
||||||
@@ -648,7 +648,7 @@ Worker lifecycle
|
|||||||
backend Store physique
|
backend Store physique
|
||||||
```
|
```
|
||||||
|
|
||||||
Graphe conceptuel :
|
Graphe matérialisé après la fondation Worker :
|
||||||
|
|
||||||
```text
|
```text
|
||||||
ksp-job-backfill-lib --------------------> ksp-raw-transaction-lib ----> ksp-store-api
|
ksp-job-backfill-lib --------------------> ksp-raw-transaction-lib ----> ksp-store-api
|
||||||
@@ -657,12 +657,14 @@ ksp-worker-raw-transaction-ingest-lib ---> ksp-raw-transaction-lib ----> ksp-sto
|
|||||||
ksp-job-backfill-lib --------------------> ksp-store-lib
|
ksp-job-backfill-lib --------------------> ksp-store-lib
|
||||||
ksp-worker-raw-transaction-ingest-lib ---> ksp-store-lib
|
ksp-worker-raw-transaction-ingest-lib ---> ksp-store-lib
|
||||||
ksp-job-backfill-lib --------------------> ksp-onchain-transport-lib
|
ksp-job-backfill-lib --------------------> ksp-onchain-transport-lib
|
||||||
ksp-worker-raw-transaction-ingest-lib ---> ksp-onchain-transport-lib
|
ksp-worker-raw-transaction-ingest-lib -X-> ksp-onchain-transport-lib # aucun adapter live productif encore
|
||||||
|
|
||||||
aucun edge Job <-> Worker
|
aucun edge Job <-> Worker
|
||||||
aucun edge Transport <-> ksp-raw-transaction-lib
|
aucun edge Transport <-> ksp-raw-transaction-lib
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Les tranches live suivantes peuvent matérialiser l'edge Worker -> Transport lorsqu'un adapter source productif le nécessite ; cet edge n'appartient pas à la fondation source-neutral.
|
||||||
|
|
||||||
La migration doit préserver exactement les golden bytes/hash RAW v1 déjà prouvés. Aucun RAW v2 n'est justifié.
|
La migration doit préserver exactement les golden bytes/hash RAW v1 déjà prouvés. Aucun RAW v2 n'est justifié.
|
||||||
|
|
||||||
### 13.1 Transaction wire Legacy/V0/V1
|
### 13.1 Transaction wire Legacy/V0/V1
|
||||||
@@ -711,9 +713,40 @@ Yellowstone Transaction/Block -> signal structuré + transaction wire fidèle
|
|||||||
RAW v1 complet -> hydration HTTP avant persistence
|
RAW v1 complet -> hydration HTTP avant persistence
|
||||||
```
|
```
|
||||||
|
|
||||||
Aucun adapter productif n'est ajouté en `pre.006`. Conformément à `TR-C2`, l'adapter `Transport DTO -> common` demeure réservé au Worker concret, désormais ouvert en `0.3.11`.
|
Aucun adapter productif n'est ajouté par la qualification cross-source. Conformément à `TR-C2`, l'adapter `Transport DTO -> common` demeure réservé au Worker concret. La fondation `0.3.11` est désormais matérialisée sans Transport ; le premier adapter productif reste donc une responsabilité des tranches live suivantes.
|
||||||
|
|
||||||
## 14. Handoff Worker live `0.3.11` à `0.3.14`
|
## 14. Handoff fondation Worker `0.3.11` vers live `0.3.12` à `0.3.14`
|
||||||
|
|
||||||
|
### 14.0 État fermé par la fondation source-neutral
|
||||||
|
|
||||||
|
La fondation du Worker est matérialisée avec :
|
||||||
|
|
||||||
|
```text
|
||||||
|
consumer de ksp-worker-api
|
||||||
|
settings réseau/Worker bornés
|
||||||
|
start/stop sur runtime Tokio caller-owned
|
||||||
|
supervision privée des tâches
|
||||||
|
mpsc central borné + backpressure
|
||||||
|
canonicalisation et assembly via ksp-raw-transaction-lib
|
||||||
|
observation key déterministe
|
||||||
|
persistence atomique via ksp-store-lib en mode Normal
|
||||||
|
concurrence Store bornée
|
||||||
|
snapshots concrete + projection WorkerSnapshotSource
|
||||||
|
faults Store/content/source/counter classifiés
|
||||||
|
shutdown deadline + abort/join sans tâche orpheline
|
||||||
|
```
|
||||||
|
|
||||||
|
Elle ne possède encore :
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun adapter HTTP/WS/Yellowstone productif
|
||||||
|
aucun edge ksp-onchain-transport-lib
|
||||||
|
aucune lecture Config
|
||||||
|
aucune API publique d'enqueue/source registration
|
||||||
|
aucun replay/gap-repair live
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette frontière est volontaire : les tranches live ajoutent des sources au supervisor existant au lieu d'introduire un second runtime ou une API d'admission publique.
|
||||||
|
|
||||||
### 14.1 Contrat fonctionnel
|
### 14.1 Contrat fonctionnel
|
||||||
|
|
||||||
@@ -937,7 +970,8 @@ Transaction V1 = wire source-neutral Legacy/V0/V1 dans common ; ac
|
|||||||
Yellowstone pre.006 = transaction wire V1 qualifié ; RAW complet via hydration HTTP
|
Yellowstone pre.006 = transaction wire V1 qualifié ; RAW complet via hydration HTTP
|
||||||
TR-C2 = adapter productif Transport DTO -> common réservé au Worker concret
|
TR-C2 = adapter productif Transport DTO -> common réservé au Worker concret
|
||||||
0.3.10 = common RAW + preuves cross-source
|
0.3.10 = common RAW + preuves cross-source
|
||||||
0.3.11 à 0.3.14 = Worker live multi-source par responsabilités bornées
|
0.3.11 = fondation Worker source-neutral, persistence et observabilité
|
||||||
|
0.3.12 à 0.3.14 = sources live multi-source, continuité et hardening par responsabilités bornées
|
||||||
0.3.16 = Job Backfill multi-stratégie historique
|
0.3.16 = Job Backfill multi-stratégie historique
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: docs/plans/002-FUNCTIONAL_RELEASE_SEQUENCE.md -->
|
<!-- file: docs/plans/002-FUNCTIONAL_RELEASE_SEQUENCE.md -->
|
||||||
<!-- version: 97 -->
|
<!-- version: 98 -->
|
||||||
|
|
||||||
# Séquence des releases fonctionnelles KSP
|
# Séquence des releases fonctionnelles KSP
|
||||||
|
|
||||||
@@ -582,7 +582,8 @@ La séquence effective a évolué par sizing et validation réels. La référenc
|
|||||||
0.3.8 ksp-app-store-desk RAW
|
0.3.8 ksp-app-store-desk RAW
|
||||||
0.3.9 ksp-worker-api + audit acquisition RawTransaction
|
0.3.9 ksp-worker-api + audit acquisition RawTransaction
|
||||||
0.3.10 common ksp-raw-transaction-lib + preuves cross-source
|
0.3.10 common ksp-raw-transaction-lib + preuves cross-source
|
||||||
0.3.11-0.3.14 Worker RAW ingest découpé par responsabilité
|
0.3.11 fondation source-neutral ksp-worker-raw-transaction-ingest-lib
|
||||||
|
0.3.12-0.3.14 sources live, convergence et continuité Worker RAW ingest
|
||||||
0.3.15 ksp-app-raw-transaction-ingest-desk
|
0.3.15 ksp-app-raw-transaction-ingest-desk
|
||||||
0.3.16 Backfill multi-source / multi-stratégie
|
0.3.16 Backfill multi-source / multi-stratégie
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -0,0 +1,790 @@
|
|||||||
|
<!-- file: docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md -->
|
||||||
|
<!-- version: 12 -->
|
||||||
|
|
||||||
|
# Plan v0.3.11 — fondation runtime du Worker RawTransaction ingest
|
||||||
|
|
||||||
|
## 1. But de la version
|
||||||
|
|
||||||
|
`0.3.11` introduit `ksp-worker-raw-transaction-ingest-lib` comme premier Worker concret KSP, mais ferme uniquement sa fondation runtime source-neutral et déterministe.
|
||||||
|
|
||||||
|
La release doit prouver :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crate + dependency firewall
|
||||||
|
settings techniques bornés
|
||||||
|
identité Worker concrète
|
||||||
|
start/stop et terminal
|
||||||
|
supervisor et tâches privés
|
||||||
|
admission bornée
|
||||||
|
canonicalisation via ksp-raw-transaction-lib
|
||||||
|
persistence via ksp-store-lib
|
||||||
|
idempotence/conflit déterministes
|
||||||
|
snapshots latest-value concrets
|
||||||
|
projection vers ksp-worker-api
|
||||||
|
shutdown/fault/backpressure bornés
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune source Yellowstone, WS, Helius ou HTTP live n'est nécessaire pour fermer `0.3.11`. Les sources réelles commencent en `0.3.12`.
|
||||||
|
|
||||||
|
## 2. Base autoritaire auditée en `pre.001`
|
||||||
|
|
||||||
|
Base opérateur :
|
||||||
|
|
||||||
|
```text
|
||||||
|
archive : khadhroony-solana-project-v0.3.10.zip
|
||||||
|
SHA-256 : befafea61304aaea15c94db7c8b3525c58b7c1622bf880a9110373bcc0ae6ac0
|
||||||
|
ZIP bytes : 7993096
|
||||||
|
ZIP entries : 1901
|
||||||
|
unzip -t : PASS
|
||||||
|
workspace.package.version : 0.3.10
|
||||||
|
workspace members : 20
|
||||||
|
deltas/0.3.10/rel.001.md : présent
|
||||||
|
ksp-raw-transaction-lib : présent
|
||||||
|
ksp-worker-api : présent
|
||||||
|
ksp-job-backfill-lib : présent et consommateur de ksp-raw-transaction-lib
|
||||||
|
ksp-worker-raw-transaction-ingest-lib : absent
|
||||||
|
```
|
||||||
|
|
||||||
|
Sécurité/forme du ZIP :
|
||||||
|
|
||||||
|
```text
|
||||||
|
une seule racine : khadhroony-solana-project
|
||||||
|
entrée absolue : 0
|
||||||
|
path traversal : 0
|
||||||
|
symlink : 0
|
||||||
|
.git/ : 0
|
||||||
|
target/ : 0
|
||||||
|
node_modules/ : 0
|
||||||
|
Cargo.lock : 0
|
||||||
|
.env : 0
|
||||||
|
```
|
||||||
|
|
||||||
|
La base satisfait donc les préconditions du prompt `030`.
|
||||||
|
|
||||||
|
## 3. Réconciliation normative obligatoire ouverte par l'audit
|
||||||
|
|
||||||
|
Deux divergences existent dans les règles actives de la stable `0.3.10`.
|
||||||
|
|
||||||
|
### 3.1 Vocabulaire D1–D4
|
||||||
|
|
||||||
|
`RULES_KSP.md` et `RULES_DEPENDENCIES.md` contenaient encore une ancienne formulation où D2 était nommé `Core canonique`, alors que les règles plus récentes et l'architecture `008` imposent :
|
||||||
|
|
||||||
|
```text
|
||||||
|
D1 RAW
|
||||||
|
D2 STRUCTURAL
|
||||||
|
D3 DECODED
|
||||||
|
D4 DOMAIN
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.001` corrige uniquement les règles actives concernées. Les deltas/plans historiques restent des traces historiques et ne sont pas réécrits.
|
||||||
|
|
||||||
|
### 3.2 Nom du Worker RAW
|
||||||
|
|
||||||
|
Les règles `KSP-WORKER-005`, `006`, `007` et `010` utilisaient encore `ksp-worker-raw-retriever`. Le nom durable retenu par l'architecture, le ROADMAP, le CHANGELOG, le handoff `0.3.10` et le prompt `030` est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.001` normalise ces quatre règles sans modifier leur responsabilité fonctionnelle.
|
||||||
|
|
||||||
|
## 4. Frontières existantes à préserver
|
||||||
|
|
||||||
|
### 4.1 `ksp-worker-api`
|
||||||
|
|
||||||
|
Surface générique conservée sans modification :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WorkerId
|
||||||
|
WorkerKindCode
|
||||||
|
WorkerState
|
||||||
|
WorkerHealth
|
||||||
|
WorkerActivity
|
||||||
|
WorkerLifecycle
|
||||||
|
WorkerStopToken
|
||||||
|
WorkerSnapshotSequence
|
||||||
|
WorkerSnapshot
|
||||||
|
WorkerSnapshotFuture
|
||||||
|
WorkerSnapshotSource
|
||||||
|
```
|
||||||
|
|
||||||
|
La crate reste runtime-neutral et dépend uniquement de `ksp-core-lib`. Aucun `start()`/`stop()` universel ni type Solana-specific n'y est ajouté pour `0.3.11`.
|
||||||
|
|
||||||
|
### 4.2 `ksp-raw-transaction-lib`
|
||||||
|
|
||||||
|
La common RAW reste propriétaire de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionMaterial
|
||||||
|
canonicalize_raw_transaction
|
||||||
|
parse/extract signature
|
||||||
|
RAW v1 wire/canonical bytes/hash
|
||||||
|
RawTransactionAcquisition
|
||||||
|
assemble_raw_transaction_acquisition
|
||||||
|
```
|
||||||
|
|
||||||
|
Elle ne reçoit aucun edge runtime/Worker/Job/Transport/Config/backend.
|
||||||
|
|
||||||
|
### 4.3 `ksp-store-lib`
|
||||||
|
|
||||||
|
Le Worker concret persiste via la façade `Store` et le contrat atomique :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionWrite::persist_raw_transaction_acquisition
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Store reste l'autorité de convergence pour `(network, signature)` et retourne les outcomes entity/observation. Le Worker ne dépend pas de `ksp-store-postgres-lib` et n'ouvre/ferme pas le Store qui lui est fourni.
|
||||||
|
|
||||||
|
### 4.4 Backfill
|
||||||
|
|
||||||
|
Les patterns réutilisables conceptuellement sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
port privé de persistence pour tests déterministes
|
||||||
|
persistence atomique entity + observation
|
||||||
|
mapping explicite du content conflict
|
||||||
|
latest-value coalescent
|
||||||
|
cancellation coopérative
|
||||||
|
redaction des surfaces publiques
|
||||||
|
```
|
||||||
|
|
||||||
|
Ne sont pas réutilisés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
BackfillRequest
|
||||||
|
scope historique
|
||||||
|
before/after
|
||||||
|
checkpoint
|
||||||
|
JobId
|
||||||
|
campagne/limit historique
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. Dépendances externes et graphe cible
|
||||||
|
|
||||||
|
Fraîcheur vérifiée le 8 septembre 2026 :
|
||||||
|
|
||||||
|
```text
|
||||||
|
tokio latest stable : 1.53.1
|
||||||
|
workspace KSP : ^1.53
|
||||||
|
futures-util latest stable : 0.3.34
|
||||||
|
workspace KSP : ^0.3
|
||||||
|
sha2 latest stable : 0.11.0
|
||||||
|
workspace KSP : ^0.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun bump de contrainte workspace n'est requis.
|
||||||
|
|
||||||
|
Graphe de production prévu pour la fondation :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
-> ksp-core-lib
|
||||||
|
-> ksp-worker-api
|
||||||
|
-> ksp-raw-transaction-lib
|
||||||
|
-> ksp-store-lib default-features=false
|
||||||
|
-> ksp-logging-lib
|
||||||
|
-> sha2
|
||||||
|
-> tokio
|
||||||
|
```
|
||||||
|
|
||||||
|
Tokio est direct parce que la crate concrète possède ses tâches runtime. Features de production retenues lorsque le runtime est matérialisé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
macros
|
||||||
|
rt
|
||||||
|
sync
|
||||||
|
time
|
||||||
|
```
|
||||||
|
|
||||||
|
Raisons :
|
||||||
|
|
||||||
|
```text
|
||||||
|
rt -> spawn / JoinHandle / JoinSet privés
|
||||||
|
sync -> mpsc borné + watch latest-value
|
||||||
|
macros -> select! avec ordre biaisé explicite pour stop/fault
|
||||||
|
time -> deadline de drain bornée
|
||||||
|
```
|
||||||
|
|
||||||
|
`rt-multi-thread` reste test-only si les tests en ont besoin ; le Worker ne crée aucun runtime.
|
||||||
|
|
||||||
|
`futures-util` n'est pas ajouté en fondation : `JoinSet`, `mpsc`, `watch`, les futures boxed standard et `select!` couvrent le modèle retenu. `ksp-onchain-transport-lib` n'est pas ajouté à `0.3.11` : aucune source live n'existe encore et un edge inutilisé violerait le principe de dépendance minimale. Il sera réaudité avec Yellowstone en `0.3.12`.
|
||||||
|
|
||||||
|
Edges interdits :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Worker -> ksp-config-lib
|
||||||
|
Worker -> ksp-job-backfill-lib
|
||||||
|
Worker -> ksp-job-api
|
||||||
|
Worker -> ksp-store-api direct sans nécessité démontrée
|
||||||
|
Worker -> ksp-store-postgres-lib
|
||||||
|
Worker -> SDK provider
|
||||||
|
common RAW -> Worker/runtime/Transport/Config
|
||||||
|
```
|
||||||
|
|
||||||
|
## 6. Surface publique minimale retenue
|
||||||
|
|
||||||
|
### 6.1 Constante d'identité
|
||||||
|
|
||||||
|
```text
|
||||||
|
RAW_TRANSACTION_INGEST_WORKER_KIND_CODE = "raw_transaction_ingest"
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction de `WorkerKindCode` reste validée par `ksp-worker-api`.
|
||||||
|
|
||||||
|
### 6.2 `RawTransactionIngestSettings`
|
||||||
|
|
||||||
|
La fondation expose seulement les réglages runtime techniques dont le Worker est propriétaire :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network: RawNetworkId
|
||||||
|
worker_id: WorkerId
|
||||||
|
admission_queue_capacity: usize
|
||||||
|
persistence_concurrency: usize
|
||||||
|
shutdown_drain_timeout: Duration
|
||||||
|
```
|
||||||
|
|
||||||
|
Bornes V1 :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission queue default : 256
|
||||||
|
admission queue min : 1
|
||||||
|
admission queue max : 65_536
|
||||||
|
|
||||||
|
persistence concurrency default : 8
|
||||||
|
persistence concurrency min : 1
|
||||||
|
persistence concurrency max : 64
|
||||||
|
|
||||||
|
shutdown drain default : 5 s
|
||||||
|
shutdown drain min : 100 ms
|
||||||
|
shutdown drain max : 30 s
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces bornes reprennent des enveloppes déjà utilisées dans KSP : queue live conservatrice à 256, plafond de channel Transport à 65 536, concurrence Store usuelle 8 et plafond 64, timeout de shutdown Store 5 s avec enveloppe 100 ms–30 s.
|
||||||
|
|
||||||
|
Aucun champ métier historique ni provider-specific n'entre dans cette structure.
|
||||||
|
|
||||||
|
### 6.3 Types différés
|
||||||
|
|
||||||
|
Ne sont pas figés publiquement en `0.3.11` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionSourceId
|
||||||
|
RawTransactionSourceSettings
|
||||||
|
RawTransactionSourceCapability
|
||||||
|
RawTransactionSourceRole
|
||||||
|
RawTransactionContinuityState
|
||||||
|
source frontier/run frontier
|
||||||
|
provider enum
|
||||||
|
Yellowstone/WS/Helius fields
|
||||||
|
retry/replay/gap-repair settings
|
||||||
|
```
|
||||||
|
|
||||||
|
Le harness déterministe utilise des types privés. Les types source publics seront ouverts lorsque `0.3.12` disposera d'une première source réelle permettant de prouver leur contrat.
|
||||||
|
|
||||||
|
## 7. Forme du start, ownership et handle
|
||||||
|
|
||||||
|
### 7.1 Start
|
||||||
|
|
||||||
|
Forme cible :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestWorker::start(
|
||||||
|
settings: RawTransactionIngestSettings,
|
||||||
|
store: Arc<ksp_store_lib::Store>,
|
||||||
|
) -> Result<RawTransactionIngestHandle>
|
||||||
|
```
|
||||||
|
|
||||||
|
`start` est synchrone et exige un runtime Tokio actif fourni par le caller. Il vérifie avant spawn :
|
||||||
|
|
||||||
|
```text
|
||||||
|
settings valides
|
||||||
|
runtime Tokio courant disponible
|
||||||
|
network settings == network Store
|
||||||
|
état initial cohérent
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker ne ferme jamais le Store ; le caller conserve son lifecycle. `Arc<Store>` permet aux tâches privées de partager la même façade sans exposer un backend.
|
||||||
|
|
||||||
|
### 7.2 Handle
|
||||||
|
|
||||||
|
Surface minimale :
|
||||||
|
|
||||||
|
```text
|
||||||
|
request_stop() -> bool
|
||||||
|
snapshot_source()
|
||||||
|
worker_snapshot_source()
|
||||||
|
wait_terminal()
|
||||||
|
```
|
||||||
|
|
||||||
|
`request_stop()` est idempotent et retourne `true` seulement pour le premier appel effectif.
|
||||||
|
|
||||||
|
`wait_terminal()` retourne une future runtime-neutral boxed appartenant à la crate concrète ; aucun `tokio::task::JoinHandle` n'apparaît dans l'API publique.
|
||||||
|
|
||||||
|
## 8. Supervisor, tâches et channels
|
||||||
|
|
||||||
|
### 8.1 Ownership privé
|
||||||
|
|
||||||
|
Le modèle retenu est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
start
|
||||||
|
-> crée stop token + signal stop privé
|
||||||
|
-> crée watch snapshot
|
||||||
|
-> crée mpsc admission borné
|
||||||
|
-> spawn supervisor privé
|
||||||
|
-> garde l'ownership des source tasks présentes/futures
|
||||||
|
-> garde l'ownership des persistence tasks
|
||||||
|
-> ferme et joint toutes les tâches avant terminal
|
||||||
|
```
|
||||||
|
|
||||||
|
Les `JoinHandle`/`JoinSet` restent strictement privés. Dropper le handle public ne détache pas le supervisor ; l'ownership interne du run reste suffisant pour atteindre un terminal ou répondre à stop par les clones encore présents.
|
||||||
|
|
||||||
|
### 8.2 Admission
|
||||||
|
|
||||||
|
Un seul channel central :
|
||||||
|
|
||||||
|
```text
|
||||||
|
tokio::sync::mpsc::channel<RawTransactionIngress>(admission_queue_capacity)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le channel est borné. `send().await` applique le backpressure ; aucun `unbounded_channel`, aucun drop silencieux et aucune event queue publique ne sont admis.
|
||||||
|
|
||||||
|
Les futures sources posséderont des clones du sender. En `0.3.11`, seuls les tests/harness privés injectent des entrées.
|
||||||
|
|
||||||
|
### 8.3 Stop signal
|
||||||
|
|
||||||
|
`WorkerStopToken` porte l'intention commune. Un `watch<bool>` privé réveille immédiatement le supervisor/source harness sans polling. Le stop branch est placé en premier dans les `select!` biaisés afin qu'aucune nouvelle admission ne soit acceptée après observation du stop.
|
||||||
|
|
||||||
|
### 8.4 Snapshot
|
||||||
|
|
||||||
|
Un `watch<RawTransactionIngestSnapshot>` privé conserve uniquement la dernière valeur. La projection commune `WorkerSnapshotSource` et la source concrète lisent ce même état ; aucune seconde event queue n'est nécessaire.
|
||||||
|
|
||||||
|
## 9. Seam/harness déterministe sans réseau
|
||||||
|
|
||||||
|
Le runtime de fondation est prouvé par un harness privé qui produit des `RawTransactionIngress` crate-private contrôlés.
|
||||||
|
|
||||||
|
Une entrée de harness contient conceptuellement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionMaterial
|
||||||
|
RawNetworkId attendu
|
||||||
|
source key privée déterministe [u8; 32]
|
||||||
|
RawAcquisitionProvenance sûre
|
||||||
|
```
|
||||||
|
|
||||||
|
Le harness permet :
|
||||||
|
|
||||||
|
```text
|
||||||
|
N acquisitions séquentielles
|
||||||
|
acquisitions concurrentes
|
||||||
|
même transaction/même observation
|
||||||
|
même transaction/nouvelle observation
|
||||||
|
contenu canonique divergent
|
||||||
|
source qui fault
|
||||||
|
source qui reste bloquée
|
||||||
|
stop pendant admission
|
||||||
|
stop pendant persistence
|
||||||
|
saturation d'admission
|
||||||
|
lecteur snapshot lent ou absent
|
||||||
|
```
|
||||||
|
|
||||||
|
Le seam n'est pas public et ne constitue pas une API d'injection de données pour les applications. `0.3.12` branchera une source productive sur le même sender privé.
|
||||||
|
|
||||||
|
## 10. Canonicalisation, observation key et persistence
|
||||||
|
|
||||||
|
### 10.1 Pipeline central
|
||||||
|
|
||||||
|
```text
|
||||||
|
private ingress
|
||||||
|
-> vérifier network
|
||||||
|
-> canonicalize_raw_transaction(material)
|
||||||
|
-> assemble_raw_transaction_acquisition(...)
|
||||||
|
-> persist_raw_transaction_acquisition(..., Normal)
|
||||||
|
-> classifier outcome
|
||||||
|
-> publier counters/snapshot
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune canonicalisation RAW v1 n'est recodée dans le Worker.
|
||||||
|
|
||||||
|
### 10.2 Observation key Worker
|
||||||
|
|
||||||
|
La Worker observation key est un SHA-256 KSP-owned versionné et domain-separated. La fondation réserve le domaine :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp.raw_transaction_ingest.observation.v1
|
||||||
|
```
|
||||||
|
|
||||||
|
Les bytes exacts/golden sont figés en `pre.006` à partir de l'identité `(network, signature)` et d'une source key privée opaque de 32 bytes. Le SHA-256 est domain-separated puis encode chaque composant avec une longueur `u64` big-endian. Aucun timestamp local aléatoire, run id ni ordre d'admission ne peut rendre une rediffusion identique non idempotente.
|
||||||
|
|
||||||
|
Les éléments provider/source utilisés pour produire cette source key ne sont pas figés publiquement avant `0.3.12`; le harness utilise une source key privée déterministe afin de prouver l'idempotence du domaine Worker sans ouvrir une API source prématurée.
|
||||||
|
|
||||||
|
### 10.3 Outcomes
|
||||||
|
|
||||||
|
Le Worker distingue au minimum :
|
||||||
|
|
||||||
|
```text
|
||||||
|
entity Inserted
|
||||||
|
entity AlreadyPresent
|
||||||
|
entity SkippedPurged
|
||||||
|
observation Inserted
|
||||||
|
observation AlreadyPresent
|
||||||
|
observation NotRecorded
|
||||||
|
content conflict
|
||||||
|
Store error
|
||||||
|
```
|
||||||
|
|
||||||
|
`Rehydrated` n'est pas produit en mode `Normal`; s'il apparaît malgré le contrat, le Worker le traite comme invariant runtime invalide plutôt que comme succès silencieux.
|
||||||
|
|
||||||
|
Le Store reste la source de vérité ; aucun cache de déduplication mémoire n'est requis en `0.3.11`.
|
||||||
|
|
||||||
|
## 11. Snapshots et projection Worker API
|
||||||
|
|
||||||
|
### 11.1 Snapshot concret
|
||||||
|
|
||||||
|
`RawTransactionIngestSnapshot` contient uniquement des données sûres :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WorkerSnapshot commun ou champs de projection équivalents
|
||||||
|
admission_queue_capacity
|
||||||
|
admission_queue_depth
|
||||||
|
persistence_concurrency
|
||||||
|
in_flight_persistence
|
||||||
|
admitted_total
|
||||||
|
canonicalized_total
|
||||||
|
persisted_total
|
||||||
|
entity_inserted_total
|
||||||
|
entity_already_present_total
|
||||||
|
entity_skipped_purged_total
|
||||||
|
observation_inserted_total
|
||||||
|
observation_already_present_total
|
||||||
|
content_conflict_total
|
||||||
|
store_failure_total
|
||||||
|
source_failure_total
|
||||||
|
backpressure_wait_total
|
||||||
|
```
|
||||||
|
|
||||||
|
Compteurs : `u64` monotones avec incrément checked. L'épuisement est un fault statique, jamais un wrap silencieux.
|
||||||
|
|
||||||
|
Aucun timestamp/frontier public n'est nécessaire en fondation. `received_at` appartient à la provenance de l'entrée source ; le run frontier et la continuité sont différés jusqu'à la première source réelle.
|
||||||
|
|
||||||
|
### 11.2 Projection commune
|
||||||
|
|
||||||
|
La même séquence `WorkerSnapshotSequence` pilote le snapshot concret et la projection commune. Mapping :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Created/Starting -> health Unknown
|
||||||
|
Running -> health Healthy tant qu'aucun fault n'est décidé
|
||||||
|
Stopping -> conserve le dernier health non terminal
|
||||||
|
Faulted -> health Unhealthy
|
||||||
|
|
||||||
|
queue ou persistence > 0 -> activity Active
|
||||||
|
sinon Running/Stopping -> activity Idle
|
||||||
|
```
|
||||||
|
|
||||||
|
La state machine reste celle de `WorkerLifecycle`.
|
||||||
|
|
||||||
|
## 12. Fault, shutdown et backpressure
|
||||||
|
|
||||||
|
### 12.1 Error codes publics prévus
|
||||||
|
|
||||||
|
```text
|
||||||
|
ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID
|
||||||
|
ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID
|
||||||
|
ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED
|
||||||
|
ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED
|
||||||
|
ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT
|
||||||
|
ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT
|
||||||
|
ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED
|
||||||
|
```
|
||||||
|
|
||||||
|
Domain ErrorCode :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest
|
||||||
|
```
|
||||||
|
|
||||||
|
Les messages/contextes restent statiques et sûrs ; aucune signature, hash de conflit, payload, URL, token ou erreur distante arbitraire n'est copiée.
|
||||||
|
|
||||||
|
### 12.2 Content conflict
|
||||||
|
|
||||||
|
Policy :
|
||||||
|
|
||||||
|
```text
|
||||||
|
fermer l'admission
|
||||||
|
request stop interne
|
||||||
|
ne pas overwrite
|
||||||
|
ne pas choisir de source gagnante
|
||||||
|
continuer seulement le drain déjà admis tant que sûr
|
||||||
|
publier Faulted(content_conflict)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le premier terminal décidé reste terminal ; une demande de stop tardive ne remplace pas le fault.
|
||||||
|
|
||||||
|
### 12.3 Store error
|
||||||
|
|
||||||
|
Une erreur Store non-conflict pendant une acquisition admise devient `Faulted(store_failed)`. Le code Store original peut être tracé uniquement s'il est déjà un `ErrorCode` sûr ; le texte d'erreur arbitraire n'est jamais propagé dans snapshot/Debug.
|
||||||
|
|
||||||
|
### 12.4 Source harness failure
|
||||||
|
|
||||||
|
Une panne de source privée devient `Faulted(source_failed)` en `0.3.11`. Les futures politiques required/optional/degraded sont reportées à la release qui possède de vraies sources.
|
||||||
|
|
||||||
|
### 12.5 Drain
|
||||||
|
|
||||||
|
Après stop/fault :
|
||||||
|
|
||||||
|
```text
|
||||||
|
fermer nouvelles admissions
|
||||||
|
signaler les tâches source
|
||||||
|
fermer les senders Worker-owned
|
||||||
|
continuer les acquisitions déjà reçues/admisses
|
||||||
|
attendre les persistence tasks jusqu'à shutdown_drain_timeout
|
||||||
|
si deadline dépassée -> abort privé des tâches restantes, join, Faulted(drain_timeout)
|
||||||
|
publier terminal seulement après absence de tâche privée survivante
|
||||||
|
```
|
||||||
|
|
||||||
|
Le timeout Tokio nécessite un runtime caller avec time driver actif ; cette précondition est documentée et testée dans les harness runtime de la crate.
|
||||||
|
|
||||||
|
## 13. Menaces et gardes de fondation
|
||||||
|
|
||||||
|
```text
|
||||||
|
payload hostile/oversized
|
||||||
|
-> common RAW/Store bounds ; aucun payload dans snapshot/log
|
||||||
|
|
||||||
|
duplicate storm
|
||||||
|
-> bounded mpsc + bounded persistence concurrency + Store idempotence
|
||||||
|
|
||||||
|
queue saturation
|
||||||
|
-> send await/backpressure observable ; aucun drop silencieux
|
||||||
|
|
||||||
|
Store lent
|
||||||
|
-> queue/concurrency bornées ; stop drain deadline
|
||||||
|
|
||||||
|
content conflict
|
||||||
|
-> fault terminal ; aucune overwrite/first-provider-wins
|
||||||
|
|
||||||
|
Store failure
|
||||||
|
-> fault statique ; aucune retry queue infinie en fondation
|
||||||
|
|
||||||
|
source task failure
|
||||||
|
-> détectée/joined ; fault sûr
|
||||||
|
|
||||||
|
task orphan
|
||||||
|
-> ownership supervisor/JoinSet ; terminal après join/abort complet
|
||||||
|
|
||||||
|
stop/fault race
|
||||||
|
-> première décision terminale cohérente ; fault d'intégrité prioritaire
|
||||||
|
|
||||||
|
slow/no snapshot reader
|
||||||
|
-> watch latest-value ; aucun impact lifecycle
|
||||||
|
|
||||||
|
secret/URL/signature leak
|
||||||
|
-> aucun champ public correspondant ; Debug redacted ; tests statiques
|
||||||
|
|
||||||
|
Job semantics leak
|
||||||
|
-> dependency/public-surface tests interdisant request/scope/checkpoint/JobId
|
||||||
|
|
||||||
|
Config leak
|
||||||
|
-> manifest/source scans ; aucun std::env/.env/ksp-config-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## 14. Preuves et suites de tests prévues
|
||||||
|
|
||||||
|
Tests unitaires :
|
||||||
|
|
||||||
|
```text
|
||||||
|
settings defaults/bounds/invalid
|
||||||
|
kind/identity
|
||||||
|
checked counters
|
||||||
|
snapshot mapping
|
||||||
|
observation-key golden privé
|
||||||
|
outcome mapping
|
||||||
|
fault precedence
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests runtime déterministes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
start sur runtime valide
|
||||||
|
start sans runtime -> erreur sûre
|
||||||
|
network Store mismatch -> erreur avant spawn
|
||||||
|
stop idempotent
|
||||||
|
terminal normal
|
||||||
|
no orphan task after terminal
|
||||||
|
bounded admission saturation/backpressure
|
||||||
|
bounded persistence concurrency
|
||||||
|
same entity/same observation idempotent
|
||||||
|
same entity/new observation conservée
|
||||||
|
content conflict terminal
|
||||||
|
Store failure terminal
|
||||||
|
source harness failure terminal
|
||||||
|
stop pendant queue/persistence
|
||||||
|
drain timeout -> abort/join -> terminal fault
|
||||||
|
slow/no listeners
|
||||||
|
common/concrete snapshot latest-value
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests externes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
dependency boundary
|
||||||
|
public API crate-root
|
||||||
|
release completeness
|
||||||
|
security hardening/redaction
|
||||||
|
source visibility crate::Item
|
||||||
|
forbidden historical request surface
|
||||||
|
manifest exactness
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun test provider live n'est requis en `0.3.11`.
|
||||||
|
|
||||||
|
## 15. Gates techniques cibles
|
||||||
|
|
||||||
|
À partir de la création de la crate :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo test -p ksp-worker-api
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-store-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
Gate technique final :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo test --workspace --all-targets --all-features
|
||||||
|
```
|
||||||
|
|
||||||
|
Une commande non exécutée n'est jamais déclarée PASS.
|
||||||
|
|
||||||
|
## 16. Hors périmètre et questions reportées
|
||||||
|
|
||||||
|
Reportés explicitement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Yellowstone live
|
||||||
|
WS standard live
|
||||||
|
Helius live
|
||||||
|
HTTP block polling live
|
||||||
|
Transport dependency productive
|
||||||
|
source capability/role publics
|
||||||
|
source required/optional policy
|
||||||
|
retry/reconnect policy provider
|
||||||
|
run frontier
|
||||||
|
continuity/gap repair
|
||||||
|
hydration productive
|
||||||
|
multi-source convergence live
|
||||||
|
hot reconfiguration desired/effective
|
||||||
|
Desk d'ingestion
|
||||||
|
Backfill multi-source
|
||||||
|
D2 STRUCTURAL
|
||||||
|
migration Store
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces éléments ne bloquent pas le gate de fondation.
|
||||||
|
|
||||||
|
## 17. Prévision souple recalibrée
|
||||||
|
|
||||||
|
### `pre.001` — audit, règles, brainstorming, sizing, plan
|
||||||
|
|
||||||
|
Budget cible : **10–15 min**. Vérifier la stable, corriger les deux divergences normatives actives, fermer le modèle runtime/dependencies/public-private/harness et créer plan + validation. Aucun Rust Worker.
|
||||||
|
|
||||||
|
### `pre.002` — crate skeleton + dependency firewall
|
||||||
|
|
||||||
|
Budget cible : **10–15 min**. Créer la crate minimale, l'ajouter au workspace, installer uniquement les edges décidés et les tests de manifest/dependency boundary. Pas de runtime comportemental.
|
||||||
|
|
||||||
|
État après matérialisation : **implémenté et gate opérateur validé**. Le manifest porte exactement les sept edges décidés (`core`, `logging`, common RAW, façade Store sans backend par défaut, Worker API, `sha2`, Tokio `macros/rt/sync/time`) ; la crate-root reste sans API ni comportement runtime et un test externe verrouille le firewall. Le gate communiqué le 8 septembre 2026 est vert sur `fmt`, audits, `check`, Clippy strict, tests de crate et les deux arbres Cargo.
|
||||||
|
|
||||||
|
### `pre.003` — identity + settings foundation
|
||||||
|
|
||||||
|
Budget cible : **10–15 min**. Kind code, settings, defaults/bounds, validation network/worker identity et erreurs correspondantes. Aucun spawn.
|
||||||
|
|
||||||
|
État après matérialisation : **implémenté et gate opérateur validé après `pre.003-fix.001`**. Le kind stable vaut `raw_transaction_ingest`; les settings portent uniquement `network`, `worker_id`, queue, concurrence de persistence et drain timeout. Les identités restent validées par leurs types propriétaires, tandis que les trois bornes techniques sont verrouillées par un code d'erreur Worker stable et des tests unitaires/externes. Le fix `pre.003-fix.001` remplace les deux propagations `?` interdites par un contrôle de flux explicite ; le gate communiqué le 8 septembre 2026 est ensuite vert sur `fmt`, audits, `check`, Clippy strict et les dix tests de la crate.
|
||||||
|
|
||||||
|
### `pre.004` — start/handle/lifecycle/terminal
|
||||||
|
|
||||||
|
Budget cible : **15–20 min**. Start sur runtime caller-owned, handle public, stop idempotent, lifecycle générique, terminal future sans JoinHandle public. Harness runtime minimal.
|
||||||
|
|
||||||
|
État après matérialisation : **implémenté et gate opérateur validé**. `RawTransactionIngestWorker::start(settings, Arc<Store>)` vérifie le runtime Tokio courant puis le réseau Store avant spawn. Le handle clonable expose uniquement `request_stop()` et `wait_terminal()` ; un `watch<bool>` privé sert au wake-up de stop et un `watch<WorkerState>` privé au terminal de fondation. Le task racine ne possédait encore aucun task enfant, aucune admission/persistence et aucun snapshot concret ; la disparition de tous les handles ferme le sender de stop et fait terminer proprement ce task. `wait_terminal()` attend le terminal puis la fermeture du sender d'état, sans exposer de `JoinHandle`. Le gate communiqué le 8 septembre 2026 est vert sur `fmt`, audits, `check`, Clippy strict, 16 tests de crate, doc-tests et les deux arbres Cargo.
|
||||||
|
|
||||||
|
Les méthodes `snapshot_source()` et `worker_snapshot_source()` prévues par la surface finale restent volontairement différées à `pre.008`, où leur contrat latest-value concret sera effectivement prouvé.
|
||||||
|
|
||||||
|
### `pre.005` — supervisor privé + ownership des tâches
|
||||||
|
|
||||||
|
Budget cible : **15–20 min**. Introduire le supervisor, les JoinSet/joins privés, intégrer le wake-up de stop déjà matérialisé à l'ownership des tâches enfants, ajouter le test source seam et prouver qu'aucune tâche enfant ne survit au terminal. Pas encore de persistence réelle.
|
||||||
|
|
||||||
|
État après matérialisation : **implémenté et gate opérateur validé**. Le task racine est désormais le supervisor privé et possède un `tokio::task::JoinSet<()>`. Un seam privé de spawn de sources reçoit le `JoinSet` et un clone du `watch<bool>` de stop ; la production utilise un seam vide tandis que les unit tests injectent des tâches coopératives déterministes. Le supervisor récole les enfants terminés pendant l'exécution puis, au stop ou à la fermeture du channel de contrôle, attend la fin de tous les enfants avant de publier `Stopped`. Un `JoinError` observé marque la supervision comme non clean ; si la fermeture de supervision est ensuite engagée, le terminal est projeté vers `Faulted(worker_raw_transaction_ingest.runtime_invalid)`. Le traitement immédiat/prioritaire des faults reste différé à `pre.009`. Aucun timeout/abort forcé n'est encore introduit : ce hardening reste réservé à `pre.009`. Le gate communiqué le 8 septembre 2026 est vert sur `fmt`, audits, `check`, Clippy strict, 18 tests de crate, doc-tests et les deux arbres Cargo.
|
||||||
|
|
||||||
|
### `pre.006` — admission bornée + canonicalisation common
|
||||||
|
|
||||||
|
Budget cible : **15–20 min**. `mpsc` borné, backpressure, ingress privé, common `RawTransactionMaterial -> RawTransaction`, observation-key domain/golden et assembly. Pas de source réseau.
|
||||||
|
|
||||||
|
État après matérialisation : **implémenté et gate opérateur validé après `pre.006-fix.001`**. Le supervisor possède un channel central `tokio::sync::mpsc` borné par `admission_queue_capacity`. Le seam source reçoit directement un clone du sender borné ainsi que le `watch<bool>` de stop ; le harness prouve `send().await` sous saturation et un `select! biased` stop-before-send, sans helper de production mort avant les vraies sources. L'ingress reste crate-private et transporte material source-neutral, network attendu, provenance sûre et une source key opaque de 32 bytes. Le Worker canonicalise exclusivement via `ksp-raw-transaction-lib`, vérifie le réseau avant/après canonicalisation, dérive l'observation key sous `ksp.raw_transaction_ingest.observation.v1` à partir de `(network, signature, source_key)`, puis assemble un `RawTransactionAcquisition` via le common RAW. `pre.006-fix.001` corrige uniquement les séparateurs de branches du `tokio::select!` de test ; le gate communiqué le 8 septembre 2026 est ensuite vert sur `fmt`, audits, `check`, Clippy strict, 22 tests de crate et doc-tests. Aucun arbre Cargo n'a été rejoué après ce fix strictement sans changement de dépendance. Aucune persistence Store, source live, Transport, snapshot concret ou policy provider n'est introduite.
|
||||||
|
|
||||||
|
### `pre.007` — persistence Store + idempotence/conflict
|
||||||
|
|
||||||
|
Budget cible : **15–20 min**. Port privé Store, mode Normal, concurrency bornée, outcomes new/idempotent/purged, observation distincte, Store error et content conflict terminal.
|
||||||
|
|
||||||
|
État après matérialisation : **implémenté et gate opérateur validé**. La crate possède désormais un port de persistence Store crate-private, implémenté par la façade `ksp-store-lib::Store`, et appelle exclusivement l'opération atomique transaction + observation en `RawTransactionAcquisitionMode::Normal`. Le supervisor possède un `JoinSet` privé de persistences ; il ne lit une nouvelle acquisition que si `in_flight < persistence_concurrency`, ce qui borne les écritures sans sémaphore ni file secondaire. Les succès distinguent `Inserted`, `AlreadyPresent`, `SkippedPurged` et les observations `Inserted`, `AlreadyPresent`, `NotRecorded`; `Rehydrated` ou toute combinaison impossible en mode Normal deviennent `runtime_invalid`. `store_api.raw_conflict` est réduit à `worker_raw_transaction_ingest.content_conflict`; toute autre erreur Store devient `worker_raw_transaction_ingest.store_failed` en ne conservant que le `ErrorCode` inférieur sûr. Le premier fault de cette tranche signale le stop privé, ferme ensuite les nouvelles admissions, draine les acquisitions déjà admises, rejoint persistences et sources, puis publie le terminal. Le gate communiqué le 8 septembre 2026 est vert sur `fmt`, audits, `check`, Clippy strict, 31 tests de crate et doc-tests. Aucun arbre Cargo n'a été requis car aucune dépendance/feature n'avait changé. Les races terminales fortes et le timeout/abort restent réservés à `pre.009`.
|
||||||
|
|
||||||
|
### `pre.008` — snapshots concrets + projection Worker API
|
||||||
|
|
||||||
|
Budget cible : **15–20 min**. `watch` latest-value, compteurs checked, common projection, slow/no listener et terminal retained.
|
||||||
|
|
||||||
|
État après matérialisation : **implémenté et gate opérateur validé après `pre.008-fix.001`**. La crate expose `RawTransactionIngestSnapshot` et `RawTransactionIngestSnapshotSource`, ce dernier implémentant directement `ksp-worker-api::WorkerSnapshotSource`. Un unique `watch<RawTransactionIngestSnapshot>` porte la valeur latest-value concrète ; `wait_terminal()` lit ce même flux. La séquence `WorkerSnapshotSequence` est identique pour la vue concrète et la projection commune. Les compteurs `u64` utilisent checked-add ; `source_failure_total` et `backpressure_wait_total` existent mais restent à zéro jusqu'à `pre.009`. Le correctif `pre.008-fix.001` a uniquement résolu `clippy::collapsible_if` dans `runtime.rs`, sans modifier comportement, dépendance ni feature. Le gate communiqué le 8 septembre 2026 est vert sur `fmt`, audits, `check`, Clippy strict, 38 tests de crate et doc-tests. Aucun arbre Cargo n'a été requis car aucune dépendance/feature n'avait changé.
|
||||||
|
|
||||||
|
### `pre.009` — hardening shutdown/backpressure/fault races
|
||||||
|
|
||||||
|
Budget cible : **15–20 min**. Drain deadline, stop/fault ordering, saturation, source failure, Store slow/failure, abort+join au timeout, no orphan tasks. Scinder immédiatement si le gate réel dépasse le budget.
|
||||||
|
|
||||||
|
État après matérialisation : **implémenté et gate opérateur validé après `pre.009-fix.003`**. Les tâches source privées retournent désormais `ksp_core_lib::Result<()>` dans le `JoinSet` du supervisor ; une erreur source ou un `JoinError` devient `worker_raw_transaction_ingest.source_failed` et incrémente `source_failure_total` sans exposer de texte provider. Le shutdown possède une deadline stricte `shutdown_drain_timeout` ; au timeout, admission, persistences et sources sont fermées/abortées puis toutes rejointes avant `Faulted(worker_raw_transaction_ingest.drain_timeout)`. Le stop reste prioritaire dans le `select!`, mais un Store fault découvert pendant le drain n'est pas masqué par un stop antérieur ; un fault terminal déjà décidé n'est pas remplacé par un stop tardif. La saturation est instrumentée de façon source-neutral au dequeue d'une queue `mpsc` pleine via `backpressure_wait_total`, sans drop ni queue non bornée. Les correctifs `pre.009-fix.001` à `.003` ont uniquement réparé les nouveaux tests (signature de `receive`, annotation du `JoinSet`, `store_guard` et `implicit_return`) sans modifier le code de production ni les dépendances. Le gate communiqué le 8 septembre 2026 est vert sur `fmt`, audits, `check`, Clippy strict, 45 tests de crate et doc-tests. Aucun arbre Cargo n'a été requis car aucune dépendance/feature n'avait changé.
|
||||||
|
|
||||||
|
### `pre.010` — hardening public/release/security
|
||||||
|
|
||||||
|
Budget cible : **10–15 min**. Tests externes exacts : API root, dependencies, historical-surface absence, redaction, error codes, module inventory et scans Config/secret/backend.
|
||||||
|
|
||||||
|
État après matérialisation : **implémenté et gate opérateur validé après `pre.010-fix.001`**. Deux suites externes nouvelles ferment la tranche sans modifier le runtime : `hardening.rs` vérifie les sept codes d'erreur publics, la redaction du `WorkerId` et des valeurs invalides, la surface de dépendances exacte, l'usage `crate::Item`, l'absence de surface historique/Backfill/retriever, les scans Config/secret/backend/Transport et l'absence de dépendance retour depuis les couches basses ; `release_completeness.rs` verrouille l'inventaire exact des huit modules de production, les vingt-quatre exports publics crate-root et la présence des canaries externes attendues. Aucun fichier `src/`, aucune dépendance et aucune feature ne changent.
|
||||||
|
|
||||||
|
### `pre.011` — gate technique final
|
||||||
|
|
||||||
|
Budget cible : **10–15 min**. Workspace tests/all-features, Clippy strict, suites ciblées, arbres normal/features, duplicate tree. Aucun nouveau scope fonctionnel.
|
||||||
|
|
||||||
|
État après matérialisation : **gate technique final opérateur validé**. Aucun code, test, manifest de crate, dépendance ou feature n’a été modifié. Le gate communiqué le 8 septembre 2026 est vert sur rustfmt check, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 56 tests Worker, `cargo test --workspace --all-targets --all-features`, les arbres Cargo Worker normal/features et `cargo tree --duplicates`. Le graphe normal Worker confirme l'absence de Transport/backend direct ; l'inventaire global de duplicates n'a révélé aucun nouvel edge/version imputable à cette tranche mécanique.
|
||||||
|
|
||||||
|
### `pre.012` — réconciliation documentaire
|
||||||
|
|
||||||
|
Budget cible : **10–15 min**. README/USAGE Worker, plan, validation et architecture/références réellement affectées. Aucun CHANGELOG/ROADMAP/prompt suivant.
|
||||||
|
|
||||||
|
État après matérialisation : **réconciliation documentaire candidate préparée avec le bump prerelease mécanique `0.3.11-pre.12`**. `README.md` racine et les références architecture réellement affectées distinguent désormais explicitement la fondation source-neutral matérialisée des sources live `0.3.12+`; les nouveaux README/USAGE de la crate documentent uniquement la surface publique et les invariants d'utilisation actuels. Aucun code, test, manifest de crate, `CHANGELOG.md`, `ROADMAP.md` ou prompt suivant n'est modifié.
|
||||||
|
|
||||||
|
### `pre.013` — préparation publication
|
||||||
|
|
||||||
|
Budget cible : **5–10 min**. Prompt `0.3.12`, CHANGELOG, ROADMAP, Cargo/delta mécaniques uniquement.
|
||||||
|
|
||||||
|
### `rel.001` — stable
|
||||||
|
|
||||||
|
Publication mécanique sans rattrapage.
|
||||||
|
|
||||||
|
## 18. Décision de sizing `pre.001`
|
||||||
|
|
||||||
|
Décision : **maintenir `0.3.11`**.
|
||||||
|
|
||||||
|
Le périmètre reste clôturable dans une seule session à condition de préserver strictement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune source live
|
||||||
|
aucun edge Transport productif
|
||||||
|
aucune modification Config
|
||||||
|
aucune migration Store
|
||||||
|
aucun gap repair/continuity live
|
||||||
|
aucune API source provider prématurée
|
||||||
|
```
|
||||||
|
|
||||||
|
Le découpage initial `pre.002`–`pre.007` du prompt a été volontairement affiné afin que chaque tranche ne mélange pas plusieurs responsabilités runtime. Si `pre.007` ou `pre.009` révèle un volume supérieur au budget, la tranche est scindée avant exécution ; si cette scission révèle qu'un sous-système live est nécessaire, la release est rescindée avant d'introduire ce nouveau scope.
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: docs/rules/RULES_DEPENDENCIES.md -->
|
<!-- file: docs/rules/RULES_DEPENDENCIES.md -->
|
||||||
<!-- version: 18 -->
|
<!-- version: 19 -->
|
||||||
|
|
||||||
# Règles des dépendances KSP
|
# Règles des dépendances KSP
|
||||||
|
|
||||||
@@ -81,7 +81,7 @@ Elles complètent les règles Rust générales et le graphe de `docs/architectur
|
|||||||
- **DEP-STORE-001** — `ksp-store-api` ne dépend pas de Program, Materializer ou Transport.
|
- **DEP-STORE-001** — `ksp-store-api` ne dépend pas de Program, Materializer ou Transport.
|
||||||
- **DEP-STORE-002** — `ksp-store-lib` dépend de `ksp-store-api`, porte la façade/runtime Store commune et peut dépendre optionnellement de crates backend compilées par feature ; il ne dépend pas des implémentations Program/Materializer/Transport.
|
- **DEP-STORE-002** — `ksp-store-lib` dépend de `ksp-store-api`, porte la façade/runtime Store commune et peut dépendre optionnellement de crates backend compilées par feature ; il ne dépend pas des implémentations Program/Materializer/Transport.
|
||||||
- **DEP-STORE-003** — Les workers/jobs spécialisés sont propriétaires des conversions entre modèles runtime et DTO persistants.
|
- **DEP-STORE-003** — Les workers/jobs spécialisés sont propriétaires des conversions entre modèles runtime et DTO persistants.
|
||||||
- **DEP-STORE-004** — Les niveaux durables sont D1 Raw, D2 Core canonique, D3 journal de matérialisation générique et D4 projections spécialisées.
|
- **DEP-STORE-004** — Les niveaux durables canoniques sont D1 `RAW`, D2 `STRUCTURAL`, D3 `DECODED` et D4 `DOMAIN` ; le journal générique de matérialisation décodée appartient à D3 et les projections/faits de domaine queryables à D4.
|
||||||
- **DEP-STORE-005** — Les replays D1 -> D2, D2 -> D3 et D3 -> D4 doivent pouvoir être exécutés indépendamment.
|
- **DEP-STORE-005** — Les replays D1 -> D2, D2 -> D3 et D3 -> D4 doivent pouvoir être exécutés indépendamment.
|
||||||
- **DEP-STORE-006** — Une notification de donnée persistée ne constitue jamais la source de vérité du backlog ; les queries Store et marqueurs durables d'idempotence/version de processor font autorité.
|
- **DEP-STORE-006** — Une notification de donnée persistée ne constitue jamais la source de vérité du backlog ; les queries Store et marqueurs durables d'idempotence/version de processor font autorité.
|
||||||
- **DEP-STORE-007** — Une notification de donnée est publiée seulement après persistence/commit réussis.
|
- **DEP-STORE-007** — Une notification de donnée est publiée seulement après persistence/commit réussis.
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: docs/rules/RULES_KSP.md -->
|
<!-- file: docs/rules/RULES_KSP.md -->
|
||||||
<!-- version: 40 -->
|
<!-- version: 41 -->
|
||||||
|
|
||||||
# Règles spécifiques à KSP
|
# Règles spécifiques à KSP
|
||||||
|
|
||||||
@@ -97,11 +97,11 @@
|
|||||||
## Niveaux durables et Store
|
## Niveaux durables et Store
|
||||||
|
|
||||||
- **KSP-DURABLE-001** — Les niveaux persistants utilisent la nomenclature D1 à D4, distincte des couches architecturales N1 à N4.
|
- **KSP-DURABLE-001** — Les niveaux persistants utilisent la nomenclature D1 à D4, distincte des couches architecturales N1 à N4.
|
||||||
- **KSP-DURABLE-002** — D1 est Raw, D2 Core canonique, D3 le journal générique de matérialisation et D4 les projections spécialisées/queryables.
|
- **KSP-DURABLE-002** — Les niveaux durables canoniques sont D1 `RAW`, D2 `STRUCTURAL`, D3 `DECODED` et D4 `DOMAIN`. D3 conserve la matérialisation générique décodée/journal durable requis ; D4 porte les faits et projections de domaine queryables lorsque leur normalisation est pertinente.
|
||||||
- **KSP-DURABLE-003** — D1/D2/D3 sont destinés à devenir fortement stables après stabilisation de la première série Store ; D4 reste plus évolutif.
|
- **KSP-DURABLE-003** — D1/D2/D3 sont destinés à devenir fortement stables après stabilisation de la première série Store ; D4 reste plus évolutif.
|
||||||
- **KSP-DURABLE-004** — Le journal D3 est durable et obligatoire ; il ne peut pas être supprimé au profit de projections D4 directes.
|
- **KSP-DURABLE-004** — Le journal D3 est durable et obligatoire ; il ne peut pas être supprimé au profit de projections D4 directes.
|
||||||
- **KSP-DURABLE-005** — Les replays D1 -> D2, D2 -> D3 et D3 -> D4 sont indépendants.
|
- **KSP-DURABLE-005** — Les replays D1 -> D2, D2 -> D3 et D3 -> D4 sont indépendants.
|
||||||
- **KSP-DURABLE-006** — Les instructions top-level et CPI restent des faits Core distincts lorsque leurs invariants/requêtes diffèrent.
|
- **KSP-DURABLE-006** — Les instructions top-level et CPI restent des faits STRUCTURAL distincts lorsque leurs invariants/requêtes diffèrent.
|
||||||
- **KSP-DURABLE-007** — D4 modélise des faits canoniques plutôt que des familles de tables par protocole lorsque les invariants sont normalisables.
|
- **KSP-DURABLE-007** — D4 modélise des faits canoniques plutôt que des familles de tables par protocole lorsque les invariants sont normalisables.
|
||||||
- **KSP-DURABLE-008** — Les temporalités blockchain et locales restent distinctes ; un `block_time` absent n'est jamais remplacé par une date locale inventée.
|
- **KSP-DURABLE-008** — Les temporalités blockchain et locales restent distinctes ; un `block_time` absent n'est jamais remplacé par une date locale inventée.
|
||||||
|
|
||||||
@@ -138,12 +138,12 @@
|
|||||||
- **KSP-WORKER-002** — Les contrats communs des workers appartiennent à `ksp-worker-api` et ne contiennent aucun contrat propre aux jobs.
|
- **KSP-WORKER-002** — Les contrats communs des workers appartiennent à `ksp-worker-api` et ne contiennent aucun contrat propre aux jobs.
|
||||||
- **KSP-WORKER-003** — `ksp-worker-api` est principalement une lifecycle API de services continus : identité, état, health, démarrage/arrêt et événements communs ; les capacités optionnelles ne deviennent universelles que si plusieurs workers les partagent réellement.
|
- **KSP-WORKER-003** — `ksp-worker-api` est principalement une lifecycle API de services continus : identité, état, health, démarrage/arrêt et événements communs ; les capacités optionnelles ne deviennent universelles que si plusieurs workers les partagent réellement.
|
||||||
- **KSP-WORKER-004** — `ksp-worker-control-lib` est une implémentation commune de gouvernance/contrôle réutilisable d'abord par les apps manager spécialisées, puis éventuellement par un futur orchestrateur/application globale ; les apps ne réimplémentent pas cette gouvernance.
|
- **KSP-WORKER-004** — `ksp-worker-control-lib` est une implémentation commune de gouvernance/contrôle réutilisable d'abord par les apps manager spécialisées, puis éventuellement par un futur orchestrateur/application globale ; les apps ne réimplémentent pas cette gouvernance.
|
||||||
- **KSP-WORKER-005** — `ksp-worker-raw-retriever` est le worker d'acquisition raw live/quasi-live. Il ne décode pas, ne matérialise pas et ne réalise pas de replay/backfill historique.
|
- **KSP-WORKER-005** — `ksp-worker-raw-transaction-ingest-lib` est le worker d'acquisition raw live/quasi-live. Il ne décode pas, ne matérialise pas et ne réalise pas de replay/backfill historique.
|
||||||
- **KSP-WORKER-006** — `ksp-worker-raw-retriever` persiste les données raw puis notifie leur disponibilité selon les contrats de données normalisés.
|
- **KSP-WORKER-006** — `ksp-worker-raw-transaction-ingest-lib` persiste les données raw puis notifie leur disponibilité selon les contrats de données normalisés.
|
||||||
- **KSP-WORKER-007** — `ksp-worker-raw-retriever` doit pouvoir faire évoluer à chaud les listeners et la sélection des données qu'il rapatrie/stocke.
|
- **KSP-WORKER-007** — `ksp-worker-raw-transaction-ingest-lib` doit pouvoir faire évoluer à chaud les listeners et la sélection des données qu'il rapatrie/stocke.
|
||||||
- **KSP-WORKER-008** — Les workers de processing ne sont pas figés à l'avance sous une chaîne globale `structural -> generic materializer -> domain projector`. RAW et STRUCTURAL peuvent disposer de workers horizontaux propres à leur couche ; à partir de DECODED, les workers/processors sont introduits au besoin avec chaque groupe fonctionnel vertical afin que décodage, matérialisation, projection spécialisée et validation d'exécution évoluent ensemble.
|
- **KSP-WORKER-008** — Les workers de processing ne sont pas figés à l'avance sous une chaîne globale `structural -> generic materializer -> domain projector`. RAW et STRUCTURAL peuvent disposer de workers horizontaux propres à leur couche ; à partir de DECODED, les workers/processors sont introduits au besoin avec chaque groupe fonctionnel vertical afin que décodage, matérialisation, projection spécialisée et validation d'exécution évoluent ensemble.
|
||||||
- **KSP-WORKER-009** — Les workers de processing utilisent notification comme wake-up mais reconstruisent leur backlog depuis le Store.
|
- **KSP-WORKER-009** — Les workers de processing utilisent notification comme wake-up mais reconstruisent leur backlog depuis le Store.
|
||||||
- **KSP-WORKER-010** — `ksp-worker-raw-retriever` distingue une configuration desired et une configuration effective lors des reconfigurations à chaud.
|
- **KSP-WORKER-010** — `ksp-worker-raw-transaction-ingest-lib` distingue une configuration desired et une configuration effective lors des reconfigurations à chaud.
|
||||||
- **KSP-WORKER-011** — Un cursor de scan est une optimisation ; les processing outcomes durables constituent la preuve qu'un input a été traité pour un processor/version/capability.
|
- **KSP-WORKER-011** — Un cursor de scan est une optimisation ; les processing outcomes durables constituent la preuve qu'un input a été traité pour un processor/version/capability.
|
||||||
- **KSP-WORKER-012** — La concurrence de processing utilise une sémantique de claim/lease récupérable après expiration/crash.
|
- **KSP-WORKER-012** — La concurrence de processing utilise une sémantique de claim/lease récupérable après expiration/crash.
|
||||||
- **KSP-WORKER-013** — Chaque worker doit pouvoir fonctionner comme service/processus indépendant afin d'être arrêté/redémarré/mis à jour sans imposer l'arrêt volontaire des autres workers.
|
- **KSP-WORKER-013** — Chaque worker doit pouvoir fonctionner comme service/processus indépendant afin d'être arrêté/redémarré/mis à jour sans imposer l'arrêt volontaire des autres workers.
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
674
prompts/031-V0_3_12_START_PROMPT.md
Normal file
674
prompts/031-V0_3_12_START_PROMPT.md
Normal file
@@ -0,0 +1,674 @@
|
|||||||
|
<!-- file: prompts/031-V0_3_12_START_PROMPT.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Prompt de démarrage `0.3.12` — Yellowstone + hydration HTTP + continuité de run du Worker `RawTransaction`
|
||||||
|
|
||||||
|
## 1. Identité de la release et base exacte requise
|
||||||
|
|
||||||
|
Ouvrir **uniquement** `0.3.12` depuis la release stable/taggée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
La base de travail fournie par l'opérateur est autoritaire sur les souvenirs de session, snippets, anciennes archives et deltas intermédiaires. Avant toute modification, vérifier au minimum :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.11
|
||||||
|
deltas/0.3.11/rel.001.md présent
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib présent et documenté
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib sans source réseau productive dans le stable 0.3.11
|
||||||
|
crates/ksp-raw-transaction-lib présent et stable
|
||||||
|
crates/ksp-onchain-transport-lib présent avec moteur Yellowstone générique
|
||||||
|
crates/ksp-store-lib présent comme seule façade Store du Worker
|
||||||
|
ksp-job-backfill-lib indépendant du Worker
|
||||||
|
```
|
||||||
|
|
||||||
|
Release ouverte :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12
|
||||||
|
```
|
||||||
|
|
||||||
|
Première livraison attendue :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.12-pre.001
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.001` est obligatoirement un gate de **lecture + audit interne/externe + brainstorming + sizing + planification**. Il ne commence pas directement l'intégration Yellowstone productive. Si le périmètre réel n'est pas clôturable dans une seule session ou si une tranche intermédiaire paraît dépasser environ 15–20 minutes de travail effectif, rescinder la release avant l'implémentation lourde.
|
||||||
|
|
||||||
|
## 2. Mission et résultat attendu
|
||||||
|
|
||||||
|
### 2.1 Mission de `0.3.12`
|
||||||
|
|
||||||
|
Étendre la fondation source-neutral de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
avec la première famille live productive, centrée sur **Yellowstone gRPC + hydration HTTP** et la continuité propre au run du Worker.
|
||||||
|
|
||||||
|
Résultat cible à réauditer pendant `pre.001` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
edge Worker -> ksp-onchain-transport-lib seulement si réellement nécessaire
|
||||||
|
adapter Yellowstone transaction/block/status vers le pipeline privé existant
|
||||||
|
réutilisation du moteur Yellowstone générique existant
|
||||||
|
projection fidèle du transaction wire vers ksp-raw-transaction-lib
|
||||||
|
hydration HTTP lorsque le matériau Yellowstone ne suffit pas au RAW v1 complet
|
||||||
|
provenance sûre de la route réellement observée
|
||||||
|
source tasks intégrées au supervisor existant
|
||||||
|
frontier de run live explicite et borné
|
||||||
|
from_slot / replay info exploités uniquement pour continuité du run
|
||||||
|
reconnexion distinguée du replay/repair
|
||||||
|
snapshots/health enrichis seulement par dimensions réellement prouvées
|
||||||
|
fixtures déterministes cross-layer
|
||||||
|
smokes live uniquement sur providers/réseaux réellement accessibles
|
||||||
|
```
|
||||||
|
|
||||||
|
`0.3.12` ne doit pas ouvrir les voies WS standard, Helius `transactionSubscribe`, HTTP live block polling multi-source complet ni le gap repair multi-source final ; ces responsabilités restent réparties sur `0.3.13` et `0.3.14`.
|
||||||
|
|
||||||
|
### 2.2 Principe de correction RAW à préserver
|
||||||
|
|
||||||
|
Le pipeline durable reste :
|
||||||
|
|
||||||
|
```text
|
||||||
|
source live
|
||||||
|
-> matériau source/protocole
|
||||||
|
-> hydration si nécessaire
|
||||||
|
-> ksp-raw-transaction-lib
|
||||||
|
-> RawTransaction + RawTransactionObservation
|
||||||
|
-> ksp-store-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune source ne doit inventer un RAW complet à partir d'un signal incomplet.
|
||||||
|
|
||||||
|
Décision conservative déjà qualifiée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Yellowstone Transaction/Block -> transaction wire fidèle + métadonnées disponibles
|
||||||
|
RAW v1 complet -> hydration HTTP tant que block_time/meta parity n'est pas prouvée suffisante
|
||||||
|
```
|
||||||
|
|
||||||
|
Si `pre.001` démontre qu'un sous-ensemble Yellowstone peut désormais produire un RAW v1 direct sans hydration, cette évolution doit être prouvée par fixture/canari byte-exact avant de modifier cette décision.
|
||||||
|
|
||||||
|
## 3. Sources de vérité internes obligatoires — ordre de lecture
|
||||||
|
|
||||||
|
### 3.1 Gouvernance générale
|
||||||
|
|
||||||
|
Lire intégralement, dans cet ordre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RULES.md
|
||||||
|
ROADMAP.md
|
||||||
|
CHANGELOG.md
|
||||||
|
docs/000-README.md
|
||||||
|
|
||||||
|
docs/rules/RULES_GENERAL.md
|
||||||
|
docs/rules/RULES_KSP.md
|
||||||
|
docs/rules/RULES_RUST.md
|
||||||
|
docs/rules/RULES_DEPENDENCIES.md
|
||||||
|
docs/rules/RULES_DOCUMENTATION.md
|
||||||
|
docs/rules/FILE_CONTRACTS.md
|
||||||
|
docs/rules/VERSION_WORKFLOW.md
|
||||||
|
docs/rules/PROMPT_STRUCTURE.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Le prompt complète ces règles ; il ne les remplace pas.
|
||||||
|
|
||||||
|
Rappels Rust bloquants :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Rust 2024
|
||||||
|
unsafe interdit
|
||||||
|
unwrap / expect / panic interdits selon les règles KSP
|
||||||
|
? interdit en production
|
||||||
|
retours explicites ; clippy::implicit_return deny
|
||||||
|
#![warn(missing_docs)]
|
||||||
|
#![deny(unreachable_pub)]
|
||||||
|
#![forbid(unsafe_code)]
|
||||||
|
|
||||||
|
pas de pub mod
|
||||||
|
pub/pub(crate) partagés reexportés jusqu'à la crate root
|
||||||
|
accès partagés via crate::Item, y compris intra-crate
|
||||||
|
item strictement module-local => private
|
||||||
|
unit tests sous unit_tests/
|
||||||
|
integration tests sous tests/
|
||||||
|
```
|
||||||
|
|
||||||
|
Après toute modification Rust :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
```
|
||||||
|
|
||||||
|
Pour tout Markdown touché :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
```
|
||||||
|
|
||||||
|
Une commande non exécutée n'est jamais déclarée PASS.
|
||||||
|
|
||||||
|
### 3.2 Handoff stable `0.3.11`
|
||||||
|
|
||||||
|
Lire intégralement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/README.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/
|
||||||
|
```
|
||||||
|
|
||||||
|
La fondation stable est une contrainte, pas un brouillon à remplacer.
|
||||||
|
|
||||||
|
### 3.3 Acquisition RAW et qualification cross-source
|
||||||
|
|
||||||
|
Lire intégralement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md
|
||||||
|
docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
crates/ksp-raw-transaction-lib/README.md
|
||||||
|
crates/ksp-raw-transaction-lib/USAGE.md
|
||||||
|
crates/ksp-raw-transaction-lib/src/
|
||||||
|
crates/ksp-raw-transaction-lib/tests/
|
||||||
|
```
|
||||||
|
|
||||||
|
Porter une attention particulière aux sections de `011` concernant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Yellowstone transactions / transactions_status / blocks / blocks_meta / slots
|
||||||
|
from_slot
|
||||||
|
SubscribeReplayInfo / continuity snapshots
|
||||||
|
hydration HTTP
|
||||||
|
reconnexion != replay
|
||||||
|
Worker continuity repair != campagne historique
|
||||||
|
qualification transaction wire Legacy/V0/V1
|
||||||
|
limites de la preuve RAW-direct Yellowstone actuelle
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.4 Transport Yellowstone et HTTP existants
|
||||||
|
|
||||||
|
Lire au minimum :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-onchain-transport-lib/Cargo.toml
|
||||||
|
crates/ksp-onchain-transport-lib/README.md
|
||||||
|
crates/ksp-onchain-transport-lib/USAGE.md
|
||||||
|
crates/ksp-onchain-transport-lib/src/grpc_*.rs
|
||||||
|
crates/ksp-onchain-transport-lib/src/rpc_transactions.rs
|
||||||
|
crates/ksp-onchain-transport-lib/src/rpc_blocks.rs
|
||||||
|
crates/ksp-onchain-transport-lib/tests/
|
||||||
|
```
|
||||||
|
|
||||||
|
Réutiliser le moteur générique existant ; ne pas créer un second client Yellowstone, un actor gRPC Worker parallèle ou un SDK provider.
|
||||||
|
|
||||||
|
Les primitives HTTP observées existantes à réauditer incluent notamment :
|
||||||
|
|
||||||
|
```text
|
||||||
|
get_transaction_observed
|
||||||
|
get_block_observed
|
||||||
|
```
|
||||||
|
|
||||||
|
L'hydration doit conserver la provenance réellement observée du winner HTTP plutôt qu'un endpoint supposé.
|
||||||
|
|
||||||
|
### 3.5 Store, Worker API et Config comme frontières
|
||||||
|
|
||||||
|
Lire au minimum :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-store-lib/README.md
|
||||||
|
crates/ksp-store-lib/USAGE.md
|
||||||
|
crates/ksp-store-lib/src/
|
||||||
|
crates/ksp-worker-api/README.md
|
||||||
|
crates/ksp-worker-api/USAGE.md
|
||||||
|
crates/ksp-worker-api/src/
|
||||||
|
crates/ksp-config-lib/README.md
|
||||||
|
crates/ksp-config-lib/USAGE.md
|
||||||
|
crates/ksp-config-lib/src/
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker continue d'écrire via `ksp-store-lib` uniquement.
|
||||||
|
|
||||||
|
`ksp-config-lib` reste propriétaire des profils, endpoints, providers/protocoles, secrets et activation/composition. **Le Worker ne lit jamais Config ou l'environnement directement.** L'éventuelle adaptation Config de `0.3.12` doit être décidée après audit de l'injection de ressources runtime ; ne pas créer automatiquement un edge Worker -> Config.
|
||||||
|
|
||||||
|
## 4. Sources externes normatives et fraîcheur à réauditer
|
||||||
|
|
||||||
|
La fraîcheur est importante dans `0.3.12`. `pre.001` doit consulter les sources primaires courantes avant toute décision d'API ou de capability :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Yellowstone gRPC upstream : proto geyser + solana-storage + README + changelog
|
||||||
|
Solana JSON-RPC officiel : getTransaction / getBlock et champs nécessaires à l'hydration
|
||||||
|
providers Yellowstone réellement envisagés/accessibles : documentation endpoint/auth/replay/from_slot
|
||||||
|
versions stables actuelles des crates externes touchées
|
||||||
|
```
|
||||||
|
|
||||||
|
Réauditer en particulier :
|
||||||
|
|
||||||
|
```text
|
||||||
|
shape courante SubscribeRequest/from_slot
|
||||||
|
SubscribeReplayInfo et sémantique de replay exposée par le moteur/proto courant
|
||||||
|
Transaction / TransactionStatus / Block / BlockMeta wire actuel
|
||||||
|
profondeur et disponibilité réelle du replay provider-specific
|
||||||
|
réseaux réellement servis
|
||||||
|
mode d'authentification actuel
|
||||||
|
limitations/tier actuels
|
||||||
|
```
|
||||||
|
|
||||||
|
Ne pas recopier comme vérité actuelle les prix, limites ou profondeurs de replay historiques présents dans `docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md`. Cette architecture conserve les décisions KSP ; les paramètres provider temporels doivent être revérifiés.
|
||||||
|
|
||||||
|
Préférer les sources primaires. Aucun SDK provider n'est ajouté pour une simple compatibilité Yellowstone standard.
|
||||||
|
|
||||||
|
## 5. État validé à préserver
|
||||||
|
|
||||||
|
### 5.1 Fondation Worker stable
|
||||||
|
|
||||||
|
Le stable `0.3.11` possède déjà :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestSettings bornés
|
||||||
|
RawTransactionIngestWorker::start
|
||||||
|
RawTransactionIngestHandle
|
||||||
|
request_stop idempotent
|
||||||
|
terminal future
|
||||||
|
supervisor privé
|
||||||
|
JoinSet sources/persistences privés
|
||||||
|
mpsc central borné
|
||||||
|
backpressure observée
|
||||||
|
canonicalisation/assembly Common RAW
|
||||||
|
persistence atomique Store mode Normal
|
||||||
|
content conflict terminal
|
||||||
|
snapshots concrete + WorkerSnapshotSource
|
||||||
|
source/store/counter/drain faults classifiés
|
||||||
|
shutdown deadline + abort/join avant terminal timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
`0.3.12` étend cette fondation ; il ne crée pas un second lifecycle, un second pipeline d'admission ou un second système de snapshots.
|
||||||
|
|
||||||
|
### 5.2 Dependency firewall
|
||||||
|
|
||||||
|
État `0.3.11` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Worker -> ksp-core-lib
|
||||||
|
Worker -> ksp-logging-lib
|
||||||
|
Worker -> ksp-raw-transaction-lib
|
||||||
|
Worker -> ksp-store-lib default-features=false
|
||||||
|
Worker -> ksp-worker-api
|
||||||
|
Worker -> sha2
|
||||||
|
Worker -> tokio
|
||||||
|
Worker -X-> ksp-onchain-transport-lib
|
||||||
|
Worker -X-> ksp-config-lib
|
||||||
|
Worker -X-> ksp-job-backfill-lib
|
||||||
|
Worker -X-> backend Store physique
|
||||||
|
```
|
||||||
|
|
||||||
|
`0.3.12` peut matérialiser l'edge `Worker -> ksp-onchain-transport-lib` si `pre.001` confirme qu'il est la frontière correcte pour la source Yellowstone/hydration. Aucun autre edge interdit n'est ouvert par simple commodité.
|
||||||
|
|
||||||
|
### 5.3 Identité et persistence
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransaction identity = (network, signature)
|
||||||
|
mainnet = identité KSP canonique
|
||||||
|
mainnet-beta = alias legacy/externe seulement
|
||||||
|
```
|
||||||
|
|
||||||
|
La convergence Store reste : une entité canonique + observations/provenances distinctes. Un contenu canonique divergent reste `content_conflict`, jamais une règle « premier provider gagne » ou « dernier provider gagne ».
|
||||||
|
|
||||||
|
### 5.4 Séparation Job / Worker
|
||||||
|
|
||||||
|
```text
|
||||||
|
Job Backfill = historique paramétré, borné, terminable
|
||||||
|
Worker Ingest = acquisition continue start/stop, sans requête historique métier
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker peut utiliser `from_slot`/replay pour restaurer **sa propre continuité de run**. Cela ne l'autorise pas à recevoir un scope historique arbitraire du caller ni à appeler `ksp-job-backfill-lib`.
|
||||||
|
|
||||||
|
## 6. Décisions acquises et questions réellement ouvertes
|
||||||
|
|
||||||
|
### 6.1 Décisions acquises
|
||||||
|
|
||||||
|
Conserver :
|
||||||
|
|
||||||
|
```text
|
||||||
|
un seul runtime Worker existant
|
||||||
|
un seul pipeline admission -> common RAW -> Store
|
||||||
|
sources réseau privées au supervisor
|
||||||
|
aucune API publique enqueue arbitraire
|
||||||
|
Transport générique réutilisé, aucun client Yellowstone dupliqué
|
||||||
|
HTTP est une capability normale du Worker pour hydration/repair live
|
||||||
|
Config/secrets restent hors Worker
|
||||||
|
provenance HTTP = route réellement observée
|
||||||
|
channels bornés
|
||||||
|
aucun drop silencieux
|
||||||
|
reconnect gRPC != replay/continuité restaurée
|
||||||
|
from_slot/replay du Worker limité à son frontier de run
|
||||||
|
Job et Worker indépendants
|
||||||
|
pas de RAW-direct Yellowstone complet sans preuve nouvelle
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6.2 Questions à fermer dans `pre.001`
|
||||||
|
|
||||||
|
Auditer avant codage :
|
||||||
|
|
||||||
|
```text
|
||||||
|
forme exacte d'injection des ressources Transport dans RawTransactionIngestWorker::start ou une API voisine
|
||||||
|
si la surface publique actuelle doit évoluer ou si un objet runtime/source bundle privé suffit
|
||||||
|
ownership exact des source ids/capabilities/priorités
|
||||||
|
rôle de Config dans 0.3.12 et direction de dépendance correcte
|
||||||
|
quelles familles Yellowstone sont P0 réellement productives : transactions, blocks, status, blocks_meta, slots
|
||||||
|
stratégie d'hydration transaction vs block
|
||||||
|
clé de coalescence entre signal Yellowstone et résultat HTTP hydraté
|
||||||
|
provenance finale quand signal et hydration proviennent de transports/providers distincts
|
||||||
|
frontier exact : slot observed, slot persisted, contiguous slot, ou plusieurs dimensions
|
||||||
|
sémantique exacte de from_slot au démarrage/reconnect
|
||||||
|
traitement de SubscribeReplayInfo et absence de preuve provider
|
||||||
|
états source health/degraded/unhealthy/faulted réellement nécessaires
|
||||||
|
politique lorsque l'hydration renvoie missing/not-available temporaire
|
||||||
|
bornes de retry/backoff et ownership entre Transport et Worker
|
||||||
|
stop pendant reconnect/replay/hydration
|
||||||
|
besoin de nouveaux compteurs/snapshot fields sans fuite de matériel
|
||||||
|
smokes live réellement accessibles et secrets opérateur disponibles
|
||||||
|
```
|
||||||
|
|
||||||
|
Ne pas figer ces réponses dans l'API avant le gate `pre.001`.
|
||||||
|
|
||||||
|
## 7. Objectifs/livrables et hors périmètre
|
||||||
|
|
||||||
|
### 7.1 Livrables attendus de `0.3.12`
|
||||||
|
|
||||||
|
Sous réserve du sizing `pre.001` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
adapter(s) Yellowstone productifs dans le Worker
|
||||||
|
projection Transport DTO -> common wire/material dans le producer Worker
|
||||||
|
hydration HTTP explicite lorsque requise
|
||||||
|
source tasks Yellowstone intégrées au supervisor existant
|
||||||
|
provenance sûre source + hydration
|
||||||
|
frontier de run et continuité bornée
|
||||||
|
from_slot/replay info exploités sans campagne historique
|
||||||
|
snapshots/health adaptés seulement si nécessaire
|
||||||
|
fixtures déterministes transaction/block/status/hydration
|
||||||
|
canaris de dependency firewall et redaction
|
||||||
|
smokes live opt-in sur accès réellement disponible
|
||||||
|
plan 0.3.12
|
||||||
|
validation 0.3.12
|
||||||
|
README/USAGE réconciliés en fin de release
|
||||||
|
```
|
||||||
|
|
||||||
|
Le `USAGE.md` reste version-neutral.
|
||||||
|
|
||||||
|
### 7.2 Hors périmètre de `0.3.12`
|
||||||
|
|
||||||
|
```text
|
||||||
|
WS logsSubscribe productif
|
||||||
|
WS blockSubscribe productif dans le Worker
|
||||||
|
Helius transactionSubscribe productif
|
||||||
|
HTTP live block polling multi-source complet
|
||||||
|
convergence simultanée de toutes les familles 0.3.13
|
||||||
|
gap repair multi-source complet 0.3.14
|
||||||
|
source redondante/failover final
|
||||||
|
EARLY shreds/deshred/feed vendor-specific
|
||||||
|
Desk d'ingestion 0.3.15
|
||||||
|
Backfill multi-source 0.3.16
|
||||||
|
D2 STRUCTURAL
|
||||||
|
backend Store direct
|
||||||
|
nouvelle migration Store sauf défaut réellement démontré et release rescindée si nécessaire
|
||||||
|
```
|
||||||
|
|
||||||
|
Si une responsabilité de `0.3.13+` devient indispensable pour rendre Yellowstone correct, `pre.001` doit expliquer le couplage et rescinder la trajectoire avant développement lourd.
|
||||||
|
|
||||||
|
## 8. Contraintes sécurité/API/architecture spécifiques
|
||||||
|
|
||||||
|
Le runtime live doit garantir au minimum :
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune URL/token/header/API key dans Debug/Display/snapshot/error public
|
||||||
|
aucun payload transaction/meta/log provider dans diagnostics
|
||||||
|
aucun remote status/message arbitraire recopié dans Error context
|
||||||
|
source ids/codes bornés et non secrets
|
||||||
|
provenance safe sans endpoint secret
|
||||||
|
frontier/replay state sans signature/payload
|
||||||
|
counter arithmetic checked/non-wrapping
|
||||||
|
queues bornées
|
||||||
|
aucun task leak après terminal
|
||||||
|
stop préemptif pendant connect/reconnect/hydration/replay
|
||||||
|
pas de double retry incontrôlé Transport + Worker
|
||||||
|
pas de busy loop en indisponibilité provider
|
||||||
|
pas de claim de continuité si un gap n'est pas prouvé réparé
|
||||||
|
```
|
||||||
|
|
||||||
|
L'hydration doit être explicitement classifiée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
signal incomplet -> pending hydration
|
||||||
|
hydration complète -> admission RAW
|
||||||
|
missing temporaire -> état/retry borné défini par la tranche
|
||||||
|
matériau contradictoire -> fault/content conflict selon frontière réellement concernée
|
||||||
|
stop -> aucune nouvelle hydration/admission après frontière de stop
|
||||||
|
```
|
||||||
|
|
||||||
|
Le tracing reste via `ksp-logging-lib` et n'expose jamais la signature, le raw body, la meta, l'URL ou le credential.
|
||||||
|
|
||||||
|
## 9. Première mission `pre.001` — audit/sizing obligatoire
|
||||||
|
|
||||||
|
`pre.001` doit produire **avant toute implémentation lourde** :
|
||||||
|
|
||||||
|
1. vérification exacte de la base stable `v0.3.11` et du `rel.001` ;
|
||||||
|
2. lecture complète des sources internes listées ci-dessus ;
|
||||||
|
3. inventaire réel des surfaces Worker/Transport/Common RAW/Store/Config ;
|
||||||
|
4. audit externe courant Yellowstone/Solana/provider et versions de crates concernées ;
|
||||||
|
5. matrice exacte `Transaction / TransactionStatus / Block / BlockMeta / Slot / from_slot / ReplayInfo` : matériau disponible, gaps RAW, besoin d'hydration, preuve actuelle ;
|
||||||
|
6. architecture d'injection des ressources Transport sans Worker -> Config ni API publique enqueue ;
|
||||||
|
7. modèle de frontier/continuité du run et distinction reconnect/replay/repair ;
|
||||||
|
8. stratégie d'hydration HTTP et provenance cross-transport ;
|
||||||
|
9. threat model live : secrets, remote errors, saturation, duplicate signals, retry storms, reconnect races, stop during hydration/replay, stale frontier ;
|
||||||
|
10. inventaire des smokes réellement accessibles et conditions opérateur ;
|
||||||
|
11. graphe Cargo cible et features minimales ;
|
||||||
|
12. sizing de chaque tranche sous le budget 15–20 minutes ;
|
||||||
|
13. décision explicite : `0.3.12` reste clôturable dans une session ou doit être rescindée avant codage ;
|
||||||
|
14. création/révision des documents de release.
|
||||||
|
|
||||||
|
Documents attendus :
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/plans/033-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY_PLAN.md
|
||||||
|
docs/validation/029-V0_3_12_YELLOWSTONE_HYDRATION_CONTINUITY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Le delta `pre.001` consigne les décisions fermées, les questions reportées, les dépendances réellement requises, les gates live possibles et le forecast recalibré.
|
||||||
|
|
||||||
|
Critère de sortie : la première tranche technique doit pouvoir être implémentée sans décider simultanément l'API source, l'hydration, le frontier et la stratégie de replay pendant le codage.
|
||||||
|
|
||||||
|
## 10. Prévision souple initiale des prereleases
|
||||||
|
|
||||||
|
Cette prévision est un **point de départ à recalibrer par `pre.001`**, pas un engagement de numérotation.
|
||||||
|
|
||||||
|
### `pre.001` — audit / brainstorming / sizing
|
||||||
|
|
||||||
|
Base stable, matrice Yellowstone, fraîcheur provider/proto, injection Transport, hydration, frontier, replay semantics, threat model, smokes accessibles, plan/validation et décision de maintien/rescission.
|
||||||
|
|
||||||
|
### `pre.002` — dependency/source contract
|
||||||
|
|
||||||
|
Matérialiser uniquement les edges/types runtime réellement nécessaires, en particulier l'edge Worker -> Transport si confirmé, sans encore ouvrir toutes les familles Yellowstone.
|
||||||
|
|
||||||
|
### `pre.003` — adapter Yellowstone transaction
|
||||||
|
|
||||||
|
Brancher la première tâche source `transactions`, projeter le transaction wire vers Common et prouver les gaps qui imposent l'hydration. Aucun RAW-direct non prouvé.
|
||||||
|
|
||||||
|
### `pre.004` — hydration HTTP transaction
|
||||||
|
|
||||||
|
Fermer signal Yellowstone -> `get_transaction_observed` -> common RAW -> admission/persistence, avec provenance sûre et comportement missing/error borné.
|
||||||
|
|
||||||
|
### `pre.005` — blocks / block metadata
|
||||||
|
|
||||||
|
Ajouter les formes Yellowstone block/block_meta réellement nécessaires, réutiliser la même canonicalisation et comparer leur matériau à l'hydration HTTP sans dupliquer le pipeline.
|
||||||
|
|
||||||
|
### `pre.006` — transaction status + source supervision
|
||||||
|
|
||||||
|
Intégrer `transactions_status` uniquement pour le rôle réellement démontré (discovery/status/hydration), source health et failures sans exposer de matériau provider.
|
||||||
|
|
||||||
|
### `pre.007` — frontier de run + from_slot / replay info
|
||||||
|
|
||||||
|
Matérialiser la continuité propre au run : frontier explicite, reconnexion distincte du replay, `from_slot` et replay info utilisés uniquement lorsqu'ils sont réellement supportés/provables.
|
||||||
|
|
||||||
|
### `pre.008` — races/retry/backpressure hardening live
|
||||||
|
|
||||||
|
Stop pendant connect/reconnect/hydration/replay, saturation, duplicate signals, retry ownership, Store slow/fault et absence de tâches orphelines. Scinder si nécessaire.
|
||||||
|
|
||||||
|
### `pre.009` — fixtures/cross-layer completeness
|
||||||
|
|
||||||
|
Canaris déterministes exacts sur les familles Yellowstone admises, hydration, provenance, common RAW, API root, dependency firewall et security scans.
|
||||||
|
|
||||||
|
### `pre.010` — gate live/technique final
|
||||||
|
|
||||||
|
Smokes accessibles si réellement configurés, workspace complet, Clippy strict, suites ciblées, arbres Cargo Worker normal/features et duplicate tree. Aucun nouveau scope fonctionnel.
|
||||||
|
|
||||||
|
### `pre.011` — réconciliation documentaire
|
||||||
|
|
||||||
|
README/USAGE, plan, validation et architecture/références réellement affectées. Aucun CHANGELOG/ROADMAP/prompt suivant.
|
||||||
|
|
||||||
|
### `pre.012` — préparation de publication
|
||||||
|
|
||||||
|
Prompt `0.3.13`, CHANGELOG, ROADMAP et fichiers mécaniques uniquement.
|
||||||
|
|
||||||
|
### `rel.001`
|
||||||
|
|
||||||
|
Publication stable mécanique sans rattrapage.
|
||||||
|
|
||||||
|
Si `pre.001` conclut que les familles Yellowstone + hydration + continuité dépassent une session, rescinder `0.3.12` avant `pre.002` au lieu de forcer ce forecast.
|
||||||
|
|
||||||
|
## 11. Versionnement, deltas, commits et tags
|
||||||
|
|
||||||
|
Règles obligatoires :
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison prerelease : 0.3.12-pre.NNN
|
||||||
|
Cargo : 0.3.12-pre.N
|
||||||
|
fix code/runtime : 0.3.12-pre.N.fix.M
|
||||||
|
fix doc-only : ne bump pas Cargo
|
||||||
|
delta : deltas/0.3.12/pre.NNN.md ou pre.NNN-fix.NNN.md
|
||||||
|
commit : v0.3.12-pre.NNN[-fix.NNN]
|
||||||
|
tag prerelease : aucun
|
||||||
|
tag stable final : v0.3.12 seulement après rel.001 validée
|
||||||
|
```
|
||||||
|
|
||||||
|
Toute nouvelle prerelease non-fix synchronise `workspace.package.version`, même documentaire. Tout fichier modifié incrémente son header de version selon les règles du dépôt.
|
||||||
|
|
||||||
|
Les archives d'échange restent des deltas minimaux.
|
||||||
|
|
||||||
|
## 12. Procédure d'application et validation opérateur
|
||||||
|
|
||||||
|
Après application d'un delta technique :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
```
|
||||||
|
|
||||||
|
Puis exécuter les tests ciblés de la tranche et conserver les logs opérateur réels. Une commande non lancée n'est jamais déclarée PASS.
|
||||||
|
|
||||||
|
Si un gate révèle un défaut, créer un fix appartenant strictement à la responsabilité de la tranche fautive. Ne pas absorber un défaut runtime dans le couloir documentaire ou publication.
|
||||||
|
|
||||||
|
## 13. Validations Rust / Transport / live / graphes pertinentes
|
||||||
|
|
||||||
|
À partir de l'activation Transport dans le Worker, le gate cible inclut progressivement :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-store-lib
|
||||||
|
cargo test -p ksp-worker-api
|
||||||
|
```
|
||||||
|
|
||||||
|
Selon les changements Config réellement retenus :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo test -p ksp-config-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Graphes à auditer lorsque l'edge Transport ou ses features sont matérialisés :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
À la fermeture technique :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo test --workspace --all-targets --all-features
|
||||||
|
```
|
||||||
|
|
||||||
|
Les smokes live sont **opt-in** et ne lisent jamais de secrets depuis le code/source. Le prompt `pre.001` doit déterminer ceux qui sont réellement accessibles. Une incapacité de tier/provider doit être documentée comme telle, pas transformée en fausse preuve PASS.
|
||||||
|
|
||||||
|
## 14. Critères de clôture de `0.3.12`
|
||||||
|
|
||||||
|
La release peut se fermer lorsque, sur la base réellement obtenue :
|
||||||
|
|
||||||
|
```text
|
||||||
|
fondation Worker 0.3.11 préservée sans second runtime
|
||||||
|
edge Transport minimal et justifié
|
||||||
|
au moins les familles Yellowstone retenues par pre.001 intégrées productivement
|
||||||
|
projection transaction wire -> common prouvée
|
||||||
|
hydration HTTP appliquée partout où le RAW complet l'exige
|
||||||
|
aucun RAW-direct Yellowstone revendiqué sans preuve byte/semantic exacte
|
||||||
|
provenance source/hydration sûre
|
||||||
|
frontier de run explicite et borné
|
||||||
|
reconnect distinct de replay/continuité
|
||||||
|
from_slot/replay info utilisés sans transformer le Worker en moteur historique
|
||||||
|
stop/fault/backpressure pendant tâches live durcis
|
||||||
|
aucune URL/token/remote payload dans diagnostics
|
||||||
|
Store uniquement via ksp-store-lib
|
||||||
|
aucun edge Worker -> Config/Job/backend physique
|
||||||
|
fixtures et canaris cross-layer verts
|
||||||
|
smokes accessibles exécutés ou blocage provider/tier explicitement qualifié
|
||||||
|
gates workspace/clippy/tests/trees verts
|
||||||
|
README/USAGE/plan/validation réconciliés
|
||||||
|
prompt 0.3.13 produit dans la dernière prerelease
|
||||||
|
```
|
||||||
|
|
||||||
|
## 15. Release suivante envisagée
|
||||||
|
|
||||||
|
`0.3.13` doit reprendre uniquement après publication stable de `0.3.12` et réauditer la mission :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WS logsSubscribe + HTTP hydration
|
||||||
|
WS blockSubscribe direct sur le sous-ensemble déjà qualifié
|
||||||
|
Helius transactionSubscribe + hydration
|
||||||
|
HTTP live block polling
|
||||||
|
convergence simultanée multi-source
|
||||||
|
observations multiples/coalescence
|
||||||
|
backpressure et content conflict sous sources concurrentes
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gap repair/hardening multi-source final reste `0.3.14`.
|
||||||
|
|
||||||
|
## 16. Instruction d'ouverture
|
||||||
|
|
||||||
|
Au début de la prochaine session :
|
||||||
|
|
||||||
|
1. vérifier que la base correspond exactement au tag stable `v0.3.11` et que `deltas/0.3.11/rel.001.md` est présent ;
|
||||||
|
2. lire les règles, le handoff `0.3.11`, l'architecture RAW acquisition et les sources Worker/Transport/Common/Store avant toute modification ;
|
||||||
|
3. réauditer les sources Yellowstone/Solana/provider actuelles et les versions de dépendances réellement concernées ;
|
||||||
|
4. produire `0.3.12-pre.001` avec matrice Yellowstone, architecture d'hydration/frontier, threat model, sizing, plan `033` et validation `029` ;
|
||||||
|
5. **ne pas ajouter l'edge Transport au Worker, ne pas coder une source Yellowstone productive et ne pas modifier Config avant fermeture de ce gate** ;
|
||||||
|
6. rescinder `0.3.12` immédiatement si le périmètre réel n'est pas clôturable dans une seule session.
|
||||||
Reference in New Issue
Block a user