Sasedev/khadhroony-solana-project
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.3.15-pre.009

Browse Source
This commit is contained in:
SinuS von SifriduS
2026-09-14 11:38:55 +02:00
parent d3eea1aa12
commit 11105fac28
26 changed files with 1194 additions and 293 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
crates/ksp-app-raw-transaction-ingest-desk/README.md
View File

@@ -1,5 +1,5 @@
<!-- file: crates/ksp-app-raw-transaction-ingest-desk/README.md -->
<!-- version: 9 -->
<!-- version: 10 -->
# `ksp-app-raw-transaction-ingest-desk`
@@ -29,7 +29,7 @@ L'inventaire de routes est calculé côté Rust à partir du composite Config va
Au Start, le backend revalide une sélection logique par `profile_id + route_id + inventory_generation + commitment`. Il recharge Config, reproouve le réseau Store/Transport, reconstruit le pool HTTP ou l'endpoint WS/gRPC exact requis, ouvre le Store via `ksp-store-lib`, exige un health `Ready` puis lance le Worker exact via `RawTransactionIngestWorker::start_with_runtime_resources`. Les ressources physiques et handles restent exclusivement côté Rust.
Le runtime est actuellement mono-route : un seul Start peut posséder le slot applicatif. Le handle Worker est installé avant l'accusé Start. Stop demande un arrêt coopératif, attend le terminal Worker, puis le monitor backend reprend l'unique `Arc<Store>` et exécute `Store::close()` avant de libérer le slot. Si le Worker est déjà devenu terminal et que son Store a déjà été fermé, un Stop tardif retourne idempotemment le dernier état terminal sûr au lieu d'échouer comme runtime inactif. Ce cache terminal ne contient ni handle ni ressource physique et est effacé dès la réservation du Start suivant. Le partage d'un Store entre plusieurs Workers reste une responsabilité de la tranche multi-route suivante.
Le runtime accepte désormais plusieurs routes simultanées sur un même réseau. Chaque route possède son Worker indépendant ; deux Starts de la même identité logique `profile_id + route_id` sont refusés, et un Start d'un autre réseau est refusé tant que le Store partagé reste ouvert. Le premier Worker ouvre et valide le Store, les suivants réutilisent le même `Arc<Store>`, puis la dernière route terminale déclenche seule la fermeture explicite du Store. Un fault d'une route ne stoppe pas les Workers siblings. Stop est ciblé par route et reste idempotent lorsqu'un terminal sûr a déjà été retenu.
Les profils applicatifs sont réseau-centriques (`devnet`, `mainnet`, `testnet`). Un profil peut agréger plusieurs profils Transport du même réseau : le provider reste une source de capability et ne devient jamais une identité réseau ni un choix de profil applicatif.
Une route explicitement liée à un provider n’est projetée pour un réseau que si le composite déclare une source Transport de ce provider. Ainsi, la route Helius Transaction existe sur Devnet/Mainnet mais n’est pas présentée sur Testnet tant qu’Helius ne fournit pas de source Testnet configurée.
@@ -50,7 +50,7 @@ Les profils standard publics committés déclarent `Block` en plus de `Logs`; `s
Une route `Configured` est uniquement **composable depuis Config**. La disponibilité réelle du Store et du Worker est revalidée lors du Start avant publication de l'accusé runtime. Les états spontanés après Start ne sont pas encore streamés vers le frontend ; l'observabilité continue est ajoutée par la tranche monitoring dédiée.
Pour `yellowstone-hydrated`, le Start utilise désormais le HTTP du **même profil Transport Yellowstone** que le gRPC. Sur Mainnet, le profil `publicnode_mainnet` hydrate donc via `https://solana-rpc.publicnode.com` au lieu de retomber sur `api.mainnet.solana.com`. Le profil PublicNode ne réutilise plus la limite locale Solana-public de `5 req/s`; il conserve une concurrence locale bornée et laisse les réponses `429` piloter le cooldown Transport. Cette correction ne supprime pas encore le coût structurel d'une hydration `getTransaction` par transaction : la réduction de cette dépendance, le pool HTTP multi-provider et le multi-route appartiennent à `pre.009`.
Pour `yellowstone-hydrated`, la stratégie Mainnet n'effectue plus un `getTransaction` par notification transactionnelle. Le Desk construit un abonnement Yellowstone Block léger ; chaque bloc observé déclenche une reconciliation HTTP `getBlock` Full/Base64 sur le même réseau. Le profil `publicnode_mainnet` expose un pool HTTP logique `default` contenant PublicNode et le RPC public Solana comme fallback de même priorité. Le pool conserve les limites propres à chaque endpoint, son health/cooldown et son round-robin interne. La reconciliation Yellowstone tolère un décalage bref entre le gRPC et les RPC HTTP grâce à un retry borné et interruptible par Stop. Le chemin protobuf -> RAW direct reste différé tant que la canonicalisation exacte du meta n'est pas prouvée pour toutes les formes supportées.
## Ports de développement

13
crates/ksp-app-raw-transaction-ingest-desk/USAGE.md
View File

@@ -1,5 +1,5 @@
<!-- file: crates/ksp-app-raw-transaction-ingest-desk/USAGE.md -->
<!-- version: 8 -->
<!-- version: 9 -->
# Utilisation de `ksp-app-raw-transaction-ingest-desk`
@@ -36,15 +36,15 @@ Une route provider-specific dont aucune source n’est déclarée pour le résea
Les URLs, tokens, metadata secrètes, handles Transport/Store/Worker, URI Store, source keys et payloads RAW restent backend-only.
### Start/Stop mono-route
### Start/Stop multi-route
Pour une route `Configured`, choisir `confirmed` ou `finalized` puis utiliser `Start`. Le backend revalide la génération d'inventaire et la composition Config, ouvre le Store correspondant, exige son état `Ready`, construit le Worker exact et installe son handle avant de renvoyer l'accusé runtime.
Pour une route `Configured`, choisir `confirmed` ou `finalized` puis utiliser `Start`. Le backend revalide la génération d'inventaire et la composition Config, ouvre le Store correspondant lorsqu'aucun Store compatible n'est encore actif, exige son état `Ready`, construit le Worker exact puis installe son handle avant de renvoyer l'accusé runtime.
Un seul Worker peut être actif ou en démarrage dans cette tranche. Un second Start est refusé tant que le Worker précédent n'est pas terminal et que son Store n'a pas été explicitement fermé. `Stop` est coopératif : la commande attend le terminal Worker et la fermeture Store avant de rendre le slot à nouveau disponible. Si le Worker est déjà terminal et nettoyé avant le clic Stop, la commande retourne idempotemment le dernier état terminal sûr ; ce comportement permet de resynchroniser le contrôle `pre.008` sans introduire le monitoring continu réservé à une tranche ultérieure.
Plusieurs routes peuvent être actives simultanément lorsqu'elles appartiennent au même réseau logique. Chaque route conserve son Worker indépendant et son Stop ciblé. Le Store est partagé entre ces Workers ; le dernier Worker terminal déclenche la fermeture explicite du Store. Une route `Faulted` ne force pas l'arrêt des autres routes. Un Start d'un autre réseau est refusé tant que le Store partagé du réseau courant reste ouvert.
Le frontend n'effectue aucun polling runtime continu dans cette tranche. Une terminaison spontanée est nettoyée côté backend ; sa projection temps réel vers l'UI appartient au monitoring ultérieur.
Le frontend ne fournit pas encore de streaming continu des snapshots runtime : une terminaison spontanée est nettoyée côté backend et un Stop tardif peut resynchroniser la route avec son dernier terminal sûr.
Sur Mainnet, `yellowstone-hydrated` compose le gRPC et l'HTTP depuis la source `transport.yellowstone` déclarée par le composite. Avec le profil committé `publicnode_mainnet`, les notifications Yellowstone PublicNode sont donc hydratées par le RPC HTTP PublicNode du même profil. Ce mode reste une route hydratée : il peut encore être limité par le débit `getTransaction`. La combinaison de plusieurs endpoints HTTP/providers et le chemin Yellowstone direct sans hydration systématique sont volontairement reportés à `pre.009`.
Sur Mainnet, `yellowstone-hydrated` utilise un abonnement Yellowstone Block comme signal de temps réel puis reconcile chaque slot via `getBlock` Full/Base64. Cette stratégie évite l'hydration `getTransaction` unitaire par transaction. Le profil PublicNode peut exposer plusieurs endpoints HTTP sous le même rôle logique ; Transport distribue alors les requêtes selon ses règles de priorité, disponibilité, concurrence et cooldown. La conversion directe du payload Yellowstone vers le RAW canonique reste réservée à une évolution ultérieure tant que la parité complète du meta n'est pas prouvée.
## 3. Raisons d'indisponibilité
@@ -53,6 +53,7 @@ Les reasons exposées sont des codes applicatifs bornés, par exemple :
```text
profile_unresolved
network_mismatch
missing_http_get_block
missing_http_get_transaction
missing_http_block_scan
missing_ws_logs_capability

11
crates/ksp-app-raw-transaction-ingest-desk/frontend/main.html
View File

@@ -1,5 +1,5 @@
<!-- file: crates/ksp-app-raw-transaction-ingest-desk/frontend/main.html -->
<!-- version: 5 -->
<!-- version: 6 -->
<!DOCTYPE html>
<html lang="fr">
@@ -51,7 +51,7 @@
<div class="d-flex align-items-start justify-content-between gap-3 flex-wrap mb-4">
<div>
<h1 class="h3 mb-1">Routes live</h1>
<p class="text-body-secondary mb-0">Inventaire Config et runtime mono-route Store + Worker avec Start/Stop backend-owned.</p>
<p class="text-body-secondary mb-0">Inventaire Config et runtime multi-route avec un Worker indépendant par route et Store partagé same-network.</p>
</div>
<button id="refreshRouteInventory" class="btn btn-outline-primary btn-sm" type="button">
<i class="fa-solid fa-rotate me-2" aria-hidden="true"></i>Refresh
@@ -79,20 +79,17 @@
</div>
</div>
<div class="alert alert-info" role="status">
<strong>Configured</strong> signifie que Config prouve la composabilité. Start revalide Config, ouvre le Store puis lance exactement un Worker. pre.008 reste volontairement mono-route.
<strong>Configured</strong> signifie que Config prouve la composabilité. Start revalide Config puis lance un Worker indépendant. Les routes du même réseau partagent le Store ; chaque fault reste isolé à sa route.
</div>
<div class="card border-primary-subtle mb-4">
<div class="card-body d-flex align-items-center justify-content-between gap-3 flex-wrap">
<div>
<div class="small text-body-secondary">Runtime mono-route</div>
<div class="small text-body-secondary">Runtime multi-route / Store partagé</div>
<div class="d-flex flex-wrap gap-2 mt-1">
<span id="routeRuntimeState" class="badge text-bg-secondary">idle</span>
<span id="routeRuntimeIdentity" class="app-route-code small text-body-secondary">aucun Worker actif</span>
</div>
</div>
<button id="stopRoute" class="btn btn-outline-danger btn-sm" type="button" disabled>
<i class="fa-solid fa-stop me-2" aria-hidden="true"></i>Stop
</button>
</div>
</div>
<div class="small text-body-secondary mb-3">Réseau affiché : <span id="routeSelectedProfile" class="app-route-code">—</span></div>

121
crates/ksp-app-raw-transaction-ingest-desk/frontend/ts/main.ts
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/frontend/ts/main.ts
// version: 5
// version: 6
import "bootstrap";
import ResizeObserver from "resize-observer-polyfill";
@@ -10,6 +10,7 @@ import type { RawIngestRouteId } from "./bindings/ksp_app_raw_transaction_ingest
import type { RawIngestRouteInventoryDto } from "./bindings/ksp_app_raw_transaction_ingest_desk/dto_route/RawIngestRouteInventoryDto.ts";
import type { RawIngestRouteRuntimeDto } from "./bindings/ksp_app_raw_transaction_ingest_desk/dto_route/RawIngestRouteRuntimeDto.ts";
import type { RawIngestRouteStartRequestDto } from "./bindings/ksp_app_raw_transaction_ingest_desk/dto_route/RawIngestRouteStartRequestDto.ts";
import type { RawIngestRouteStopRequestDto } from "./bindings/ksp_app_raw_transaction_ingest_desk/dto_route/RawIngestRouteStopRequestDto.ts";
import { frontendDebug, frontendInfo, frontendTrace, frontendWarn, installFrontendConsoleBridge } from "./frontend_log";
import { invokeKsp } from "./invoke";
@@ -23,8 +24,8 @@ const viewTitles: Record<ViewId, string> = {
diagnostics: "Diagnostics",
};
let activeRuntime: RawIngestRouteRuntimeDto | null = null;
let lastRuntime: RawIngestRouteRuntimeDto | null = null;
const activeRuntimes = new Map<string, RawIngestRouteRuntimeDto>();
const lastRuntimes = new Map<string, RawIngestRouteRuntimeDto>();
let routeInventory: RawIngestRouteInventoryDto | null = null;
let selectedProfileId: string | null = null;
@@ -88,6 +89,7 @@ function reasonLabel(reason: string | null): string {
const labels: Record<string, string> = {
profile_unresolved: "profil Config non résolu",
network_mismatch: "réseau Transport/Store incohérent",
missing_http_get_block: "HTTP getBlock indisponible",
missing_http_get_transaction: "HTTP getTransaction indisponible",
missing_http_block_scan: "HTTP block scan incomplet",
missing_ws_logs_capability: "capability WS Logs absente",
@@ -102,6 +104,10 @@ function reasonLabel(reason: string | null): string {
return labels[reason] ?? reason;
}
function runtimeKey(profileId: string, routeId: RawIngestRouteId): string {
return `${profileId}::${routeId}`;
}
function renderSelectedProfile(): void {
if (routeInventory === null) {
return;
@@ -118,6 +124,9 @@ function renderSelectedProfile(): void {
if (routeRoot) {
routeRoot.replaceChildren();
for (const route of profile.routes) {
const key = runtimeKey(profile.profileId, route.routeId);
const runtime = activeRuntimes.get(key) ?? lastRuntimes.get(key) ?? null;
const active = runtime !== null && runtimeIsActive(runtime);
const column = document.createElement("div");
column.className = "col-12 col-xl-6 app-route-column";
const card = document.createElement("div");
@@ -133,8 +142,9 @@ function renderSelectedProfile(): void {
const meta = document.createElement("div");
meta.className = "d-flex flex-wrap gap-2 mt-3 app-route-meta";
const state = document.createElement("span");
state.className = route.selectable ? "badge text-bg-success" : "badge text-bg-secondary";
state.textContent = route.state;
const projectedState = runtime?.state ?? route.state;
state.className = projectedState === "faulted" ? "badge text-bg-danger" : active ? "badge text-bg-primary" : route.selectable ? "badge text-bg-success" : "badge text-bg-secondary";
state.textContent = projectedState;
const network = document.createElement("span");
network.className = "badge text-bg-light border text-dark";
network.textContent = route.network ?? "network unresolved";
@@ -145,17 +155,29 @@ function renderSelectedProfile(): void {
body.append(title, code, meta);
if (route.selectable) {
const actions = document.createElement("div");
actions.className = "mt-3";
actions.className = "d-flex gap-2 mt-3";
const start = document.createElement("button");
start.className = "btn btn-primary btn-sm";
start.type = "button";
start.textContent = "Start";
start.disabled = activeRuntime !== null;
start.disabled = active;
start.dataset.routeId = route.routeId;
start.addEventListener("click", () => {
void startRoute(route.routeId).catch(() => frontendWarn("main", "Raw Transaction Ingest Desk route Start failed"));
});
actions.append(start);
if (active && runtime !== null) {
const stop = document.createElement("button");
stop.className = "btn btn-outline-danger btn-sm";
stop.type = "button";
stop.textContent = "Stop";
stop.disabled = runtime.state === "stopping";
stop.dataset.routeId = route.routeId;
stop.addEventListener("click", () => {
void stopRoute(runtime).catch(() => frontendWarn("main", "Raw Transaction Ingest Desk route Stop failed"));
});
actions.append(stop);
}
body.append(actions);
}
card.append(body);
@@ -164,6 +186,7 @@ function renderSelectedProfile(): void {
}
}
frontendTrace("main", "Raw Transaction Ingest Desk Config route inventory profile rendered", {
activeRouteCount: activeRuntimes.size,
profileId: profile.profileId,
routeCount: profile.routes.length,
selectableCount: profile.routes.filter(route => route.selectable).length,
@@ -179,33 +202,28 @@ function runtimeIsActive(runtime: RawIngestRouteRuntimeDto): boolean {
return runtime.state === "starting" || runtime.state === "running" || runtime.state === "stopping";
}
function renderMonoRouteRuntime(): void {
const projected = activeRuntime ?? lastRuntime;
function renderMultiRouteRuntime(): void {
const active = Array.from(activeRuntimes.values());
const state = document.querySelector<HTMLElement>("#routeRuntimeState");
if (state) {
state.textContent = projected?.state ?? "idle";
state.className = projected?.state === "faulted" ? "badge text-bg-danger" : activeRuntime ? "badge text-bg-primary" : "badge text-bg-secondary";
}
text(
"#routeRuntimeIdentity",
projected ? `${projected.profileId} / ${projected.routeId} / ${projected.commitment}` : "aucun Worker actif",
);
const stop = document.querySelector<HTMLButtonElement>("#stopRoute");
if (stop) {
stop.disabled = activeRuntime === null || activeRuntime.state === "stopping";
state.textContent = active.length === 0 ? "idle" : `${active.length} active`;
state.className = active.length === 0 ? "badge text-bg-secondary" : "badge text-bg-primary";
}
const identities = active.map(runtime => `${runtime.routeId}/${runtime.commitment}`).join(", ");
text("#routeRuntimeIdentity", identities.length === 0 ? "aucun Worker actif" : identities);
const profile = document.querySelector<HTMLSelectElement>("#routeProfile");
if (profile) {
profile.disabled = activeRuntime !== null;
}
const commitment = document.querySelector<HTMLSelectElement>("#routeCommitment");
if (commitment) {
commitment.disabled = activeRuntime !== null;
profile.disabled = active.length > 0;
}
}
async function startRoute(routeId: RawIngestRouteId): Promise<void> {
if (routeInventory === null || selectedProfileId === null || activeRuntime !== null) {
if (routeInventory === null || selectedProfileId === null) {
return;
}
const key = runtimeKey(selectedProfileId, routeId);
const existing = activeRuntimes.get(key);
if (existing !== undefined && runtimeIsActive(existing)) {
return;
}
const request: RawIngestRouteStartRequestDto = {
@@ -214,37 +232,49 @@ async function startRoute(routeId: RawIngestRouteId): Promise<void> {
profileId: selectedProfileId,
routeId,
};
frontendDebug("main", "Raw Transaction Ingest Desk mono-route Start requested", {
frontendDebug("main", "Raw Transaction Ingest Desk route Start requested", {
commitment: request.commitment,
profileId: request.profileId,
routeId: request.routeId,
});
const response = await invokeKsp<RawIngestRouteRuntimeDto>("main", "start_route", { request });
lastRuntime = response;
activeRuntime = runtimeIsActive(response) ? response : null;
renderMonoRouteRuntime();
lastRuntimes.set(key, response);
if (runtimeIsActive(response)) {
activeRuntimes.set(key, response);
} else {
activeRuntimes.delete(key);
}
renderMultiRouteRuntime();
renderSelectedProfile();
frontendInfo("main", "Raw Transaction Ingest Desk mono-route Start acknowledged", {
frontendInfo("main", "Raw Transaction Ingest Desk route Start acknowledged", {
activeRouteCount: activeRuntimes.size,
profileId: response.profileId,
routeId: response.routeId,
state: response.state,
});
}
async function stopRoute(): Promise<void> {
if (activeRuntime === null) {
return;
}
frontendDebug("main", "Raw Transaction Ingest Desk mono-route Stop requested", {
profileId: activeRuntime.profileId,
routeId: activeRuntime.routeId,
async function stopRoute(runtime: RawIngestRouteRuntimeDto): Promise<void> {
const key = runtimeKey(runtime.profileId, runtime.routeId);
const request: RawIngestRouteStopRequestDto = {
profileId: runtime.profileId,
routeId: runtime.routeId,
};
frontendDebug("main", "Raw Transaction Ingest Desk targeted route Stop requested", {
profileId: request.profileId,
routeId: request.routeId,
});
const response = await invokeKsp<RawIngestRouteRuntimeDto>("main", "stop_route");
lastRuntime = response;
activeRuntime = runtimeIsActive(response) ? response : null;
renderMonoRouteRuntime();
const response = await invokeKsp<RawIngestRouteRuntimeDto>("main", "stop_route", { request });
lastRuntimes.set(key, response);
if (runtimeIsActive(response)) {
activeRuntimes.set(key, response);
} else {
activeRuntimes.delete(key);
}
renderMultiRouteRuntime();
renderSelectedProfile();
frontendInfo("main", "Raw Transaction Ingest Desk mono-route Stop completed", {
frontendInfo("main", "Raw Transaction Ingest Desk targeted route Stop completed", {
activeRouteCount: activeRuntimes.size,
profileId: response.profileId,
routeId: response.routeId,
state: response.state,
@@ -271,7 +301,7 @@ function renderRouteInventory(inventory: RawIngestRouteInventoryDto): void {
text("#runtimeInventoryGeneration", inventory.generation.toString());
text("#runtimeInventoryDefaultProfile", inventory.defaultProfile);
renderSelectedProfile();
renderMonoRouteRuntime();
renderMultiRouteRuntime();
frontendTrace("main", "Raw Transaction Ingest Desk Config route inventory rendered", {
generation: inventory.generation,
profileCount: inventory.profiles.length,
@@ -323,9 +353,6 @@ function bindRefreshControls(): void {
document.querySelector<HTMLButtonElement>("#refreshRuntimeStatus")?.addEventListener("click", () => {
void refreshRuntimeStatus().catch(() => frontendWarn("main", "Raw Transaction Ingest Desk runtime status refresh failed"));
});
document.querySelector<HTMLButtonElement>("#stopRoute")?.addEventListener("click", () => {
void stopRoute().catch(() => frontendWarn("main", "Raw Transaction Ingest Desk route Stop failed"));
});
document.querySelector<HTMLSelectElement>("#routeProfile")?.addEventListener("change", event => {
const selector = event.currentTarget;
if (!(selector instanceof HTMLSelectElement)) {
@@ -345,7 +372,7 @@ async function initializeMain(): Promise<void> {
activateView("routes", "startup");
await refreshRouteInventory();
await refreshRuntimeStatus();
frontendInfo("main", "Raw Transaction Ingest Desk mono-route runtime frontend ready");
frontendInfo("main", "Raw Transaction Ingest Desk multi-route shared-Store runtime frontend ready");
}
void initializeMain().catch(() => frontendWarn("main", "Raw Transaction Ingest Desk frontend initialization failed"));

12
crates/ksp-app-raw-transaction-ingest-desk/src/app_state.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/src/app_state.rs
// version: 5
// version: 6
//! Shared backend state owned by the Raw Transaction Ingest Desk Tauri application.
@@ -115,7 +115,7 @@ impl crate::AppState {
return crate::validate_route_start(&self.config_management, generation, request);
}
/// Starts one real mono-route Store + Worker runtime after repeating the complete Start-time revalidation.
/// Starts one independent route Worker after repeating complete Start-time revalidation, sharing Store only on an exact network match.
pub(crate) async fn start_route(&self, request: &crate::RawIngestRouteStartRequestDto) -> ksp_core_lib::Result<crate::RouteRuntimeLaunch> {
let generation = {
let generation = self.inventory_generation.lock();
@@ -137,9 +137,9 @@ impl crate::AppState {
return crate::start_route_runtime(std::sync::Arc::clone(&self.route_runtime), prepared).await;
}
/// Requests cooperative Stop and waits until the Worker is terminal and Store cleanup has released the mono-route slot.
pub(crate) async fn stop_route(&self) -> ksp_core_lib::Result<crate::RawIngestRouteRuntimeDto> {
return self.route_runtime.stop_and_wait().await;
/// Requests cooperative Stop for one exact logical route and waits until its Worker cleanup is complete.
pub(crate) async fn stop_route(&self, request: &crate::RawIngestRouteStopRequestDto) -> ksp_core_lib::Result<crate::RawIngestRouteRuntimeDto> {
return self.route_runtime.stop_and_wait(request).await;
}
/// Builds the safe scaffold status exposed by the shell.
@@ -175,7 +175,7 @@ impl crate::AppState {
application_version: env!("CARGO_PKG_VERSION").to_owned(),
config_document_count: document_count,
fallback_logging_active: runtime.fallback_active,
shell_phase: "pre.008-mono-route-runtime".to_owned(),
shell_phase: "pre.009-multi-route-mainnet".to_owned(),
startup_diagnostic: runtime.startup_diagnostic.clone(),
});
}

4
crates/ksp-app-raw-transaction-ingest-desk/src/constants.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/src/constants.rs
// version: 5
// version: 6
//! Application-owned tracing targets and Config component identifiers.
@@ -21,7 +21,7 @@ pub(crate) const TRACING_DOMAIN_BOOTSTRAP: &str = "raw_transaction_ingest.bootst
pub(crate) const TRACING_DOMAIN_FRONTEND: &str = "frontend";
/// Structured domain used while rebuilding Config-only route inventory.
pub(crate) const TRACING_DOMAIN_INVENTORY: &str = "raw_transaction_ingest.inventory";
/// Structured domain used by mono-route Store and Worker lifecycle operations.
/// Structured domain used by multi-route shared-Store and Worker lifecycle operations.
pub(crate) const TRACING_DOMAIN_ROUTE_RUNTIME: &str = "raw_transaction_ingest.route_runtime";
/// Structured domain used while revalidating future Start requests and reconstructing route resources.
pub(crate) const TRACING_DOMAIN_ROUTE_START: &str = "raw_transaction_ingest.route_start";

23
crates/ksp-app-raw-transaction-ingest-desk/src/dto_route.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/src/dto_route.rs
// version: 5
// version: 6
//! Safe route-identity, Config-composability and lifecycle DTOs for Raw Transaction Ingest Desk.
@@ -10,7 +10,7 @@ use ts_rs::TS; // rust-rules: trait-import
#[serde(rename_all = "kebab-case")]
#[ts(export, export_to = "../frontend/ts/bindings/ksp_app_raw_transaction_ingest_desk/dto_route/RawIngestRouteId.ts")]
pub(crate) enum RawIngestRouteId {
/// Yellowstone Subscribe plus same-network HTTP hydration.
/// Yellowstone block subscription plus same-network HTTP block reconciliation.
YellowstoneHydrated,
/// Standard Solana logs subscription plus same-network HTTP hydration.
StandardLogsHydrated,
@@ -45,7 +45,7 @@ impl crate::RawIngestRouteId {
#[must_use]
pub(crate) const fn label(self) -> &'static str {
return match self {
Self::YellowstoneHydrated => "Yellowstone + HTTP hydration",
Self::YellowstoneHydrated => "Yellowstone Block + HTTP block reconciliation",
Self::StandardLogsHydrated => "Standard Logs + HTTP hydration",
Self::StandardBlockDirect => "Standard Block direct",
Self::HeliusTransactionHydrated => "Helius Transaction + HTTP hydration",
@@ -112,6 +112,17 @@ pub(crate) struct RawIngestRouteStartRequestDto {
pub(crate) route_id: crate::RawIngestRouteId,
}
/// Safe targeted Stop request containing only the logical Worker route identity.
#[derive(Clone, Debug, serde::Deserialize, TS)]
#[serde(rename_all = "camelCase")]
#[ts(export, export_to = "../frontend/ts/bindings/ksp_app_raw_transaction_ingest_desk/dto_route/RawIngestRouteStopRequestDto.ts")]
pub(crate) struct RawIngestRouteStopRequestDto {
/// Safe logical network-profile identifier that owns the Worker route.
pub(crate) profile_id: String,
/// Stable logical route identifier to stop.
pub(crate) route_id: crate::RawIngestRouteId,
}
/// Safe acknowledgement that exact Start-time Transport resources were reconstructed and validated without Worker launch.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[serde(rename_all = "camelCase")]
@@ -129,7 +140,7 @@ pub(crate) struct RawIngestRouteStartValidationDto {
pub(crate) route_id: crate::RawIngestRouteId,
}
/// Safe mono-route runtime acknowledgement containing only logical identity and lifecycle state.
/// Safe per-route runtime acknowledgement containing only logical identity and lifecycle state.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[serde(rename_all = "camelCase")]
#[ts(export, export_to = "../frontend/ts/bindings/ksp_app_raw_transaction_ingest_desk/dto_route/RawIngestRouteRuntimeDto.ts")]
@@ -142,7 +153,7 @@ pub(crate) struct RawIngestRouteRuntimeDto {
pub(crate) network: String,
/// Safe logical network-profile identifier.
pub(crate) profile_id: String,
/// Exact logical route owned by the mono-route Worker.
/// Exact logical route owned by the independent Worker.
pub(crate) route_id: crate::RawIngestRouteId,
/// Safe application-owned projection of current Worker lifecycle.
pub(crate) state: crate::RawIngestRouteState,
@@ -174,6 +185,8 @@ pub(crate) enum RawIngestRouteUnavailableReason {
ProfileUnresolved,
/// Transport and Store do not prove one identical logical network.
NetworkMismatch,
/// No HTTP role can satisfy `getBlock` for same-network block reconciliation.
MissingHttpGetBlock,
/// No HTTP role can satisfy `getTransaction` for same-network hydration.
MissingHttpGetTransaction,
/// No HTTP role can satisfy the complete block-polling method set.

11
crates/ksp-app-raw-transaction-ingest-desk/src/errors.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/src/errors.rs
// version: 4
// version: 5
//! Stable Raw Transaction Ingest Desk application error codes.
@@ -26,11 +26,14 @@ pub(crate) const ERROR_CODE_ROUTE_INVENTORY_INVALID: ksp_core_lib::ErrorCode =
/// Exact Start-time Transport resources cannot be reconstructed safely.
pub(crate) const ERROR_CODE_ROUTE_START_PREPARATION_FAILED: ksp_core_lib::ErrorCode =
ksp_core_lib::ErrorCode::new("raw_transaction_ingest_desk", "route_start_preparation_failed");
/// A mono-route Worker runtime already owns the application slot.
/// The selected logical route is already starting/active or shared Store lifecycle is temporarily exclusive.
pub(crate) const ERROR_CODE_ROUTE_RUNTIME_ACTIVE: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("raw_transaction_ingest_desk", "route_runtime_active");
/// No mono-route Worker runtime is currently active.
/// No selected route Worker runtime is currently active.
pub(crate) const ERROR_CODE_ROUTE_RUNTIME_NOT_ACTIVE: ksp_core_lib::ErrorCode =
ksp_core_lib::ErrorCode::new("raw_transaction_ingest_desk", "route_runtime_not_active");
/// A selected route cannot share the currently open Store because its network differs.
pub(crate) const ERROR_CODE_ROUTE_RUNTIME_NETWORK_MISMATCH: ksp_core_lib::ErrorCode =
ksp_core_lib::ErrorCode::new("raw_transaction_ingest_desk", "route_runtime_network_mismatch");
/// Store or Worker startup failed after Start-time route revalidation.
pub(crate) const ERROR_CODE_ROUTE_RUNTIME_START_FAILED: ksp_core_lib::ErrorCode =
ksp_core_lib::ErrorCode::new("raw_transaction_ingest_desk", "route_runtime_start_failed");
@@ -40,7 +43,7 @@ pub(crate) const ERROR_CODE_ROUTE_RUNTIME_STORE_NOT_READY: ksp_core_lib::ErrorCo
/// The Store could not be reclaimed or explicitly closed after Worker termination.
pub(crate) const ERROR_CODE_ROUTE_RUNTIME_STORE_SHUTDOWN_FAILED: ksp_core_lib::ErrorCode =
ksp_core_lib::ErrorCode::new("raw_transaction_ingest_desk", "route_runtime_store_shutdown_failed");
/// The mono-route runtime reservation/handle state changed unexpectedly.
/// The multi-route runtime reservation/handle state changed unexpectedly.
pub(crate) const ERROR_CODE_ROUTE_RUNTIME_STATE_INVALID: ksp_core_lib::ErrorCode =
ksp_core_lib::ErrorCode::new("raw_transaction_ingest_desk", "route_runtime_state_invalid");
/// Start request references an inventory generation that is no longer current.

20
crates/ksp-app-raw-transaction-ingest-desk/src/lib.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/src/lib.rs
// version: 5
// version: 6
//! Tauri desktop application scaffold for composing and supervising KSP live RAW transaction ingest routes.
@@ -58,7 +58,7 @@ pub(crate) use self::constants::TRACING_DOMAIN_BOOTSTRAP;
pub(crate) use self::constants::TRACING_DOMAIN_FRONTEND;
/// Structured domain used while rebuilding Config-only route inventory.
pub(crate) use self::constants::TRACING_DOMAIN_INVENTORY;
/// Structured domain used by mono-route Store and Worker lifecycle operations.
/// Structured domain used by multi-route shared-Store and Worker lifecycle operations.
pub(crate) use self::constants::TRACING_DOMAIN_ROUTE_RUNTIME;
/// Structured domain used while revalidating future Start requests.
pub(crate) use self::constants::TRACING_DOMAIN_ROUTE_START;
@@ -92,7 +92,7 @@ pub(crate) use self::dto_route::RawIngestRouteFoundationDto;
pub(crate) use self::dto_route::RawIngestRouteId;
/// Complete deterministic Config-only route inventory.
pub(crate) use self::dto_route::RawIngestRouteInventoryDto;
/// Safe mono-route runtime acknowledgement containing only logical identity and lifecycle.
/// Safe per-route runtime acknowledgement containing only logical identity and lifecycle.
pub(crate) use self::dto_route::RawIngestRouteRuntimeDto;
/// Safe future Start request containing only logical route selection and inventory generation.
pub(crate) use self::dto_route::RawIngestRouteStartRequestDto;
@@ -100,6 +100,8 @@ pub(crate) use self::dto_route::RawIngestRouteStartRequestDto;
pub(crate) use self::dto_route::RawIngestRouteStartValidationDto;
/// Stable route lifecycle states reserved for runtime projections.
pub(crate) use self::dto_route::RawIngestRouteState;
/// Safe targeted Stop request containing only logical route identity.
pub(crate) use self::dto_route::RawIngestRouteStopRequestDto;
/// Safe reason explaining why one logical route is not composable from Config.
pub(crate) use self::dto_route::RawIngestRouteUnavailableReason;
/// Shared Raw Transaction Ingest Desk application state is internally inconsistent.
@@ -116,13 +118,15 @@ pub(crate) use self::errors::ERROR_CODE_FRONTEND_LOG_TARGET_INVALID;
pub(crate) use self::errors::ERROR_CODE_LOGGING_BOOTSTRAP_FAILED;
/// Config-only route inventory cannot be projected safely.
pub(crate) use self::errors::ERROR_CODE_ROUTE_INVENTORY_INVALID;
/// A mono-route Worker runtime already owns the application slot.
/// A Worker runtime already owns the same logical route in the shared application runtime.
pub(crate) use self::errors::ERROR_CODE_ROUTE_RUNTIME_ACTIVE;
/// No mono-route Worker runtime is currently active.
/// Selected route cannot share the currently open Store because its network differs.
pub(crate) use self::errors::ERROR_CODE_ROUTE_RUNTIME_NETWORK_MISMATCH;
/// No selected route Worker runtime is currently active.
pub(crate) use self::errors::ERROR_CODE_ROUTE_RUNTIME_NOT_ACTIVE;
/// Store or Worker startup failed after Start-time route revalidation.
pub(crate) use self::errors::ERROR_CODE_ROUTE_RUNTIME_START_FAILED;
/// The mono-route runtime reservation/handle state changed unexpectedly.
/// The multi-route runtime reservation or handle state changed unexpectedly.
pub(crate) use self::errors::ERROR_CODE_ROUTE_RUNTIME_STATE_INVALID;
/// The opened Store did not prove Ready before Worker launch.
pub(crate) use self::errors::ERROR_CODE_ROUTE_RUNTIME_STORE_NOT_READY;
@@ -154,9 +158,9 @@ pub(crate) use self::logging_runtime::launch_identity;
pub(crate) use self::route_inventory::build_route_inventory;
/// Rebuilds the route inventory from one already captured Config environment snapshot.
pub(crate) use self::route_inventory::build_route_inventory_with_environment;
/// Launch ownership for one mono-route Worker and its caller-owned Store.
/// Launch ownership for one independent route Worker sharing the application Store when eligible.
pub(crate) use self::route_runtime::RouteRuntimeLaunch;
/// Shared mono-route runtime state admitting at most one starting or active Worker.
/// Shared multi-route runtime state with one independent Worker per route and one same-network Store.
pub(crate) use self::route_runtime::RouteRuntimeState;
/// Opens Store and starts one exact route Worker after Start-time revalidation.
pub(crate) use self::route_runtime::start_route_runtime;

8
crates/ksp-app-raw-transaction-ingest-desk/src/route_inventory.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/src/route_inventory.rs
// version: 5
// version: 6
//! Config-only logical route inventory for Raw Transaction Ingest Desk.
@@ -18,6 +18,7 @@ struct RouteCapabilities {
has_helius_transaction: bool,
helius_source_declared: bool,
helius_source_missing_secret: bool,
yellowstone_has_get_block: bool,
yellowstone_source_missing_secret: bool,
}
@@ -236,6 +237,9 @@ fn build_profile_inventory(
crate::RawIngestRouteUnavailableReason::NetworkMismatch,
);
}
if component_id == crate::COMPOSITE_COMPONENT_ID_TRANSPORT_YELLOWSTONE {
capabilities.yellowstone_has_get_block = http_role_supports_methods(resolved.http_settings(), &["getBlock"]);
}
capabilities.absorb_transport(&resolved, network.as_str());
}
let routes = build_configured_profile_routes(&capabilities, network.as_str());
@@ -268,7 +272,7 @@ fn build_configured_profile_routes(capabilities: &RouteCapabilities, network: &s
network,
&[
(capabilities.has_yellowstone, yellowstone_reason),
(capabilities.has_get_transaction, crate::RawIngestRouteUnavailableReason::MissingHttpGetTransaction),
(capabilities.yellowstone_has_get_block, crate::RawIngestRouteUnavailableReason::MissingHttpGetBlock),
],
),
route_with_requirements(

549
crates/ksp-app-raw-transaction-ingest-desk/src/route_runtime.rs
View File

@@ -1,29 +1,41 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/src/route_runtime.rs
// version: 4
// version: 5
//! Mono-route Store and Worker lifecycle owned by Raw Transaction Ingest Desk.
//! Multi-route Store and independent Worker lifecycle owned by Raw Transaction Ingest Desk.
const ROUTE_STOP_CLEANUP_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(2);
const STORE_RECLAIM_POLL_INTERVAL: std::time::Duration = std::time::Duration::from_millis(5);
const STORE_RECLAIM_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(1);
struct RouteRuntimeInner {
last_terminal: std::option::Option<crate::RawIngestRouteRuntimeDto>,
last_terminals: std::vec::Vec<crate::RawIngestRouteRuntimeDto>,
network: std::option::Option<String>,
next_sequence: u64,
slot: RouteRuntimeSlot,
routes: std::vec::Vec<RouteRuntimeSlot>,
store: std::option::Option<std::sync::Arc<ksp_store_lib::Store>>,
store_closing_token: std::option::Option<u64>,
store_opening_token: std::option::Option<u64>,
}
/// Shared mono-route runtime state. `pre.008` admits at most one active or starting Worker.
/// Shared multi-route runtime state. Each route owns an independent Worker while same-network routes share one Store.
pub(crate) struct RouteRuntimeState {
inner: std::sync::Mutex<RouteRuntimeInner>,
}
impl crate::RouteRuntimeState {
/// Creates an idle mono-route runtime state.
/// Creates an idle multi-route runtime state without Store or Worker ownership.
#[must_use]
pub(crate) fn new() -> Self {
return Self {
inner: std::sync::Mutex::new(RouteRuntimeInner { last_terminal: std::option::Option::None, next_sequence: 0, slot: RouteRuntimeSlot::Idle }),
inner: std::sync::Mutex::new(RouteRuntimeInner {
last_terminals: std::vec::Vec::new(),
network: std::option::Option::None,
next_sequence: 0,
routes: std::vec::Vec::new(),
store: std::option::Option::None,
store_closing_token: std::option::Option::None,
store_opening_token: std::option::Option::None,
}),
};
}
@@ -33,14 +45,34 @@ impl crate::RouteRuntimeState {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::result::Result::Err(runtime_lock_error()),
};
if !matches!(inner.slot, RouteRuntimeSlot::Idle) {
if inner.store_closing_token.is_some() {
return std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_ACTIVE,
"Raw Transaction Ingest Desk already owns one mono-route Worker runtime",
"Raw Transaction Ingest Desk Store is still closing after the previous route set",
));
}
let sequence = inner.next_sequence.checked_add(1);
let sequence = match sequence {
if inner.routes.iter().any(|slot| return slot.matches_identity(prepared.profile_id.as_str(), prepared.route_id)) {
return std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_ACTIVE,
"Raw Transaction Ingest Desk already owns one Worker for the selected logical route",
));
}
if let std::option::Option::Some(network) = inner.network.as_deref()
&& network != prepared.network.as_str()
{
return std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_NETWORK_MISMATCH,
"Raw Transaction Ingest Desk cannot share one Store across different logical networks",
));
}
let needs_store_open = inner.store.is_none();
if needs_store_open && inner.store_opening_token.is_some() {
return std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_ACTIVE,
"Raw Transaction Ingest Desk same-network Store is still opening",
));
}
let sequence = match inner.next_sequence.checked_add(1) {
std::option::Option::Some(value) => value,
std::option::Option::None => {
return std::result::Result::Err(ksp_core_lib::Error::new(
@@ -66,10 +98,55 @@ impl crate::RouteRuntimeState {
profile_id: prepared.profile_id.clone(),
route_id: prepared.route_id,
};
inner.last_terminal = std::option::Option::None;
if inner.network.is_none() {
inner.last_terminals.clear();
inner.network = std::option::Option::Some(prepared.network.clone());
}
inner.last_terminals.retain(|terminal| return !identity.matches_dto(terminal));
inner.next_sequence = sequence;
inner.slot = RouteRuntimeSlot::Starting { token: sequence };
return std::result::Result::Ok(RouteRuntimeReservation { identity, token: sequence, worker_id });
if needs_store_open {
inner.store_opening_token = std::option::Option::Some(sequence);
}
inner.routes.push(RouteRuntimeSlot::Starting { identity: identity.clone(), token: sequence });
return std::result::Result::Ok(RouteRuntimeReservation { identity, needs_store_open, token: sequence, worker_id });
}
fn install_store(&self, token: u64, store: std::sync::Arc<ksp_store_lib::Store>) -> ksp_core_lib::Result<()> {
let inner = self.inner.lock();
let mut inner = match inner {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::result::Result::Err(runtime_lock_error()),
};
if inner.store_opening_token != std::option::Option::Some(token) || inner.store.is_some() {
return std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_STATE_INVALID,
"Raw Transaction Ingest Desk shared Store reservation changed before installation",
));
}
inner.store = std::option::Option::Some(store);
inner.store_opening_token = std::option::Option::None;
return std::result::Result::Ok(());
}
fn shared_store(&self, expected_network: &str) -> ksp_core_lib::Result<std::sync::Arc<ksp_store_lib::Store>> {
let inner = self.inner.lock();
let inner = match inner {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::result::Result::Err(runtime_lock_error()),
};
if inner.network.as_deref() != std::option::Option::Some(expected_network) {
return std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_NETWORK_MISMATCH,
"Raw Transaction Ingest Desk shared Store network no longer matches the selected route",
));
}
return match &inner.store {
std::option::Option::Some(store) => std::result::Result::Ok(std::sync::Arc::clone(store)),
std::option::Option::None => std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_STATE_INVALID,
"Raw Transaction Ingest Desk shared Store is not installed",
)),
};
}
fn activate(
@@ -82,117 +159,226 @@ impl crate::RouteRuntimeState {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::result::Result::Err(runtime_lock_error()),
};
match &inner.slot {
RouteRuntimeSlot::Starting { token } if *token == reservation.token => {},
RouteRuntimeSlot::Idle | RouteRuntimeSlot::Starting { .. } | RouteRuntimeSlot::Active { .. } => {
let position = inner.routes.iter().position(|slot| return slot.starting_token() == std::option::Option::Some(reservation.token));
let position = match position {
std::option::Option::Some(value) => value,
std::option::Option::None => {
return std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_STATE_INVALID,
"Raw Transaction Ingest Desk mono-route runtime reservation changed before Worker activation",
"Raw Transaction Ingest Desk route reservation changed before Worker activation",
));
},
}
};
let state = project_worker_state(handle.snapshot_source().current().worker_snapshot().state());
inner.slot = RouteRuntimeSlot::Active { handle, identity: reservation.identity.clone(), token: reservation.token };
inner.routes[position] = RouteRuntimeSlot::Active { handle, identity: reservation.identity.clone(), token: reservation.token };
return std::result::Result::Ok(reservation.identity.dto(state));
}
fn rollback(&self, token: u64) {
fn rollback(&self, token: u64) -> ksp_core_lib::Result<std::option::Option<std::sync::Arc<ksp_store_lib::Store>>> {
let inner = self.inner.lock();
let mut inner = match inner {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return,
std::result::Result::Err(_) => return std::result::Result::Err(runtime_lock_error()),
};
if matches!(&inner.slot, RouteRuntimeSlot::Starting { token: current, .. } if *current == token) {
inner.slot = RouteRuntimeSlot::Idle;
inner.routes.retain(|slot| return slot.token() != token);
if inner.store_opening_token == std::option::Option::Some(token) {
inner.store_opening_token = std::option::Option::None;
}
}
fn finish(&self, token: u64, terminal: crate::RawIngestRouteRuntimeDto) {
let inner = self.inner.lock();
let mut inner = match inner {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return,
};
if matches!(&inner.slot, RouteRuntimeSlot::Active { token: current, .. } if *current == token) {
inner.last_terminal = std::option::Option::Some(terminal);
inner.slot = RouteRuntimeSlot::Idle;
}
}
/// Requests cooperative Stop, waits for terminal Worker state and returns after the terminal monitor has released the mono-route slot.
pub(crate) async fn stop_and_wait(&self) -> ksp_core_lib::Result<crate::RawIngestRouteRuntimeDto> {
let active = {
let inner = self.inner.lock();
let inner = match inner {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::result::Result::Err(runtime_lock_error()),
};
match &inner.slot {
RouteRuntimeSlot::Active { handle, identity, .. } => std::result::Result::Ok((handle.clone(), identity.clone())),
RouteRuntimeSlot::Idle => match &inner.last_terminal {
std::option::Option::Some(terminal) => return std::result::Result::Ok(terminal.clone()),
std::option::Option::None => std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_NOT_ACTIVE,
"Raw Transaction Ingest Desk has no active mono-route Worker to stop",
)),
},
RouteRuntimeSlot::Starting { .. } => std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_NOT_ACTIVE,
"Raw Transaction Ingest Desk mono-route Worker is still starting and cannot be stopped yet",
)),
if inner.routes.is_empty() {
let store = inner.store.take();
if store.is_some() {
inner.store_closing_token = std::option::Option::Some(token);
} else {
inner.network = std::option::Option::None;
}
return std::result::Result::Ok(store);
}
return std::result::Result::Ok(std::option::Option::None);
}
fn finish(&self, token: u64, terminal: crate::RawIngestRouteRuntimeDto) -> ksp_core_lib::Result<std::option::Option<std::sync::Arc<ksp_store_lib::Store>>> {
let inner = self.inner.lock();
let mut inner = match inner {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::result::Result::Err(runtime_lock_error()),
};
let (handle, identity) = match active {
let before = inner.routes.len();
inner.routes.retain(|slot| return slot.token() != token);
if inner.routes.len() == before {
return std::result::Result::Ok(std::option::Option::None);
}
inner.last_terminals.retain(|previous| return previous.profile_id != terminal.profile_id || previous.route_id != terminal.route_id);
inner.last_terminals.push(terminal);
if inner.routes.is_empty() {
let store = inner.store.take();
if store.is_some() {
inner.store_closing_token = std::option::Option::Some(token);
} else {
inner.network = std::option::Option::None;
}
return std::result::Result::Ok(store);
}
return std::result::Result::Ok(std::option::Option::None);
}
fn complete_store_close(&self, token: u64) {
let inner = self.inner.lock();
let mut inner = match inner {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return,
};
if inner.store_closing_token == std::option::Option::Some(token) {
inner.store_closing_token = std::option::Option::None;
if inner.routes.is_empty() && inner.store.is_none() && inner.store_opening_token.is_none() {
inner.network = std::option::Option::None;
}
}
}
/// Requests cooperative Stop for one exact logical route and waits for its Worker cleanup.
pub(crate) async fn stop_and_wait(&self, request: &crate::RawIngestRouteStopRequestDto) -> ksp_core_lib::Result<crate::RawIngestRouteRuntimeDto> {
let target = self.stop_target(request);
let target = match target {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
let (handle, identity, token) = match target {
RouteStopTarget::Active { handle, identity, token } => (handle, identity, token),
RouteStopTarget::Terminal(value) => return std::result::Result::Ok(value),
RouteStopTarget::TerminalClosing => return self.wait_terminal_store_cleanup(request).await,
};
let _first_stop_request = handle.request_stop();
let terminal = handle.wait_terminal().await;
let terminal = match terminal {
if let std::result::Result::Err(error) = handle.wait_terminal().await {
return std::result::Result::Err(
ksp_core_lib::Error::new(crate::ERROR_CODE_ROUTE_RUNTIME_STATE_INVALID, "Cannot observe terminal Raw Transaction Ingest Desk Worker state")
.with_source(error),
);
}
return self.wait_route_cleanup(token, &identity).await;
}
fn stop_target(&self, request: &crate::RawIngestRouteStopRequestDto) -> ksp_core_lib::Result<RouteStopTarget> {
let inner = self.inner.lock();
let inner = match inner {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => {
return std::result::Result::Err(
ksp_core_lib::Error::new(crate::ERROR_CODE_ROUTE_RUNTIME_STATE_INVALID, "Cannot observe terminal Raw Transaction Ingest Desk Worker state")
.with_source(error),
);
std::result::Result::Err(_) => return std::result::Result::Err(runtime_lock_error()),
};
let slot = inner.routes.iter().find(|slot| return slot.matches_identity(request.profile_id.as_str(), request.route_id));
return match slot {
std::option::Option::Some(RouteRuntimeSlot::Active { handle, identity, token }) => {
std::result::Result::Ok(RouteStopTarget::Active { handle: handle.clone(), identity: identity.clone(), token: *token })
},
std::option::Option::Some(RouteRuntimeSlot::Starting { .. }) => std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_NOT_ACTIVE,
"Raw Transaction Ingest Desk route Worker is still starting and cannot be stopped yet",
)),
std::option::Option::None => {
let terminal =
inner.last_terminals.iter().find(|terminal| return terminal.profile_id == request.profile_id && terminal.route_id == request.route_id);
match terminal {
std::option::Option::Some(value) if inner.store_closing_token.is_none() => {
std::result::Result::Ok(RouteStopTarget::Terminal(value.clone()))
},
std::option::Option::Some(_) => std::result::Result::Ok(RouteStopTarget::TerminalClosing),
std::option::Option::None => std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_NOT_ACTIVE,
"Raw Transaction Ingest Desk has no active Worker for the selected logical route",
)),
}
},
};
}
async fn wait_route_cleanup(&self, token: u64, identity: &RouteRuntimeIdentity) -> ksp_core_lib::Result<crate::RawIngestRouteRuntimeDto> {
let started = std::time::Instant::now();
loop {
let idle = self.is_idle();
let idle = match idle {
let cleanup = self.route_cleanup_complete(token, identity);
let cleanup = match cleanup {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
if idle {
return std::result::Result::Ok(identity.dto(project_worker_state(terminal)));
if let std::option::Option::Some(value) = cleanup {
return std::result::Result::Ok(value);
}
if started.elapsed() >= ROUTE_STOP_CLEANUP_TIMEOUT {
return std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_STORE_SHUTDOWN_FAILED,
"Raw Transaction Ingest Desk terminal cleanup did not release the mono-route slot",
"Raw Transaction Ingest Desk terminal cleanup did not release the selected route",
));
}
tokio::time::sleep(STORE_RECLAIM_POLL_INTERVAL).await;
}
}
fn is_idle(&self) -> ksp_core_lib::Result<bool> {
async fn wait_terminal_store_cleanup(&self, request: &crate::RawIngestRouteStopRequestDto) -> ksp_core_lib::Result<crate::RawIngestRouteRuntimeDto> {
let started = std::time::Instant::now();
loop {
let terminal = self.late_terminal_after_store_cleanup(request);
let terminal = match terminal {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
if let std::option::Option::Some(value) = terminal {
return std::result::Result::Ok(value);
}
if started.elapsed() >= ROUTE_STOP_CLEANUP_TIMEOUT {
return std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_STORE_SHUTDOWN_FAILED,
"Raw Transaction Ingest Desk final shared Store cleanup did not complete for the selected terminal route",
));
}
tokio::time::sleep(STORE_RECLAIM_POLL_INTERVAL).await;
}
}
fn late_terminal_after_store_cleanup(
&self,
request: &crate::RawIngestRouteStopRequestDto,
) -> ksp_core_lib::Result<std::option::Option<crate::RawIngestRouteRuntimeDto>> {
let inner = self.inner.lock();
let inner = match inner {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::result::Result::Err(runtime_lock_error()),
};
return std::result::Result::Ok(matches!(inner.slot, RouteRuntimeSlot::Idle));
let terminal = inner.last_terminals.iter().find(|terminal| return terminal.profile_id == request.profile_id && terminal.route_id == request.route_id);
let terminal = match terminal {
std::option::Option::Some(value) => value,
std::option::Option::None => {
return std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_NOT_ACTIVE,
"Raw Transaction Ingest Desk terminal route is no longer retained",
));
},
};
if inner.store_closing_token.is_some() {
return std::result::Result::Ok(std::option::Option::None);
}
return std::result::Result::Ok(std::option::Option::Some(terminal.clone()));
}
fn route_cleanup_complete(
&self,
token: u64,
identity: &RouteRuntimeIdentity,
) -> ksp_core_lib::Result<std::option::Option<crate::RawIngestRouteRuntimeDto>> {
let inner = self.inner.lock();
let inner = match inner {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::result::Result::Err(runtime_lock_error()),
};
if inner.routes.iter().any(|slot| return slot.token() == token) || inner.store_closing_token == std::option::Option::Some(token) {
return std::result::Result::Ok(std::option::Option::None);
}
let terminal = inner.last_terminals.iter().find(|value| return identity.matches_dto(value));
return std::result::Result::Ok(terminal.cloned());
}
}
/// Launch ownership returned only after Store readiness, Worker creation and handle installation have succeeded.
/// Launch ownership returned only after shared Store readiness, Worker creation and route-handle installation have succeeded.
pub(crate) struct RouteRuntimeLaunch {
acknowledgement: crate::RawIngestRouteRuntimeDto,
handle: ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle,
runtime_state: std::sync::Arc<crate::RouteRuntimeState>,
store: std::sync::Arc<ksp_store_lib::Store>,
token: u64,
}
@@ -203,7 +389,7 @@ impl crate::RouteRuntimeLaunch {
return self.acknowledgement.clone();
}
/// Waits for terminal Worker state, explicitly closes Store after the Worker releases its `Arc`, then frees the mono-route slot.
/// Waits for this Worker terminal state, removes only its route and closes Store only after the final same-network route terminates.
pub(crate) async fn monitor(self) {
let terminal = self.handle.wait_terminal().await;
let terminal_state = match terminal {
@@ -212,7 +398,7 @@ impl crate::RouteRuntimeLaunch {
target: crate::TRACING_TARGET,
domain = crate::TRACING_DOMAIN_ROUTE_RUNTIME,
worker_state = state.code(),
"Raw Transaction Ingest Desk mono-route Worker reached terminal state"
"Raw Transaction Ingest Desk route Worker reached terminal state"
);
project_worker_state(state)
},
@@ -227,64 +413,99 @@ impl crate::RouteRuntimeLaunch {
crate::RawIngestRouteState::Faulted
},
};
let close = close_store_arc(self.store).await;
if let std::result::Result::Err(error) = close {
ksp_logging_lib::warn!(
target: crate::TRACING_TARGET,
domain = crate::TRACING_DOMAIN_ROUTE_RUNTIME,
error_domain = error.code().domain(),
error_code = error.code().code(),
"Raw Transaction Ingest Desk Store shutdown failed after terminal Worker"
);
}
let mut terminal = self.acknowledgement;
terminal.state = terminal_state;
self.runtime_state.finish(self.token, terminal);
let store = self.runtime_state.finish(self.token, terminal);
let store = match store {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => {
ksp_logging_lib::warn!(
target: crate::TRACING_TARGET,
domain = crate::TRACING_DOMAIN_ROUTE_RUNTIME,
error_domain = error.code().domain(),
error_code = error.code().code(),
"Raw Transaction Ingest Desk route cleanup state update failed"
);
return;
},
};
if let std::option::Option::Some(store) = store {
let close = close_store_arc(store).await;
if let std::result::Result::Err(error) = close {
ksp_logging_lib::warn!(
target: crate::TRACING_TARGET,
domain = crate::TRACING_DOMAIN_ROUTE_RUNTIME,
error_domain = error.code().domain(),
error_code = error.code().code(),
"Raw Transaction Ingest Desk shared Store shutdown failed after final Worker"
);
}
self.runtime_state.complete_store_close(self.token);
}
}
}
/// Opens one Store, starts one exact Worker route and atomically installs its handle in the mono-route application slot.
/// Starts one exact route Worker and reuses the same ready Store for any already-active same-network route.
pub(crate) async fn start_route_runtime(
runtime_state: std::sync::Arc<crate::RouteRuntimeState>,
prepared: crate::PreparedRouteStart,
) -> ksp_core_lib::Result<crate::RouteRuntimeLaunch> {
let reservation = runtime_state.reserve(&prepared);
let reservation = match reservation {
let reservation = match runtime_state.reserve(&prepared) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
let store = ksp_store_lib::Store::open(prepared.store_settings).await;
let store = match store {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => {
runtime_state.rollback(reservation.token);
return std::result::Result::Err(
ksp_core_lib::Error::new(crate::ERROR_CODE_ROUTE_RUNTIME_START_FAILED, "Cannot open Raw Transaction Ingest Desk Store runtime")
let store = if reservation.needs_store_open {
let store = ksp_store_lib::Store::open(prepared.store_settings).await;
let store = match store {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => {
let _rollback = runtime_state.rollback(reservation.token);
return std::result::Result::Err(
ksp_core_lib::Error::new(crate::ERROR_CODE_ROUTE_RUNTIME_START_FAILED, "Cannot open Raw Transaction Ingest Desk shared Store runtime")
.with_source(error),
);
},
};
let health = store.health().await;
if health.state() != ksp_store_lib::StoreHealthState::Ready {
let close = store.close().await;
let _rollback = runtime_state.rollback(reservation.token);
if let std::result::Result::Err(error) = close {
return std::result::Result::Err(
ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_STORE_SHUTDOWN_FAILED,
"Cannot close non-ready Raw Transaction Ingest Desk shared Store runtime",
)
.with_source(error),
);
},
};
let health = store.health().await;
if health.state() != ksp_store_lib::StoreHealthState::Ready {
let close = store.close().await;
runtime_state.rollback(reservation.token);
if let std::result::Result::Err(error) = close {
return std::result::Result::Err(
ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_STORE_SHUTDOWN_FAILED,
"Cannot close non-ready Raw Transaction Ingest Desk Store runtime",
)
.with_source(error),
);
);
}
return std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_STORE_NOT_READY,
"Raw Transaction Ingest Desk shared Store readiness probe did not prove Ready",
));
}
return std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_STORE_NOT_READY,
"Raw Transaction Ingest Desk Store readiness probe did not prove Ready",
));
}
let store = std::sync::Arc::new(store);
if let std::result::Result::Err(error) = runtime_state.install_store(reservation.token, std::sync::Arc::clone(&store)) {
let close = close_store_arc(store).await;
let _rollback = runtime_state.rollback(reservation.token);
if let std::result::Result::Err(close_error) = close {
return std::result::Result::Err(close_error);
}
return std::result::Result::Err(error);
}
store
} else {
std::mem::drop(prepared.store_settings);
match runtime_state.shared_store(prepared.network.as_str()) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => {
let _rollback = runtime_state.rollback(reservation.token);
return std::result::Result::Err(error);
},
}
};
let network = store.runtime_snapshot().network().clone();
let settings = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::with_defaults(network, reservation.worker_id.clone());
let store = std::sync::Arc::new(store);
let handle = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWorker::start_with_runtime_resources(
settings,
std::sync::Arc::clone(&store),
@@ -293,13 +514,21 @@ pub(crate) async fn start_route_runtime(
let handle = match handle {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => {
let close = close_store_arc(store).await;
runtime_state.rollback(reservation.token);
if let std::result::Result::Err(close_error) = close {
return std::result::Result::Err(close_error);
let cleanup_store = runtime_state.rollback(reservation.token);
let cleanup_store = match cleanup_store {
std::result::Result::Ok(value) => value,
std::result::Result::Err(rollback_error) => return std::result::Result::Err(rollback_error),
};
std::mem::drop(store);
if let std::option::Option::Some(value) = cleanup_store {
let close = close_store_arc(value).await;
runtime_state.complete_store_close(reservation.token);
if let std::result::Result::Err(close_error) = close {
return std::result::Result::Err(close_error);
}
}
return std::result::Result::Err(
ksp_core_lib::Error::new(crate::ERROR_CODE_ROUTE_RUNTIME_START_FAILED, "Cannot start Raw Transaction Ingest Desk Worker runtime")
ksp_core_lib::Error::new(crate::ERROR_CODE_ROUTE_RUNTIME_START_FAILED, "Cannot start Raw Transaction Ingest Desk route Worker runtime")
.with_source(error),
);
},
@@ -310,14 +539,23 @@ pub(crate) async fn start_route_runtime(
std::result::Result::Err(error) => {
let _stop_requested = handle.request_stop();
let _terminal = handle.wait_terminal().await;
let close = close_store_arc(store).await;
runtime_state.rollback(reservation.token);
if let std::result::Result::Err(close_error) = close {
return std::result::Result::Err(close_error);
let cleanup_store = runtime_state.rollback(reservation.token);
let cleanup_store = match cleanup_store {
std::result::Result::Ok(value) => value,
std::result::Result::Err(rollback_error) => return std::result::Result::Err(rollback_error),
};
std::mem::drop(store);
if let std::option::Option::Some(value) = cleanup_store {
let close = close_store_arc(value).await;
runtime_state.complete_store_close(reservation.token);
if let std::result::Result::Err(close_error) = close {
return std::result::Result::Err(close_error);
}
}
return std::result::Result::Err(error);
},
};
std::mem::drop(store);
ksp_logging_lib::info!(
target: crate::TRACING_TARGET,
domain = crate::TRACING_DOMAIN_ROUTE_RUNTIME,
@@ -325,9 +563,9 @@ pub(crate) async fn start_route_runtime(
network = reservation.identity.network.as_str(),
route_id = reservation.identity.route_id.as_str(),
commitment = reservation.identity.commitment.as_str(),
"Raw Transaction Ingest Desk mono-route Worker handle installed"
"Raw Transaction Ingest Desk independent route Worker handle installed on shared Store"
);
return std::result::Result::Ok(crate::RouteRuntimeLaunch { acknowledgement, handle, runtime_state, store, token: reservation.token });
return std::result::Result::Ok(crate::RouteRuntimeLaunch { acknowledgement, handle, runtime_state, token: reservation.token });
}
#[derive(Clone)]
@@ -350,17 +588,32 @@ impl RouteRuntimeIdentity {
state,
};
}
fn matches_dto(&self, value: &crate::RawIngestRouteRuntimeDto) -> bool {
return self.profile_id == value.profile_id && self.route_id == value.route_id;
}
}
enum RouteStopTarget {
Active {
handle: ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle,
identity: RouteRuntimeIdentity,
token: u64,
},
Terminal(crate::RawIngestRouteRuntimeDto),
TerminalClosing,
}
struct RouteRuntimeReservation {
identity: RouteRuntimeIdentity,
needs_store_open: bool,
token: u64,
worker_id: ksp_worker_api::WorkerId,
}
enum RouteRuntimeSlot {
Idle,
Starting {
identity: RouteRuntimeIdentity,
token: u64,
},
Active {
@@ -370,6 +623,28 @@ enum RouteRuntimeSlot {
},
}
impl RouteRuntimeSlot {
fn matches_identity(&self, profile_id: &str, route_id: crate::RawIngestRouteId) -> bool {
let identity = match self {
Self::Starting { identity, .. } | Self::Active { identity, .. } => identity,
};
return identity.profile_id == profile_id && identity.route_id == route_id;
}
fn starting_token(&self) -> std::option::Option<u64> {
return match self {
Self::Starting { token, .. } => std::option::Option::Some(*token),
Self::Active { .. } => std::option::Option::None,
};
}
fn token(&self) -> u64 {
return match self {
Self::Starting { token, .. } | Self::Active { token, .. } => *token,
};
}
}
async fn close_store_arc(store: std::sync::Arc<ksp_store_lib::Store>) -> ksp_core_lib::Result<()> {
let started = std::time::Instant::now();
let mut store = store;
@@ -382,20 +657,18 @@ async fn close_store_arc(store: std::sync::Arc<ksp_store_lib::Store>) -> ksp_cor
std::result::Result::Err(error) => std::result::Result::Err(
ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_STORE_SHUTDOWN_FAILED,
"Cannot close Raw Transaction Ingest Desk Store runtime",
"Cannot close Raw Transaction Ingest Desk shared Store runtime",
)
.with_source(error),
),
};
},
std::result::Result::Err(shared) => {
store = shared;
},
std::result::Result::Err(shared) => store = shared,
}
if started.elapsed() >= STORE_RECLAIM_TIMEOUT {
return std::result::Result::Err(ksp_core_lib::Error::new(
crate::ERROR_CODE_ROUTE_RUNTIME_STORE_SHUTDOWN_FAILED,
"Raw Transaction Ingest Desk Store remained shared after terminal Worker",
"Raw Transaction Ingest Desk shared Store remained referenced after final Worker",
));
}
tokio::time::sleep(STORE_RECLAIM_POLL_INTERVAL).await;
@@ -414,7 +687,7 @@ fn project_worker_state(state: ksp_worker_api::WorkerState) -> crate::RawIngestR
}
fn runtime_lock_error() -> ksp_core_lib::Error {
return ksp_core_lib::Error::new(crate::ERROR_CODE_APP_STATE_LOCK_FAILED, "Raw Transaction Ingest Desk mono-route runtime state lock is poisoned");
return ksp_core_lib::Error::new(crate::ERROR_CODE_APP_STATE_LOCK_FAILED, "Raw Transaction Ingest Desk multi-route runtime state lock is poisoned");
}
#[cfg(test)]

12
crates/ksp-app-raw-transaction-ingest-desk/src/route_start.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/src/route_start.rs
// version: 3
// version: 4
//! Start-time Config revalidation and exact Transport/Store resource reconstruction for route validation and runtime launch.
@@ -278,18 +278,22 @@ fn prepare_yellowstone(
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
let filter_name = ksp_onchain_transport_lib::YellowstoneSubscribeFilterName::new("raw-ingest-all-transactions");
let filter_name = ksp_onchain_transport_lib::YellowstoneSubscribeFilterName::new("raw-ingest-blocks");
let filter_name = match filter_name {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
let mut block_filter = ksp_onchain_transport_lib::YellowstoneSubscribeBlockFilter::new();
block_filter.set_include_transactions(std::option::Option::Some(false));
block_filter.set_include_accounts(std::option::Option::Some(false));
block_filter.set_include_entries(std::option::Option::Some(false));
let mut request = ksp_onchain_transport_lib::YellowstoneSubscribeRequest::new();
let inserted = request.insert_transaction_filter(filter_name, ksp_onchain_transport_lib::YellowstoneSubscribeTransactionFilter::new());
let inserted = request.insert_block_filter(filter_name, block_filter);
if let std::result::Result::Err(error) = inserted {
return std::result::Result::Err(error);
}
request.set_commitment(std::option::Option::Some(commitment));
let role = single_http_role(transport.http_settings(), network, &["getTransaction"]);
let role = single_http_role(transport.http_settings(), network, &["getBlock"]);
let role = match role {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),

9
crates/ksp-app-raw-transaction-ingest-desk/src/tauri.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/src/tauri.rs
// version: 5
// version: 6
//! Tauri runtime assembly for the KSP Raw Transaction Ingest desktop application.
@@ -164,8 +164,11 @@ async fn start_route(
}
#[tauri::command]
async fn stop_route(state: tauri::State<'_, crate::AppState>) -> std::result::Result<crate::RawIngestRouteRuntimeDto, crate::CommandErrorDto> {
let result = state.stop_route().await;
async fn stop_route(
request: crate::RawIngestRouteStopRequestDto,
state: tauri::State<'_, crate::AppState>,
) -> std::result::Result<crate::RawIngestRouteRuntimeDto, crate::CommandErrorDto> {
let result = state.stop_route(&request).await;
return match result {
std::result::Result::Ok(value) => std::result::Result::Ok(value),
std::result::Result::Err(error) => std::result::Result::Err(project_command_error("stop_route", crate::TRACING_DOMAIN_ROUTE_RUNTIME, &error)),

17
crates/ksp-app-raw-transaction-ingest-desk/tests/desktop_contract.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/tests/desktop_contract.rs
// version: 5
// version: 6
//! Desktop scaffold contract canaries for Raw Transaction Ingest Desk.
@@ -186,17 +186,18 @@ fn pre_007_start_preflight_is_backend_owned_generation_bound_and_does_not_launch
}
#[test]
fn pre_008_start_stop_commands_and_frontend_controls_are_mono_route_and_backend_owned() {
fn pre_009_start_stop_controls_are_multi_route_targeted_and_backend_owned() {
let tauri = read_text(app_root().join("src/tauri.rs").as_path());
for required in ["start_route", "stop_route", "launch.monitor().await"] {
assert!(tauri.contains(required), "missing pre.008 Tauri runtime marker {required}");
for required in ["start_route", "stop_route", "RawIngestRouteStopRequestDto", "launch.monitor().await"] {
assert!(tauri.contains(required), "missing pre.009 Tauri runtime marker {required}");
}
let html = read_text(app_root().join("frontend/main.html").as_path());
for required in ["id=\"routeCommitment\"", "id=\"routeRuntimeState\"", "id=\"routeRuntimeIdentity\"", "id=\"stopRoute\""] {
assert!(html.contains(required), "missing pre.008 frontend runtime marker {required}");
for required in ["id=\"routeCommitment\"", "id=\"routeRuntimeState\"", "id=\"routeRuntimeIdentity\"", "Runtime multi-route / Store partagé"] {
assert!(html.contains(required), "missing pre.009 frontend runtime marker {required}");
}
assert!(!html.contains("id=\"stopRoute\""));
let main = read_text(app_root().join("frontend/ts/main.ts").as_path());
for required in ["mono-route Start requested", "mono-route Stop requested", "start_route", "stop_route"] {
assert!(main.contains(required), "missing pre.008 frontend control marker {required}");
for required in ["activeRuntimes", "targeted route Stop requested", "start_route", "stop_route", "RawIngestRouteStopRequestDto"] {
assert!(main.contains(required), "missing pre.009 frontend control marker {required}");
}
}

45
crates/ksp-app-raw-transaction-ingest-desk/tests/release_completeness.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/tests/release_completeness.rs
// version: 7
// version: 8
//! Release-completeness canaries for Raw Transaction Ingest Desk Start-resource reconstruction.
@@ -92,23 +92,35 @@ fn pre_007_start_revalidation_reuses_inventory_and_requires_profile_network_iden
}
#[test]
fn pre_008_mono_route_slot_blocks_second_start_until_terminal_store_cleanup() {
fn pre_009_multi_route_runtime_keeps_independent_workers_on_one_same_network_store() {
let runtime = read_text(app_root().join("src/route_runtime.rs").as_path());
for required in [
"RouteRuntimeSlot::Starting",
"RouteRuntimeSlot::Active",
"ERROR_CODE_ROUTE_RUNTIME_ACTIVE",
"inner.last_terminal = std::option::Option::None",
"std::option::Option::Some(terminal) => return std::result::Result::Ok(terminal.clone())",
"routes: std::vec::Vec<RouteRuntimeSlot>",
"store: std::option::Option<std::sync::Arc<ksp_store_lib::Store>>",
"ERROR_CODE_ROUTE_RUNTIME_NETWORK_MISMATCH",
"already owns one Worker for the selected logical route",
"shared_store(prepared.network.as_str())",
"self.runtime_state.finish(self.token, terminal)",
"Arc::try_unwrap",
] {
assert!(runtime.contains(required), "missing mono-route ownership marker {required}");
assert!(runtime.contains(required), "missing pre.009 shared-Store marker {required}");
}
assert!(!runtime.contains("RouteRuntimeSlot::Idle"));
}
#[test]
fn pre_008_fix_004_yellowstone_hydration_uses_its_declared_transport_source_not_base_transport() {
fn pre_009_targeted_stop_is_route_scoped_and_late_terminal_safe() {
let dto = read_text(app_root().join("src/dto_route.rs").as_path());
assert!(dto.contains("RawIngestRouteStopRequestDto"));
let runtime = read_text(app_root().join("src/route_runtime.rs").as_path());
assert!(runtime.contains("stop_and_wait(&self, request: &crate::RawIngestRouteStopRequestDto)"));
assert!(runtime.contains("terminal.profile_id == request.profile_id && terminal.route_id == request.route_id"));
let tauri = read_text(app_root().join("src/tauri.rs").as_path());
assert!(tauri.contains("request: crate::RawIngestRouteStopRequestDto"));
}
#[test]
fn pre_009_yellowstone_uses_block_subscription_and_one_get_block_path_instead_of_per_transaction_hydration() {
let route_start = read_text(app_root().join("src/route_start.rs").as_path());
let start = route_start.find("fn prepare_yellowstone(");
let end = route_start.find("fn prepare_http_polling(");
@@ -118,9 +130,14 @@ fn pre_008_fix_004_yellowstone_hydration_uses_its_declared_transport_source_not_
(std::option::Option::Some(start), std::option::Option::Some(end)) if start < end => &route_start[start..end],
_ => "",
};
assert!(yellowstone.contains("resolve_optional_transport"));
assert!(yellowstone.contains("COMPOSITE_COMPONENT_ID_TRANSPORT_YELLOWSTONE"));
assert!(yellowstone.contains("single_http_role(transport.http_settings(), network, &[\"getTransaction\"])"));
assert!(yellowstone.contains("HttpTransportPool::new(transport.http_settings().clone())"));
assert!(!yellowstone.contains("base_transport"));
assert!(yellowstone.contains("insert_block_filter"));
assert!(yellowstone.contains("set_include_transactions(std::option::Option::Some(false))"));
assert!(yellowstone.contains("single_http_role(transport.http_settings(), network, &[\"getBlock\"])"));
assert!(!yellowstone.contains("insert_transaction_filter"));
let workspace = read_text(app_root().join("../ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs").as_path());
assert!(workspace.contains("RawTransactionIngestYellowstoneMode::BlockHydration"));
assert!(workspace.contains("fetch_yellowstone_block_ingresses"));
assert!(workspace.contains("get_block_observed"));
assert!(workspace.contains("RAW_TRANSACTION_INGEST_YELLOWSTONE_BLOCK_RECONCILIATION_ATTEMPTS"));
assert!(workspace.contains("stop_receiver.changed()"));
}

4
crates/ksp-app-raw-transaction-ingest-desk/unit_tests/dto_route.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/unit_tests/dto_route.rs
// version: 3
// version: 4
#[test]
fn pre_005_route_ids_are_exact_and_stably_ordered() {
@@ -31,6 +31,7 @@ fn pre_006_unavailable_reason_codes_are_exact_and_safe() {
let values = [
crate::RawIngestRouteUnavailableReason::ProfileUnresolved,
crate::RawIngestRouteUnavailableReason::NetworkMismatch,
crate::RawIngestRouteUnavailableReason::MissingHttpGetBlock,
crate::RawIngestRouteUnavailableReason::MissingHttpGetTransaction,
crate::RawIngestRouteUnavailableReason::MissingHttpBlockScan,
crate::RawIngestRouteUnavailableReason::MissingWsLogsCapability,
@@ -47,6 +48,7 @@ fn pre_006_unavailable_reason_codes_are_exact_and_safe() {
serde_json::json!([
"profile_unresolved",
"network_mismatch",
"missing_http_get_block",
"missing_http_get_transaction",
"missing_http_block_scan",
"missing_ws_logs_capability",

64
crates/ksp-app-raw-transaction-ingest-desk/unit_tests/route_runtime.rs
View File

@@ -1,12 +1,64 @@
// file: crates/ksp-app-raw-transaction-ingest-desk/unit_tests/route_runtime.rs
// version: 1
// version: 2
#[test]
fn pre_008_mono_route_runtime_starts_idle_without_worker_or_store_material() {
fn pre_009_multi_route_runtime_starts_without_workers_or_shared_store_material() {
let state = crate::RouteRuntimeState::new();
let idle = state.is_idle();
assert!(idle.is_ok());
if let std::result::Result::Ok(idle) = idle {
assert!(idle);
let inner = state.inner.lock();
assert!(inner.is_ok());
if let std::result::Result::Ok(inner) = inner {
assert!(inner.routes.is_empty());
assert!(inner.store.is_none());
assert!(inner.network.is_none());
assert!(inner.store_opening_token.is_none());
assert!(inner.store_closing_token.is_none());
}
}
#[test]
fn pre_009_late_targeted_stop_returns_only_the_matching_terminal_route() {
let state = crate::RouteRuntimeState::new();
let terminal = crate::RawIngestRouteRuntimeDto {
commitment: crate::RawIngestCommitment::Confirmed,
inventory_generation: 9,
network: "mainnet".to_owned(),
profile_id: "mainnet".to_owned(),
route_id: crate::RawIngestRouteId::YellowstoneHydrated,
state: crate::RawIngestRouteState::Faulted,
};
let inner = state.inner.lock();
assert!(inner.is_ok());
if let std::result::Result::Ok(mut inner) = inner {
inner.last_terminals.push(terminal.clone());
}
let request = crate::RawIngestRouteStopRequestDto { profile_id: "mainnet".to_owned(), route_id: crate::RawIngestRouteId::YellowstoneHydrated };
let target = state.stop_target(&request);
assert!(target.is_ok());
assert!(matches!(&target, std::result::Result::Ok(super::RouteStopTarget::Terminal(_))));
if let std::result::Result::Ok(super::RouteStopTarget::Terminal(value)) = target {
assert_eq!(value.profile_id, terminal.profile_id);
assert_eq!(value.route_id, terminal.route_id);
assert_eq!(value.state, crate::RawIngestRouteState::Faulted);
}
}
#[test]
fn pre_009_late_targeted_stop_waits_while_final_shared_store_is_closing() {
let state = crate::RouteRuntimeState::new();
let inner = state.inner.lock();
assert!(inner.is_ok());
if let std::result::Result::Ok(mut inner) = inner {
inner.last_terminals.push(crate::RawIngestRouteRuntimeDto {
commitment: crate::RawIngestCommitment::Finalized,
inventory_generation: 11,
network: "mainnet".to_owned(),
profile_id: "mainnet".to_owned(),
route_id: crate::RawIngestRouteId::HttpBlockPolling,
state: crate::RawIngestRouteState::Stopped,
});
inner.store_closing_token = std::option::Option::Some(77);
}
let request = crate::RawIngestRouteStopRequestDto { profile_id: "mainnet".to_owned(), route_id: crate::RawIngestRouteId::HttpBlockPolling };
let target = state.stop_target(&request);
assert!(matches!(target, std::result::Result::Ok(super::RouteStopTarget::TerminalClosing)));
}

26
crates/ksp-config-lib/unit_tests/transport.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-config-lib/unit_tests/transport.rs
// version: 17
// version: 18
#[test]
fn fixture_transport_profile_maps_complete_runtime_contract() {
@@ -274,17 +274,25 @@ fn committed_v3_publicnode_profiles_map_provider_neutral_yellowstone_grpc_with_n
};
assert_eq!(resolved.profile_id(), profile_id);
if profile_id == "publicnode_mainnet" {
assert_eq!(resolved.settings().endpoints().len(), 1);
let http = &resolved.settings().endpoints()[0];
assert_eq!(http.name(), "publicnode_solana_mainnet_rpc");
assert_eq!(http.provider().as_str(), "publicnode");
assert_eq!(http.cluster().as_str(), "mainnet");
assert_eq!(http.url().as_str(), "https://solana-rpc.publicnode.com");
assert_eq!(http.roles().len(), 1);
let limits = http.roles()[0].limits();
assert_eq!(resolved.settings().endpoints().len(), 2);
let publicnode = &resolved.settings().endpoints()[0];
assert_eq!(publicnode.name(), "publicnode_solana_mainnet_rpc");
assert_eq!(publicnode.provider().as_str(), "publicnode");
assert_eq!(publicnode.cluster().as_str(), "mainnet");
assert_eq!(publicnode.url().as_str(), "https://solana-rpc.publicnode.com");
assert_eq!(publicnode.roles().len(), 1);
let limits = publicnode.roles()[0].limits();
assert!(limits.requests_per_second().is_none());
assert!(limits.burst_capacity().is_none());
assert_eq!(limits.max_concurrent_requests().map(std::num::NonZeroU32::get), std::option::Option::Some(8));
let solana_public = &resolved.settings().endpoints()[1];
assert_eq!(solana_public.name(), "solana_mainnet_public_repair");
assert_eq!(solana_public.provider().as_str(), "solana-public");
assert_eq!(solana_public.cluster().as_str(), "mainnet");
assert_eq!(solana_public.roles()[0].role().as_str(), "default");
let solana_limits = solana_public.roles()[0].limits();
assert_eq!(solana_limits.requests_per_second().map(std::num::NonZeroU32::get), std::option::Option::Some(5));
assert_eq!(solana_limits.burst_capacity().map(std::num::NonZeroU32::get), std::option::Option::Some(10));
}
assert!(resolved.ws_settings().is_some());
let grpc = resolved.grpc_settings();

280
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
// version: 44
// version: 45
use sha2::Digest; // rust-rules: trait-import
@@ -32,6 +32,9 @@ const RAW_TRANSACTION_INGEST_STANDARD_BLOCK_PROTOCOL: &str = "solana_ws";
const RAW_TRANSACTION_INGEST_STANDARD_LOGS_FILTER_FINGERPRINT_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.standard_logs.filter.v1\0";
const RAW_TRANSACTION_INGEST_STANDARD_LOGS_HTTP_PROTOCOL: &str = "solana_ws_http";
const RAW_TRANSACTION_INGEST_STANDARD_LOGS_HTTP_SOURCE_KEY_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.standard_logs_http.source_key.v1\0";
const RAW_TRANSACTION_INGEST_YELLOWSTONE_BLOCK_HTTP_ACQUISITION_METHOD: &str = "block_subscribe_get_block";
const RAW_TRANSACTION_INGEST_YELLOWSTONE_BLOCK_RECONCILIATION_ATTEMPTS: u8 = 4;
const RAW_TRANSACTION_INGEST_YELLOWSTONE_BLOCK_RECONCILIATION_DELAY: std::time::Duration = std::time::Duration::from_millis(100);
const RAW_TRANSACTION_INGEST_YELLOWSTONE_COVERAGE_SCOPE_FINGERPRINT_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.yellowstone.coverage_scope.v1\0";
const RAW_TRANSACTION_INGEST_YELLOWSTONE_FILTER_FINGERPRINT_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.yellowstone.filters.v1\0";
const RAW_TRANSACTION_INGEST_YELLOWSTONE_HTTP_PROTOCOL: &str = "yellowstone_http";
@@ -371,10 +374,15 @@ impl RawTransactionIngestLiveSource {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
let known_reference_hydration =
match http_role_supports_rpc_method(&source.http_pool, &source.hydration_role, "getTransaction", source.network.as_str()) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
let reference_bearing =
source.subscribe_request.transaction_filter_count() > 0 || source.subscribe_request.transaction_status_filter_count() > 0;
let live_block_material = source.subscribe_request.block_filter_count() > 0;
(reference_bearing, live_block_material, true, http_block_scan, true)
(reference_bearing, live_block_material, true, http_block_scan, known_reference_hydration)
},
};
let block_material = live_block_material || http_block_scan;
@@ -404,7 +412,11 @@ impl RawTransactionIngestLiveSource {
}
fn uses_hydration(&self) -> bool {
return matches!(self, Self::HeliusTransaction(_) | Self::StandardLogs(_) | Self::Yellowstone(_));
return match self {
Self::HeliusTransaction(_) | Self::StandardLogs(_) => true,
Self::Yellowstone(source) => source.uses_transaction_hydration(),
Self::HttpBlockPolling(_) | Self::StandardBlock(_) => false,
};
}
async fn run(
@@ -975,6 +987,12 @@ struct RawTransactionIngestHydrationContext {
source_key_domain: &'static [u8],
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum RawTransactionIngestYellowstoneMode {
TransactionHydration,
BlockHydration,
}
/// Validated Yellowstone plus HTTP runtime source owned by the continuous RAW transaction ingest Worker.
///
/// The Transport-owned channel, subscribe request, HTTP pool and hydration role remain private. Construction validates deterministic source-composition
@@ -984,6 +1002,7 @@ pub struct RawTransactionIngestYellowstoneSource {
subscribe_request: ksp_onchain_transport_lib::YellowstoneSubscribeRequest,
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
mode: RawTransactionIngestYellowstoneMode,
network: ksp_store_lib::RawNetworkId,
route: RawTransactionIngestSourceRoute,
source_key: [u8; 32],
@@ -1023,7 +1042,19 @@ impl crate::RawTransactionIngestYellowstoneSource {
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("runtime_resources.yellowstone_endpoint_unrepresentable")),
};
let route = RawTransactionIngestSourceRoute { endpoint_id, provider };
let method = match ksp_onchain_transport_lib::find_http_rpc_method("getTransaction") {
let mode = if subscribe_request.block_filter_count() > 0
&& subscribe_request.transaction_filter_count() == 0
&& subscribe_request.transaction_status_filter_count() == 0
{
RawTransactionIngestYellowstoneMode::BlockHydration
} else {
RawTransactionIngestYellowstoneMode::TransactionHydration
};
let method_name = match mode {
RawTransactionIngestYellowstoneMode::TransactionHydration => "getTransaction",
RawTransactionIngestYellowstoneMode::BlockHydration => "getBlock",
};
let method = match ksp_onchain_transport_lib::find_http_rpc_method(method_name) {
std::option::Option::Some(value) => value,
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_method_missing")),
};
@@ -1040,7 +1071,11 @@ impl crate::RawTransactionIngestYellowstoneSource {
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("runtime_resources.yellowstone_identity_invalid")),
};
let source_key = yellowstone_live_source_key(&network, &route, subscribe_request.commitment(), &request_identity);
return std::result::Result::Ok(Self { yellowstone_channel, subscribe_request, http_pool, hydration_role, network, route, source_key });
return std::result::Result::Ok(Self { yellowstone_channel, subscribe_request, http_pool, hydration_role, mode, network, route, source_key });
}
fn uses_transaction_hydration(&self) -> bool {
return self.mode == RawTransactionIngestYellowstoneMode::TransactionHydration;
}
fn hydration_context(&self) -> RawTransactionIngestHydrationContext {
@@ -1070,6 +1105,9 @@ impl crate::RawTransactionIngestYellowstoneSource {
processing_frontier_sender: tokio::sync::watch::Sender<crate::RawTransactionIngestProcessingFrontierProjection>,
shared: RawTransactionIngestSourceRuntimeShared,
) -> ksp_core_lib::Result<()> {
if self.mode == RawTransactionIngestYellowstoneMode::BlockHydration {
return self.run_block_hydration(settings, stop_receiver, admission_sender, processing_frontier_sender).await;
}
let RawTransactionIngestSourceRuntimeShared { continuity_contracts, global_hydration_registry, hydration_in_flight_limit, hydration_pending_limit } =
shared;
let opened = tokio::select! {
@@ -1197,6 +1235,119 @@ impl crate::RawTransactionIngestYellowstoneSource {
},
};
}
async fn run_block_hydration(
self,
settings: crate::RawTransactionIngestSettings,
mut stop_receiver: tokio::sync::watch::Receiver<bool>,
admission_sender: tokio::sync::mpsc::Sender<crate::RawTransactionIngress>,
processing_frontier_sender: tokio::sync::watch::Sender<crate::RawTransactionIngestProcessingFrontierProjection>,
) -> ksp_core_lib::Result<()> {
let opened = tokio::select! {
biased;
_ = stop_receiver.changed() => {
return std::result::Result::Ok(());
}
result = self.yellowstone_channel.open_standard_subscribe(self.subscribe_request.clone()) => result,
};
let mut session = match opened {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(source_transport_error(error.code())),
};
let mut processing_frontier = RawTransactionIngestProcessingFrontierReporter::new(processing_frontier_sender);
let snapshot_source = session.snapshot_source();
if let std::result::Result::Err(error) = processing_frontier.observe_session_snapshot(snapshot_source.current()) {
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Failed);
let _closed = session.close().await;
return std::result::Result::Err(error);
}
let mut session_snapshot_source = std::option::Option::Some(snapshot_source);
let mut fault = std::option::Option::None;
loop {
if *stop_receiver.borrow() {
break;
}
tokio::select! {
biased;
_ = stop_receiver.changed() => {
break;
}
source_snapshot = wait_yellowstone_session_snapshot(&mut session_snapshot_source) => {
match source_snapshot {
std::option::Option::Some(snapshot) => {
if let std::result::Result::Err(error) = processing_frontier.observe_session_snapshot(snapshot) {
fault = std::option::Option::Some(error);
break;
}
},
std::option::Option::None => session_snapshot_source = std::option::Option::None,
}
}
update = session.next_update() => {
let update = match update {
std::result::Result::Ok(std::option::Option::Some(value)) => value,
std::result::Result::Ok(std::option::Option::None) => {
fault = std::option::Option::Some(crate::runtime_error("source.session_closed"));
break;
},
std::result::Result::Err(error) => {
fault = std::option::Option::Some(source_transport_error(error.code()));
break;
},
};
if let ksp_onchain_transport_lib::YellowstoneSubscribeUpdate::Block(value) = update {
let fetched = fetch_yellowstone_block_ingresses(&self, &settings, value.slot(), &mut stop_receiver).await;
let ingresses = match fetched {
std::result::Result::Ok(std::option::Option::Some(value)) => value,
std::result::Result::Ok(std::option::Option::None) => break,
std::result::Result::Err(error) => {
fault = std::option::Option::Some(error);
break;
},
};
for ingress in ingresses {
let sent = tokio::select! {
biased;
_ = stop_receiver.changed() => {
break;
}
result = admission_sender.send(ingress) => result,
};
if sent.is_err() {
if !*stop_receiver.borrow() {
fault = std::option::Option::Some(crate::runtime_error("source.admission_closed"));
}
break;
}
}
if fault.is_some() || *stop_receiver.borrow() {
break;
}
if let std::result::Result::Err(error) = processing_frontier.observe_settled(value.slot()) {
fault = std::option::Option::Some(error);
break;
}
}
}
}
}
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Closing);
let closed = session.close().await;
if let std::option::Option::Some(error) = fault {
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Failed);
return std::result::Result::Err(error);
}
return match closed {
std::result::Result::Ok(()) => {
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Closed);
std::result::Result::Ok(())
},
std::result::Result::Err(error) => {
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Failed);
std::result::Result::Err(source_transport_error(error.code()))
},
};
}
}
/// Validated Helius `transactionSubscribe` plus HTTP hydration source owned by the continuous RAW transaction ingest Worker.
@@ -3198,6 +3349,125 @@ fn validate_http_block_polling_discovery(next_scan_slot: u64, discovered: &[u64]
return std::result::Result::Ok(());
}
async fn fetch_yellowstone_block_ingresses(
source: &crate::RawTransactionIngestYellowstoneSource,
settings: &crate::RawTransactionIngestSettings,
slot: u64,
stop_receiver: &mut tokio::sync::watch::Receiver<bool>,
) -> ksp_core_lib::Result<std::option::Option<std::vec::Vec<crate::RawTransactionIngress>>> {
let commitment = match source.subscribe_request.commitment() {
std::option::Option::Some(value) => value,
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("source.yellowstone_commitment_missing")),
};
let config = ksp_onchain_transport_lib::SolanaGetBlockConfig::new(
std::option::Option::Some(commitment),
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionEncoding::Base64),
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionDetails::Full),
std::option::Option::Some(1),
std::option::Option::Some(false),
);
let mut attempt = 0_u8;
let observed = loop {
attempt = attempt.saturating_add(1);
let request = tokio::select! {
biased;
_ = stop_receiver.changed() => {
return std::result::Result::Ok(std::option::Option::None);
}
result = source.http_pool.get_block_observed(&source.hydration_role, slot, std::option::Option::Some(&config)) => result,
};
match request {
std::result::Result::Ok(value) if value.value().is_some() => break value,
std::result::Result::Ok(_) if attempt < RAW_TRANSACTION_INGEST_YELLOWSTONE_BLOCK_RECONCILIATION_ATTEMPTS => {},
std::result::Result::Ok(_) => {
return std::result::Result::Err(crate::runtime_error("source.yellowstone_block_not_available"));
},
std::result::Result::Err(_) if attempt < RAW_TRANSACTION_INGEST_YELLOWSTONE_BLOCK_RECONCILIATION_ATTEMPTS => {},
std::result::Result::Err(error) => return std::result::Result::Err(source_transport_error(error.code())),
}
tokio::select! {
biased;
_ = stop_receiver.changed() => {
return std::result::Result::Ok(std::option::Option::None);
}
_ = tokio::time::sleep(RAW_TRANSACTION_INGEST_YELLOWSTONE_BLOCK_RECONCILIATION_DELAY) => {},
}
};
let block = match observed.value() {
std::option::Option::Some(value) => value,
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("source.yellowstone_block_not_available")),
};
let received_at = match current_raw_timestamp() {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
let transactions = match block.transactions() {
ksp_onchain_transport_lib::SolanaWireField::Value(value) => value,
ksp_onchain_transport_lib::SolanaWireField::Omitted | ksp_onchain_transport_lib::SolanaWireField::Null => {
return std::result::Result::Err(crate::runtime_error("source.yellowstone_block_transactions_missing"));
},
};
let provenance = build_yellowstone_block_provenance(source, settings, observed.endpoint_name(), observed.provider(), received_at);
let provenance = match provenance {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
let mut ingresses = std::vec::Vec::with_capacity(transactions.len());
for (position, transaction) in transactions.iter().enumerate() {
let material = build_http_block_polling_material_from_transaction(&source.network, slot, block.block_time(), transaction, position);
let material = match material {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
ingresses.push(crate::RawTransactionIngress {
material,
network: source.network.clone(),
provenance: provenance.clone(),
source_key: source.source_key,
});
}
return std::result::Result::Ok(std::option::Option::Some(ingresses));
}
fn build_yellowstone_block_provenance(
source: &crate::RawTransactionIngestYellowstoneSource,
settings: &crate::RawTransactionIngestSettings,
endpoint_name: &str,
provider_name: &ksp_onchain_transport_lib::HttpProviderName,
received_at: ksp_store_lib::RawTimestamp,
) -> ksp_core_lib::Result<ksp_store_lib::RawAcquisitionProvenance> {
let (provider, endpoint_id) = match composite_provenance_codes(&source.route, "ys", provider_name.as_str(), endpoint_name) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
let protocol = match ksp_store_lib::RawProvenanceCode::new(RAW_TRANSACTION_INGEST_YELLOWSTONE_HTTP_PROTOCOL) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("source.yellowstone_block_protocol_unrepresentable")),
};
let acquisition_method = match ksp_store_lib::RawProvenanceCode::new(RAW_TRANSACTION_INGEST_YELLOWSTONE_BLOCK_HTTP_ACQUISITION_METHOD) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("source.yellowstone_block_method_unrepresentable")),
};
let capture_session = match ksp_store_lib::RawProvenanceCode::new(settings.worker_id().as_str()) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("source.yellowstone_block_capture_session_unrepresentable")),
};
let commitment = match source.subscribe_request.commitment() {
std::option::Option::Some(value) => value,
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("source.yellowstone_commitment_missing")),
};
let commitment = match ksp_store_lib::RawProvenanceCode::new(commitment.as_str()) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("source.yellowstone_block_commitment_unrepresentable")),
};
return std::result::Result::Ok(
ksp_store_lib::RawAcquisitionProvenance::new(provider, protocol, acquisition_method, ksp_store_lib::RawAcquisitionOrigin::Live, received_at)
.with_endpoint_id(endpoint_id)
.with_commitment(commitment)
.with_capture_session_id(capture_session),
);
}
fn project_http_block_polling_ingresses(
source: &crate::RawTransactionIngestHttpBlockPollingSource,
settings: &crate::RawTransactionIngestSettings,

75
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
View File

@@ -1,5 +1,5 @@
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
// version: 35
// version: 36
fn grpc_endpoint(cluster: &str) -> std::option::Option<ksp_onchain_transport_lib::YellowstoneGrpcEndpointSettings> {
return grpc_endpoint_with_identity(cluster, "yellowstone-fixture", "fixture-provider");
@@ -359,6 +359,25 @@ fn transaction_request(
return std::option::Option::Some(request);
}
fn block_request(
commitment: std::option::Option<ksp_onchain_transport_lib::SolanaCommitment>,
) -> std::option::Option<ksp_onchain_transport_lib::YellowstoneSubscribeRequest> {
let name = match ksp_onchain_transport_lib::YellowstoneSubscribeFilterName::new("block-fixture") {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::option::Option::None,
};
let mut filter = ksp_onchain_transport_lib::YellowstoneSubscribeBlockFilter::new();
filter.set_include_transactions(std::option::Option::Some(false));
filter.set_include_accounts(std::option::Option::Some(false));
filter.set_include_entries(std::option::Option::Some(false));
let mut request = ksp_onchain_transport_lib::YellowstoneSubscribeRequest::new();
if request.insert_block_filter(name, filter).is_err() {
return std::option::Option::None;
}
request.set_commitment(commitment);
return std::option::Option::Some(request);
}
struct SignalViewFixture {
created_at: std::option::Option<ksp_onchain_transport_lib::YellowstoneUpdateTimestamp>,
family: super::RawTransactionIngestSourceFamily,
@@ -836,6 +855,60 @@ async fn pre_002_source_accepts_matching_confirmed_transaction_and_get_transacti
return;
}
#[tokio::test(flavor = "current_thread")]
async fn pre_009_yellowstone_block_mode_uses_get_block_without_transaction_hydration() {
let endpoint = match grpc_endpoint("devnet") {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let channel = match ksp_onchain_transport_lib::YellowstoneGrpcChannel::prepare(&endpoint) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return,
};
let request = match block_request(std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Confirmed)) {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let pool = match http_pool("devnet", "hydration", "get_block") {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let source = crate::RawTransactionIngestYellowstoneSource::new(channel, request, pool, ksp_onchain_transport_lib::HttpRoleName::new("hydration"));
let source = match source {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return,
};
assert_eq!(source.mode, super::RawTransactionIngestYellowstoneMode::BlockHydration);
assert!(!source.uses_transaction_hydration());
}
#[tokio::test(flavor = "current_thread")]
async fn pre_009_yellowstone_block_mode_rejects_get_transaction_only_http_role() {
let endpoint = match grpc_endpoint("devnet") {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let channel = match ksp_onchain_transport_lib::YellowstoneGrpcChannel::prepare(&endpoint) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return,
};
let request = match block_request(std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Confirmed)) {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let pool = match http_pool("devnet", "hydration", "get_transaction") {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let source = crate::RawTransactionIngestYellowstoneSource::new(channel, request, pool, ksp_onchain_transport_lib::HttpRoleName::new("hydration"));
let error = match source {
std::result::Result::Ok(_) => return,
std::result::Result::Err(error) => error,
};
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
assert!(error.context().iter().any(|context| return context.value() == "runtime_resources.hydration_role_unsupported"));
}
#[tokio::test(flavor = "current_thread")]
async fn pre_002_source_rejects_processed_and_implicit_commitment() {
for commitment in [std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Processed), std::option::Option::None] {