0.7.44
This commit is contained in:
@@ -74,3 +74,4 @@
|
||||
0.7.41 - Raydium AMM v4 swap decoder v1 : décodage des inner instructions `675kPX...`, extraction pool/state, authority, vaults, mints, routeSource et montants exploitables, matérialisation trades/candles sur transactions OK, matrice AMM v4 passée en `supported`, et validation locale avec invariants trade/candle propres.
|
||||
0.7.42 - Consolidation famille Raydium : audit conservatoire des instructions Raydium non décodées, décodage CLMM legacy `swap`, cleanup des audits remplacés, classification HTTP `getTransaction` comme requête lourde avec retry/backoff de backfill, mapping des événements non-swap prouvés `raydium_clmm` (`increase_liquidity_v2`, `decrease_liquidity_v2`, `open_position_with_token22_nft`, `close_position`) et `raydium_cpmm` (`initialize`, `withdraw`, `collect_creator_fee`), matérialisation de 25 liquidity events, 1 lifecycle event et 2 fee events sur corpus élargi, conservation des non-swaps AMM v4 legacy en audit.
|
||||
0.7.43-E5C - Reprise documentaire et normalisation DEX-first : `0.7.43` est conservé comme point de reprise non clos pour le lot Meteora, la suite est redécoupée par DEX/version séparés, le besoin d’un ledger de décodage/replay est acté, les statuts `known` / `observed` / `decoded` / `materialized` / `verified_by_corpus` deviennent obligatoires, et aucun `program_id` ne doit être marqué vérifié sans preuve/corpus reproductible.
|
||||
0.7.44 - Ledger de décodage/replay DEX : ajout de `k_sol_dex_decode_replay_ledger`, des DTO/entities/queries associés, des re-exports DB/lib, et intégration dans le replay local pour skipper uniquement l’étape de décodage DEX lorsqu’un passage certifié existe pour la même version logique de decoder. Les transactions multi-event ou multi-token restent marquées `unsafe` et sont redécodées sauf option future plus explicite ; le replay continue de reconstruire détection, matérialisation, trades, candles et classifications à partir des events persistés.
|
||||
|
||||
@@ -8,7 +8,7 @@ members = [
|
||||
]
|
||||
|
||||
[workspace.package]
|
||||
version = "0.7.43"
|
||||
version = "0.7.44"
|
||||
edition = "2024"
|
||||
license = "MIT"
|
||||
repository = "https://git.sasedev.com/Sasedev/khadhroony-bobobot"
|
||||
|
||||
30
ROADMAP.md
30
ROADMAP.md
@@ -1035,19 +1035,27 @@ Objectif : figer le point de reprise après saturation de session, clarifier l
|
||||
- maintenir la règle : aucun `program_id` n’est vérifié sans signature/corpus/requête de validation.
|
||||
|
||||
### 6.076. Version `0.7.44` — Ledger de décodage/replay et skip sûr
|
||||
Objectif : empêcher le replay local de rescanner inutilement les transactions/instructions dont le décodage est déjà certain, tout en gardant la possibilité de forcer ou de retraiter les cas ambigus.
|
||||
Objectif : empêcher le replay local de rescanner inutilement les transactions dont le décodage DEX est déjà certifié pour la même version logique de decoder, tout en laissant les tables dérivées se reconstruire normalement.
|
||||
|
||||
À faire :
|
||||
Statut : implémenté en première tranche transaction-level.
|
||||
|
||||
- ajouter une table de suivi type `k_sol_decode_attempts` ou `k_sol_replay_decode_ledger` dans `kb_lib/src/db/schema.rs` ;
|
||||
- stocker `transaction_id`, `signature`, `instruction_id` lorsque disponible, `program_id`, `protocol_name`, `decoder_code`, `decoder_version`, `decode_status`, `certainty`, `event_count`, hash d’entrée, reason/error et timestamps ;
|
||||
- ajouter les entities/dtos/queries associées ;
|
||||
- mettre à jour les re-exports dans `kb_lib/src/db.rs`, puis `kb_lib/src/lib.rs` si nécessaire ;
|
||||
- intégrer le ledger dans `local_pipeline_replay.rs` sans changer la sémantique trade/candle ;
|
||||
- ajouter une option `force` pour ignorer le ledger ;
|
||||
- ne pas skipper automatiquement les transactions multi-token, multi-pool, multi-event ou marquées `partial` / `ambiguous` ;
|
||||
- retraiter les lignes concernées lorsqu’un decoder change de version logique ;
|
||||
- ajouter les diagnostics SQL permettant de mesurer skipped/replayed/ambiguous/forced.
|
||||
Fait :
|
||||
|
||||
- ajout de `k_sol_dex_decode_replay_ledger` dans `kb_lib/src/db/schema.rs` ;
|
||||
- stockage de `transaction_id`, `signature`, `decoder_scope`, `decoder_version`, `decode_status`, `certainty`, `event_count`, `distinct_token_mint_count`, `force_replay_required`, reason et timestamps ;
|
||||
- ajout des entities/dtos/queries associées ;
|
||||
- mise à jour des re-exports dans `kb_lib/src/db.rs` puis `kb_lib/src/lib.rs` ;
|
||||
- intégration dans `local_pipeline_replay.rs` sans changer la sémantique trade/candle : le skip ne concerne que `DexDecodeService`, pas la détection, la matérialisation non-trade, les trades, candles, signaux analytiques ou classifications ;
|
||||
- ajout de `skip_certified_dex_decode` et `force_decode_replay` dans `LocalPipelineReplayConfig` ;
|
||||
- marquage `unsafe` des transactions multi-event ou avec plus de deux mints distincts dans les events décodés ;
|
||||
- version logique initiale `dex_decode.v0.7.44.ledger1`, à incrémenter lorsqu’un decoder change de comportement.
|
||||
|
||||
Reste à faire plus tard :
|
||||
|
||||
- descendre le ledger au niveau instruction/program lorsque nécessaire ;
|
||||
- ajouter un hash d’entrée transaction/instruction pour détecter les mutations de payload ;
|
||||
- exposer l’option `force_decode_replay` dans l’UI si besoin ;
|
||||
- ajouter des diagnostics dédiés dans `local_pipeline_diagnostics`.
|
||||
|
||||
### 6.077. Version `0.7.45` — `meteora_dlmm` séparé
|
||||
Objectif : consolider `meteora_dlmm` comme DEX effectif séparé, avec corpus dédié et events utiles au trading.
|
||||
|
||||
@@ -155,6 +155,24 @@
|
||||
<input id="demoPipeline2ReplayMetadataLimitInput" type="number" min="1" step="1" class="form-control" value="250" />
|
||||
</div>
|
||||
|
||||
<div class="form-check mb-2">
|
||||
<input class="form-check-input" type="checkbox" id="demoPipeline2ReplaySkipCertifiedDexDecodeCheckbox" checked />
|
||||
<label class="form-check-label" for="demoPipeline2ReplaySkipCertifiedDexDecodeCheckbox">
|
||||
Skip certified DEX decode ledger rows
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="form-check mb-3">
|
||||
<input class="form-check-input" type="checkbox" id="demoPipeline2ReplayForceDexDecodeCheckbox" />
|
||||
<label class="form-check-label" for="demoPipeline2ReplayForceDexDecodeCheckbox">
|
||||
Force DEX decode replay
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<p class="small text-body-secondary mb-3">
|
||||
Le skip ne concerne que l’étape de décodage DEX certifiée par le ledger. Le reste du replay continue pour reconstruire les tables dérivées.
|
||||
</p>
|
||||
|
||||
<div class="d-flex gap-2">
|
||||
<button id="demoPipeline2ReplayLocalPipelineButton" type="button" class="btn btn-outline-primary">
|
||||
Replay local pipeline
|
||||
|
||||
@@ -57,10 +57,17 @@ interface LocalPipelineReplayResult {
|
||||
pairCandleErrorCount: number;
|
||||
analyticSignalErrorCount: number;
|
||||
decodedEventCount: number;
|
||||
decodeSkippedCount: number;
|
||||
decodeSkippedEventCount: number;
|
||||
decodeLedgerUpsertCount: number;
|
||||
decodeLedgerUnsafeCount: number;
|
||||
detectionCount: number;
|
||||
tradeEventCount: number;
|
||||
liquidityEventCount: number;
|
||||
poolLifecycleEventCount: number;
|
||||
feeEventCount: number;
|
||||
rewardEventCount: number;
|
||||
poolAdminEventCount: number;
|
||||
pairCandleUpsertCount: number;
|
||||
analyticSignalUpsertCount: number;
|
||||
transactionClassificationCount: number;
|
||||
@@ -365,6 +372,8 @@ document.addEventListener("DOMContentLoaded", async () => {
|
||||
const replayLimitInput = document.querySelector<HTMLInputElement>("#demoPipeline2ReplayLimitInput");
|
||||
const replayMetadataCheckbox = document.querySelector<HTMLInputElement>("#demoPipeline2ReplayMetadataCheckbox");
|
||||
const replayMetadataLimitInput = document.querySelector<HTMLInputElement>("#demoPipeline2ReplayMetadataLimitInput");
|
||||
const replaySkipCertifiedDexDecodeCheckbox = document.querySelector<HTMLInputElement>("#demoPipeline2ReplaySkipCertifiedDexDecodeCheckbox");
|
||||
const replayForceDexDecodeCheckbox = document.querySelector<HTMLInputElement>("#demoPipeline2ReplayForceDexDecodeCheckbox");
|
||||
const replayLocalPipelineButton = document.querySelector<HTMLButtonElement>("#demoPipeline2ReplayLocalPipelineButton");
|
||||
const diagnoseLocalPipelineButton = document.querySelector<HTMLButtonElement>("#demoPipeline2DiagnoseLocalPipelineButton");
|
||||
const validateLocalPipelineButton = document.querySelector<HTMLButtonElement>("#demoPipeline2ValidateLocalPipelineButton");
|
||||
@@ -414,6 +423,8 @@ document.addEventListener("DOMContentLoaded", async () => {
|
||||
!replayLimitInput ||
|
||||
!replayMetadataCheckbox ||
|
||||
!replayMetadataLimitInput ||
|
||||
!replaySkipCertifiedDexDecodeCheckbox ||
|
||||
!replayForceDexDecodeCheckbox ||
|
||||
!replayLocalPipelineButton ||
|
||||
!diagnoseLocalPipelineButton ||
|
||||
!validateLocalPipelineButton ||
|
||||
@@ -649,7 +660,7 @@ document.addEventListener("DOMContentLoaded", async () => {
|
||||
|
||||
appendLogLine(
|
||||
logTextarea,
|
||||
`[ui] launching local pipeline replay limit='${replayLimit ?? "none"}' metadata='${replayMetadataCheckbox.checked ? "yes" : "no"}'`,
|
||||
`[ui] launching local pipeline replay limit='${replayLimit ?? "none"}' metadata='${replayMetadataCheckbox.checked ? "yes" : "no"}' skipDexDecode='${replaySkipCertifiedDexDecodeCheckbox.checked ? "yes" : "no"}' forceDexDecode='${replayForceDexDecodeCheckbox.checked ? "yes" : "no"}'`,
|
||||
);
|
||||
|
||||
try {
|
||||
@@ -659,6 +670,8 @@ document.addEventListener("DOMContentLoaded", async () => {
|
||||
limit: replayLimit,
|
||||
refreshMissingTokenMetadata: replayMetadataCheckbox.checked,
|
||||
tokenMetadataLimit,
|
||||
skipCertifiedDexDecode: replaySkipCertifiedDexDecodeCheckbox.checked,
|
||||
forceDecodeReplay: replayForceDexDecodeCheckbox.checked,
|
||||
},
|
||||
);
|
||||
|
||||
@@ -666,7 +679,7 @@ document.addEventListener("DOMContentLoaded", async () => {
|
||||
|
||||
appendLogLine(
|
||||
logTextarea,
|
||||
`[ui] local pipeline replay completed: ${result.replayedTransactionCount.toString()} replayed, ${result.tradeEventCount.toString()} trades, ${result.liquidityEventCount.toString()} liquidity, ${result.poolLifecycleEventCount.toString()} lifecycle, ${result.pairCandleUpsertCount.toString()} candle upserts, resetDeleted='${result.resetMarketMaterializationDeletedCount.toString()}'`,
|
||||
`[ui] local pipeline replay completed: ${result.replayedTransactionCount.toString()} replayed, ${result.decodeSkippedCount.toString()} decode skipped, ${result.decodeLedgerUpsertCount.toString()} ledger upserts, ${result.decodeLedgerUnsafeCount.toString()} unsafe ledger rows, ${result.tradeEventCount.toString()} trades, ${result.liquidityEventCount.toString()} liquidity, ${result.poolLifecycleEventCount.toString()} lifecycle, ${result.pairCandleUpsertCount.toString()} candle upserts, resetDeleted='${result.resetMarketMaterializationDeletedCount.toString()}'`,
|
||||
);
|
||||
|
||||
await refreshCatalog();
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "kb-demo-app",
|
||||
"private": true,
|
||||
"version": "0.7.43",
|
||||
"version": "0.7.44",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
|
||||
@@ -1748,12 +1748,16 @@ pub(crate) async fn demo_pipeline2_replay_local_pipeline(
|
||||
limit: std::option::Option<i64>,
|
||||
refresh_missing_token_metadata: bool,
|
||||
token_metadata_limit: std::option::Option<i64>,
|
||||
skip_certified_dex_decode: bool,
|
||||
force_decode_replay: bool,
|
||||
) -> Result<kb_lib::LocalPipelineReplayResult, std::string::String> {
|
||||
let config = kb_lib::LocalPipelineReplayConfig {
|
||||
limit,
|
||||
refresh_missing_token_metadata,
|
||||
token_metadata_limit,
|
||||
reset_market_materialization_before_replay: true,
|
||||
skip_certified_dex_decode,
|
||||
force_decode_replay,
|
||||
};
|
||||
let database = state.database.clone();
|
||||
let service = if refresh_missing_token_metadata {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"$schema": "https://schema.tauri.app/config/2",
|
||||
"productName": "kb-demo-app",
|
||||
"version": "0.7.43",
|
||||
"version": "0.7.44",
|
||||
"identifier": "com.sasedev.kb-demo-app",
|
||||
"build": {
|
||||
"beforeDevCommand": "npm run dev",
|
||||
|
||||
@@ -22,6 +22,7 @@ pub use dtos::ChainTransactionDto;
|
||||
pub use dtos::DbMetadataDto;
|
||||
pub use dtos::DbRuntimeEventDto;
|
||||
pub use dtos::DexDecodedEventDto;
|
||||
pub use dtos::DexDecodeReplayLedgerDto;
|
||||
pub use dtos::DexDto;
|
||||
pub use dtos::FeeEventDto;
|
||||
pub use dtos::KnownHttpEndpointDto;
|
||||
@@ -88,6 +89,7 @@ pub use entities::ChainTransactionEntity;
|
||||
pub use entities::DbMetadataEntity;
|
||||
pub use entities::DbRuntimeEventEntity;
|
||||
pub use entities::DexDecodedEventEntity;
|
||||
pub use entities::DexDecodeReplayLedgerEntity;
|
||||
pub use entities::DexEntity;
|
||||
pub use entities::FeeEventEntity;
|
||||
pub use entities::KnownHttpEndpointEntity;
|
||||
@@ -145,6 +147,9 @@ pub use queries::query_dex_decoded_events_get_by_key;
|
||||
pub use queries::query_dex_decoded_events_get_latest_pump_fun_create_payload_by_mint;
|
||||
pub use queries::query_dex_decoded_events_list_by_transaction_id;
|
||||
pub use queries::query_dex_decoded_events_upsert;
|
||||
pub use queries::query_dex_decode_replay_ledger_get_by_signature;
|
||||
pub use queries::query_dex_decode_replay_ledger_get_by_transaction;
|
||||
pub use queries::query_dex_decode_replay_ledger_upsert;
|
||||
pub use queries::query_dexs_get_by_code;
|
||||
pub use queries::query_dexs_list;
|
||||
pub use queries::query_dexs_upsert;
|
||||
|
||||
@@ -10,6 +10,7 @@ mod db_metadata;
|
||||
mod db_runtime_event;
|
||||
mod dex;
|
||||
mod dex_decoded_event;
|
||||
mod dex_decode_replay_ledger;
|
||||
mod fee_event;
|
||||
mod known_http_endpoint;
|
||||
mod known_ws_endpoint;
|
||||
@@ -76,6 +77,7 @@ pub use db_metadata::DbMetadataDto;
|
||||
pub use db_runtime_event::DbRuntimeEventDto;
|
||||
pub use dex::DexDto;
|
||||
pub use dex_decoded_event::DexDecodedEventDto;
|
||||
pub use dex_decode_replay_ledger::DexDecodeReplayLedgerDto;
|
||||
pub use fee_event::FeeEventDto;
|
||||
pub use known_http_endpoint::KnownHttpEndpointDto;
|
||||
pub use known_ws_endpoint::KnownWsEndpointDto;
|
||||
|
||||
130
kb_lib/src/db/dtos/dex_decode_replay_ledger.rs
Normal file
130
kb_lib/src/db/dtos/dex_decode_replay_ledger.rs
Normal file
@@ -0,0 +1,130 @@
|
||||
// file: kb_lib/src/db/dtos/dex_decode_replay_ledger.rs
|
||||
|
||||
//! Application-facing DEX decode replay ledger DTO.
|
||||
|
||||
/// Application-facing replay ledger row for one transaction and decoder version.
|
||||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
|
||||
pub struct DexDecodeReplayLedgerDto {
|
||||
/// Optional numeric primary key.
|
||||
pub id: std::option::Option<i64>,
|
||||
/// Related chain transaction id.
|
||||
pub transaction_id: i64,
|
||||
/// Transaction signature.
|
||||
pub signature: std::string::String,
|
||||
/// Logical decoder scope.
|
||||
pub decoder_scope: std::string::String,
|
||||
/// Logical decoder version.
|
||||
pub decoder_version: std::string::String,
|
||||
/// Decode status.
|
||||
pub decode_status: std::string::String,
|
||||
/// Certainty level used by replay skip logic.
|
||||
pub certainty: std::string::String,
|
||||
/// Number of decoded events produced by this decoder pass.
|
||||
pub event_count: i64,
|
||||
/// Number of distinct token mints observed in decoded event mint fields.
|
||||
pub distinct_token_mint_count: i64,
|
||||
/// Whether replay must force decode for safety.
|
||||
pub force_replay_required: bool,
|
||||
/// Optional status reason.
|
||||
pub status_reason: std::option::Option<std::string::String>,
|
||||
/// Creation timestamp.
|
||||
pub created_at: chrono::DateTime<chrono::Utc>,
|
||||
/// Update timestamp.
|
||||
pub updated_at: chrono::DateTime<chrono::Utc>,
|
||||
}
|
||||
|
||||
impl DexDecodeReplayLedgerDto {
|
||||
/// Status used when a decoder pass produced one or more events.
|
||||
pub const STATUS_DECODED: &'static str = "decoded";
|
||||
/// Status used when a decoder pass completed but produced no event.
|
||||
pub const STATUS_NO_EVENTS: &'static str = "no_events";
|
||||
/// Status used when a decoder pass failed.
|
||||
pub const STATUS_FAILED: &'static str = "failed";
|
||||
/// Certainty used when the ledger row may safely skip a future decode pass.
|
||||
pub const CERTAINTY_SURE: &'static str = "sure";
|
||||
/// Certainty used when the ledger row must not skip a future decode pass.
|
||||
pub const CERTAINTY_UNSAFE: &'static str = "unsafe";
|
||||
|
||||
/// Creates a new DEX decode replay ledger DTO.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn new(
|
||||
transaction_id: i64,
|
||||
signature: std::string::String,
|
||||
decoder_scope: std::string::String,
|
||||
decoder_version: std::string::String,
|
||||
decode_status: std::string::String,
|
||||
certainty: std::string::String,
|
||||
event_count: i64,
|
||||
distinct_token_mint_count: i64,
|
||||
force_replay_required: bool,
|
||||
status_reason: std::option::Option<std::string::String>,
|
||||
) -> Self {
|
||||
let now = chrono::Utc::now();
|
||||
return Self {
|
||||
id: None,
|
||||
transaction_id,
|
||||
signature,
|
||||
decoder_scope,
|
||||
decoder_version,
|
||||
decode_status,
|
||||
certainty,
|
||||
event_count,
|
||||
distinct_token_mint_count,
|
||||
force_replay_required,
|
||||
status_reason,
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
};
|
||||
}
|
||||
|
||||
/// Returns whether this ledger row certifies that DEX decoding can be skipped.
|
||||
pub fn can_skip_decode(&self) -> bool {
|
||||
let status_allows_skip = self.decode_status == Self::STATUS_DECODED
|
||||
|| self.decode_status == Self::STATUS_NO_EVENTS;
|
||||
return status_allows_skip
|
||||
&& self.certainty == Self::CERTAINTY_SURE
|
||||
&& !self.force_replay_required;
|
||||
}
|
||||
}
|
||||
|
||||
impl TryFrom<crate::DexDecodeReplayLedgerEntity> for DexDecodeReplayLedgerDto {
|
||||
type Error = crate::Error;
|
||||
|
||||
fn try_from(entity: crate::DexDecodeReplayLedgerEntity) -> Result<Self, Self::Error> {
|
||||
let created_at_result = chrono::DateTime::parse_from_rfc3339(&entity.created_at);
|
||||
let created_at = match created_at_result {
|
||||
Ok(created_at) => created_at.with_timezone(&chrono::Utc),
|
||||
Err(error) => {
|
||||
return Err(crate::Error::Db(format!(
|
||||
"cannot parse dex decode replay ledger created_at '{}': {}",
|
||||
entity.created_at, error
|
||||
)));
|
||||
},
|
||||
};
|
||||
let updated_at_result = chrono::DateTime::parse_from_rfc3339(&entity.updated_at);
|
||||
let updated_at = match updated_at_result {
|
||||
Ok(updated_at) => updated_at.with_timezone(&chrono::Utc),
|
||||
Err(error) => {
|
||||
return Err(crate::Error::Db(format!(
|
||||
"cannot parse dex decode replay ledger updated_at '{}': {}",
|
||||
entity.updated_at, error
|
||||
)));
|
||||
},
|
||||
};
|
||||
return Ok(Self {
|
||||
id: Some(entity.id),
|
||||
transaction_id: entity.transaction_id,
|
||||
signature: entity.signature,
|
||||
decoder_scope: entity.decoder_scope,
|
||||
decoder_version: entity.decoder_version,
|
||||
decode_status: entity.decode_status,
|
||||
certainty: entity.certainty,
|
||||
event_count: entity.event_count,
|
||||
distinct_token_mint_count: entity.distinct_token_mint_count,
|
||||
force_replay_required: entity.force_replay_required != 0,
|
||||
status_reason: entity.status_reason,
|
||||
created_at,
|
||||
updated_at,
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -12,6 +12,7 @@ mod db_metadata;
|
||||
mod db_runtime_event;
|
||||
mod dex;
|
||||
mod dex_decoded_event;
|
||||
mod dex_decode_replay_ledger;
|
||||
mod fee_event;
|
||||
mod known_http_endpoint;
|
||||
mod known_ws_endpoint;
|
||||
@@ -54,6 +55,7 @@ pub use db_metadata::DbMetadataEntity;
|
||||
pub use db_runtime_event::DbRuntimeEventEntity;
|
||||
pub use dex::DexEntity;
|
||||
pub use dex_decoded_event::DexDecodedEventEntity;
|
||||
pub use dex_decode_replay_ledger::DexDecodeReplayLedgerEntity;
|
||||
pub use fee_event::FeeEventEntity;
|
||||
pub use known_http_endpoint::KnownHttpEndpointEntity;
|
||||
pub use known_ws_endpoint::KnownWsEndpointEntity;
|
||||
|
||||
34
kb_lib/src/db/entities/dex_decode_replay_ledger.rs
Normal file
34
kb_lib/src/db/entities/dex_decode_replay_ledger.rs
Normal file
@@ -0,0 +1,34 @@
|
||||
// file: kb_lib/src/db/entities/dex_decode_replay_ledger.rs
|
||||
|
||||
//! Database entity for DEX decode replay ledger rows.
|
||||
|
||||
/// Persisted replay ledger row for one transaction and decoder version.
|
||||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, sqlx::FromRow)]
|
||||
pub struct DexDecodeReplayLedgerEntity {
|
||||
/// Internal row id.
|
||||
pub id: i64,
|
||||
/// Related chain transaction id.
|
||||
pub transaction_id: i64,
|
||||
/// Transaction signature.
|
||||
pub signature: std::string::String,
|
||||
/// Logical decoder scope.
|
||||
pub decoder_scope: std::string::String,
|
||||
/// Logical decoder version.
|
||||
pub decoder_version: std::string::String,
|
||||
/// Decode status.
|
||||
pub decode_status: std::string::String,
|
||||
/// Certainty level used by replay skip logic.
|
||||
pub certainty: std::string::String,
|
||||
/// Number of decoded events produced by this decoder pass.
|
||||
pub event_count: i64,
|
||||
/// Number of distinct token mints observed in decoded event mint fields.
|
||||
pub distinct_token_mint_count: i64,
|
||||
/// Whether replay must force decode for safety.
|
||||
pub force_replay_required: i64,
|
||||
/// Optional status reason.
|
||||
pub status_reason: std::option::Option<std::string::String>,
|
||||
/// Creation timestamp.
|
||||
pub created_at: std::string::String,
|
||||
/// Update timestamp.
|
||||
pub updated_at: std::string::String,
|
||||
}
|
||||
@@ -10,6 +10,7 @@ mod db_metadata;
|
||||
mod db_runtime_event;
|
||||
mod dex;
|
||||
mod dex_decoded_event;
|
||||
mod dex_decode_replay_ledger;
|
||||
mod fee_event;
|
||||
mod known_http_endpoint;
|
||||
mod known_ws_endpoint;
|
||||
@@ -70,6 +71,9 @@ pub use dex_decoded_event::query_dex_decoded_events_get_by_key;
|
||||
pub use dex_decoded_event::query_dex_decoded_events_get_latest_pump_fun_create_payload_by_mint;
|
||||
pub use dex_decoded_event::query_dex_decoded_events_list_by_transaction_id;
|
||||
pub use dex_decoded_event::query_dex_decoded_events_upsert;
|
||||
pub use dex_decode_replay_ledger::query_dex_decode_replay_ledger_get_by_signature;
|
||||
pub use dex_decode_replay_ledger::query_dex_decode_replay_ledger_get_by_transaction;
|
||||
pub use dex_decode_replay_ledger::query_dex_decode_replay_ledger_upsert;
|
||||
pub use fee_event::query_fee_events_get_by_decoded_event_id;
|
||||
pub use fee_event::query_fee_events_list_recent;
|
||||
pub use fee_event::query_fee_events_upsert;
|
||||
|
||||
285
kb_lib/src/db/queries/dex_decode_replay_ledger.rs
Normal file
285
kb_lib/src/db/queries/dex_decode_replay_ledger.rs
Normal file
@@ -0,0 +1,285 @@
|
||||
// file: kb_lib/src/db/queries/dex_decode_replay_ledger.rs
|
||||
|
||||
//! Queries for `k_sol_dex_decode_replay_ledger`.
|
||||
|
||||
/// Inserts or updates one DEX decode replay ledger row.
|
||||
pub async fn query_dex_decode_replay_ledger_upsert(
|
||||
database: &crate::Database,
|
||||
dto: &crate::DexDecodeReplayLedgerDto,
|
||||
) -> Result<i64, crate::Error> {
|
||||
let force_replay_required = if dto.force_replay_required { 1_i64 } else { 0_i64 };
|
||||
match database.connection() {
|
||||
crate::DatabaseConnection::Sqlite(pool) => {
|
||||
let query_result = sqlx::query(
|
||||
r#"
|
||||
INSERT INTO k_sol_dex_decode_replay_ledger (
|
||||
transaction_id,
|
||||
signature,
|
||||
decoder_scope,
|
||||
decoder_version,
|
||||
decode_status,
|
||||
certainty,
|
||||
event_count,
|
||||
distinct_token_mint_count,
|
||||
force_replay_required,
|
||||
status_reason,
|
||||
created_at,
|
||||
updated_at
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(transaction_id, decoder_scope, decoder_version) DO UPDATE SET
|
||||
signature = excluded.signature,
|
||||
decode_status = excluded.decode_status,
|
||||
certainty = excluded.certainty,
|
||||
event_count = excluded.event_count,
|
||||
distinct_token_mint_count = excluded.distinct_token_mint_count,
|
||||
force_replay_required = excluded.force_replay_required,
|
||||
status_reason = excluded.status_reason,
|
||||
updated_at = excluded.updated_at
|
||||
"#,
|
||||
)
|
||||
.bind(dto.transaction_id)
|
||||
.bind(dto.signature.clone())
|
||||
.bind(dto.decoder_scope.clone())
|
||||
.bind(dto.decoder_version.clone())
|
||||
.bind(dto.decode_status.clone())
|
||||
.bind(dto.certainty.clone())
|
||||
.bind(dto.event_count)
|
||||
.bind(dto.distinct_token_mint_count)
|
||||
.bind(force_replay_required)
|
||||
.bind(dto.status_reason.clone())
|
||||
.bind(dto.created_at.to_rfc3339())
|
||||
.bind(dto.updated_at.to_rfc3339())
|
||||
.execute(pool)
|
||||
.await;
|
||||
if let Err(error) = query_result {
|
||||
return Err(crate::Error::Db(format!(
|
||||
"cannot upsert k_sol_dex_decode_replay_ledger on sqlite: {}",
|
||||
error
|
||||
)));
|
||||
}
|
||||
let id_result = sqlx::query_scalar::<sqlx::Sqlite, i64>(
|
||||
r#"
|
||||
SELECT id
|
||||
FROM k_sol_dex_decode_replay_ledger
|
||||
WHERE transaction_id = ?
|
||||
AND decoder_scope = ?
|
||||
AND decoder_version = ?
|
||||
LIMIT 1
|
||||
"#,
|
||||
)
|
||||
.bind(dto.transaction_id)
|
||||
.bind(dto.decoder_scope.clone())
|
||||
.bind(dto.decoder_version.clone())
|
||||
.fetch_one(pool)
|
||||
.await;
|
||||
match id_result {
|
||||
Ok(id) => return Ok(id),
|
||||
Err(error) => {
|
||||
return Err(crate::Error::Db(format!(
|
||||
"cannot fetch k_sol_dex_decode_replay_ledger id for transaction_id '{}' on sqlite: {}",
|
||||
dto.transaction_id, error
|
||||
)));
|
||||
},
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/// Reads one replay ledger row by transaction id, decoder scope, and decoder version.
|
||||
pub async fn query_dex_decode_replay_ledger_get_by_transaction(
|
||||
database: &crate::Database,
|
||||
transaction_id: i64,
|
||||
decoder_scope: &str,
|
||||
decoder_version: &str,
|
||||
) -> Result<std::option::Option<crate::DexDecodeReplayLedgerDto>, crate::Error> {
|
||||
match database.connection() {
|
||||
crate::DatabaseConnection::Sqlite(pool) => {
|
||||
let query_result =
|
||||
sqlx::query_as::<sqlx::Sqlite, crate::DexDecodeReplayLedgerEntity>(
|
||||
r#"
|
||||
SELECT
|
||||
id,
|
||||
transaction_id,
|
||||
signature,
|
||||
decoder_scope,
|
||||
decoder_version,
|
||||
decode_status,
|
||||
certainty,
|
||||
event_count,
|
||||
distinct_token_mint_count,
|
||||
force_replay_required,
|
||||
status_reason,
|
||||
created_at,
|
||||
updated_at
|
||||
FROM k_sol_dex_decode_replay_ledger
|
||||
WHERE transaction_id = ?
|
||||
AND decoder_scope = ?
|
||||
AND decoder_version = ?
|
||||
LIMIT 1
|
||||
"#,
|
||||
)
|
||||
.bind(transaction_id)
|
||||
.bind(decoder_scope.to_string())
|
||||
.bind(decoder_version.to_string())
|
||||
.fetch_optional(pool)
|
||||
.await;
|
||||
let entity_option = match query_result {
|
||||
Ok(entity_option) => entity_option,
|
||||
Err(error) => {
|
||||
return Err(crate::Error::Db(format!(
|
||||
"cannot fetch k_sol_dex_decode_replay_ledger for transaction_id '{}' on sqlite: {}",
|
||||
transaction_id, error
|
||||
)));
|
||||
},
|
||||
};
|
||||
match entity_option {
|
||||
Some(entity) => {
|
||||
let dto_result = crate::DexDecodeReplayLedgerDto::try_from(entity);
|
||||
match dto_result {
|
||||
Ok(dto) => return Ok(Some(dto)),
|
||||
Err(error) => return Err(error),
|
||||
}
|
||||
},
|
||||
None => return Ok(None),
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/// Reads one replay ledger row by signature, decoder scope, and decoder version.
|
||||
pub async fn query_dex_decode_replay_ledger_get_by_signature(
|
||||
database: &crate::Database,
|
||||
signature: &str,
|
||||
decoder_scope: &str,
|
||||
decoder_version: &str,
|
||||
) -> Result<std::option::Option<crate::DexDecodeReplayLedgerDto>, crate::Error> {
|
||||
match database.connection() {
|
||||
crate::DatabaseConnection::Sqlite(pool) => {
|
||||
let query_result =
|
||||
sqlx::query_as::<sqlx::Sqlite, crate::DexDecodeReplayLedgerEntity>(
|
||||
r#"
|
||||
SELECT
|
||||
id,
|
||||
transaction_id,
|
||||
signature,
|
||||
decoder_scope,
|
||||
decoder_version,
|
||||
decode_status,
|
||||
certainty,
|
||||
event_count,
|
||||
distinct_token_mint_count,
|
||||
force_replay_required,
|
||||
status_reason,
|
||||
created_at,
|
||||
updated_at
|
||||
FROM k_sol_dex_decode_replay_ledger
|
||||
WHERE signature = ?
|
||||
AND decoder_scope = ?
|
||||
AND decoder_version = ?
|
||||
LIMIT 1
|
||||
"#,
|
||||
)
|
||||
.bind(signature.to_string())
|
||||
.bind(decoder_scope.to_string())
|
||||
.bind(decoder_version.to_string())
|
||||
.fetch_optional(pool)
|
||||
.await;
|
||||
let entity_option = match query_result {
|
||||
Ok(entity_option) => entity_option,
|
||||
Err(error) => {
|
||||
return Err(crate::Error::Db(format!(
|
||||
"cannot fetch k_sol_dex_decode_replay_ledger for signature '{}' on sqlite: {}",
|
||||
signature, error
|
||||
)));
|
||||
},
|
||||
};
|
||||
match entity_option {
|
||||
Some(entity) => {
|
||||
let dto_result = crate::DexDecodeReplayLedgerDto::try_from(entity);
|
||||
match dto_result {
|
||||
Ok(dto) => return Ok(Some(dto)),
|
||||
Err(error) => return Err(error),
|
||||
}
|
||||
},
|
||||
None => return Ok(None),
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
async fn make_database() -> crate::Database {
|
||||
let tempdir = tempfile::tempdir().expect("tempdir must succeed");
|
||||
let database_path = tempdir.path().join("dex_decode_replay_ledger.sqlite3");
|
||||
let config = crate::DatabaseConfig {
|
||||
enabled: true,
|
||||
backend: crate::DatabaseBackend::Sqlite,
|
||||
sqlite: crate::SqliteDatabaseConfig {
|
||||
path: database_path.to_string_lossy().to_string(),
|
||||
create_if_missing: true,
|
||||
busy_timeout_ms: 5000,
|
||||
max_connections: 1,
|
||||
auto_initialize_schema: true,
|
||||
use_wal: true,
|
||||
},
|
||||
};
|
||||
return crate::Database::connect_and_initialize(&config)
|
||||
.await
|
||||
.expect("database init must succeed");
|
||||
}
|
||||
|
||||
async fn insert_chain_transaction(database: &crate::Database) -> i64 {
|
||||
let slot_dto = crate::ChainSlotDto::new(777, Some(776), Some(1_700_000_007));
|
||||
crate::query_chain_slots_upsert(database, &slot_dto)
|
||||
.await
|
||||
.expect("slot upsert must succeed");
|
||||
let dto = crate::ChainTransactionDto::new(
|
||||
"ledger_sig".to_string(),
|
||||
Some(777),
|
||||
Some(1_700_000_007),
|
||||
Some("test".to_string()),
|
||||
None,
|
||||
None,
|
||||
Some("{}".to_string()),
|
||||
"{}".to_string(),
|
||||
);
|
||||
return crate::query_chain_transactions_upsert(database, &dto)
|
||||
.await
|
||||
.expect("transaction upsert must succeed");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dex_decode_replay_ledger_roundtrip_works() {
|
||||
let database = make_database().await;
|
||||
let transaction_id = insert_chain_transaction(&database).await;
|
||||
let dto = crate::DexDecodeReplayLedgerDto::new(
|
||||
transaction_id,
|
||||
"ledger_sig".to_string(),
|
||||
"dex_decode.local_pipeline".to_string(),
|
||||
"test-version".to_string(),
|
||||
crate::DexDecodeReplayLedgerDto::STATUS_DECODED.to_string(),
|
||||
crate::DexDecodeReplayLedgerDto::CERTAINTY_SURE.to_string(),
|
||||
1,
|
||||
2,
|
||||
false,
|
||||
Some("single-pair decode completed".to_string()),
|
||||
);
|
||||
let upsert_id = crate::query_dex_decode_replay_ledger_upsert(&database, &dto)
|
||||
.await
|
||||
.expect("ledger upsert must succeed");
|
||||
assert!(upsert_id > 0);
|
||||
let fetched = crate::query_dex_decode_replay_ledger_get_by_signature(
|
||||
&database,
|
||||
"ledger_sig",
|
||||
"dex_decode.local_pipeline",
|
||||
"test-version",
|
||||
)
|
||||
.await
|
||||
.expect("ledger fetch must succeed")
|
||||
.expect("ledger row must exist");
|
||||
assert_eq!(fetched.transaction_id, transaction_id);
|
||||
assert!(fetched.can_skip_decode());
|
||||
}
|
||||
}
|
||||
@@ -230,6 +230,22 @@ pub(crate) async fn ensure_schema(database: &crate::Database) -> Result<(), crat
|
||||
if let Err(error) = result {
|
||||
return Err(error);
|
||||
}
|
||||
let result = create_tbl_dex_decode_replay_ledger(pool).await;
|
||||
if let Err(error) = result {
|
||||
return Err(error);
|
||||
}
|
||||
let result = create_uix_dex_decode_replay_ledger_transaction_scope_version(pool).await;
|
||||
if let Err(error) = result {
|
||||
return Err(error);
|
||||
}
|
||||
let result = create_idx_dex_decode_replay_ledger_signature(pool).await;
|
||||
if let Err(error) = result {
|
||||
return Err(error);
|
||||
}
|
||||
let result = create_idx_dex_decode_replay_ledger_status(pool).await;
|
||||
if let Err(error) = result {
|
||||
return Err(error);
|
||||
}
|
||||
let result = create_tbl_transaction_classifications(pool).await;
|
||||
if let Err(error) = result {
|
||||
return Err(error);
|
||||
@@ -1445,6 +1461,80 @@ ON k_sol_dex_decoded_events (transaction_id, instruction_id, event_kind)
|
||||
.await;
|
||||
}
|
||||
|
||||
/// Creates `k_sol_dex_decode_replay_ledger`.
|
||||
async fn create_tbl_dex_decode_replay_ledger(
|
||||
pool: &sqlx::SqlitePool,
|
||||
) -> Result<(), crate::Error> {
|
||||
return execute_sqlite_schema_statement(
|
||||
pool,
|
||||
"create_tbl_dex_decode_replay_ledger",
|
||||
r#"
|
||||
CREATE TABLE IF NOT EXISTS k_sol_dex_decode_replay_ledger (
|
||||
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
|
||||
transaction_id INTEGER NOT NULL,
|
||||
signature TEXT NOT NULL,
|
||||
decoder_scope TEXT NOT NULL,
|
||||
decoder_version TEXT NOT NULL,
|
||||
decode_status TEXT NOT NULL,
|
||||
certainty TEXT NOT NULL,
|
||||
event_count INTEGER NOT NULL,
|
||||
distinct_token_mint_count INTEGER NOT NULL,
|
||||
force_replay_required INTEGER NOT NULL,
|
||||
status_reason TEXT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
FOREIGN KEY(transaction_id) REFERENCES k_sol_chain_transactions(id)
|
||||
)
|
||||
"#,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
/// Creates unique index on `(transaction_id, decoder_scope, decoder_version)`.
|
||||
async fn create_uix_dex_decode_replay_ledger_transaction_scope_version(
|
||||
pool: &sqlx::SqlitePool,
|
||||
) -> Result<(), crate::Error> {
|
||||
return execute_sqlite_schema_statement(
|
||||
pool,
|
||||
"create_uix_dex_decode_replay_ledger_transaction_scope_version",
|
||||
r#"
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uix_dex_decode_replay_ledger_transaction_scope_version
|
||||
ON k_sol_dex_decode_replay_ledger (transaction_id, decoder_scope, decoder_version)
|
||||
"#,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
/// Creates index on `(signature, decoder_scope, decoder_version)`.
|
||||
async fn create_idx_dex_decode_replay_ledger_signature(
|
||||
pool: &sqlx::SqlitePool,
|
||||
) -> Result<(), crate::Error> {
|
||||
return execute_sqlite_schema_statement(
|
||||
pool,
|
||||
"create_idx_dex_decode_replay_ledger_signature",
|
||||
r#"
|
||||
CREATE INDEX IF NOT EXISTS idx_dex_decode_replay_ledger_signature
|
||||
ON k_sol_dex_decode_replay_ledger (signature, decoder_scope, decoder_version)
|
||||
"#,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
/// Creates index on `(decode_status, certainty, force_replay_required)`.
|
||||
async fn create_idx_dex_decode_replay_ledger_status(
|
||||
pool: &sqlx::SqlitePool,
|
||||
) -> Result<(), crate::Error> {
|
||||
return execute_sqlite_schema_statement(
|
||||
pool,
|
||||
"create_idx_dex_decode_replay_ledger_status",
|
||||
r#"
|
||||
CREATE INDEX IF NOT EXISTS idx_dex_decode_replay_ledger_status
|
||||
ON k_sol_dex_decode_replay_ledger (decode_status, certainty, force_replay_required)
|
||||
"#,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
async fn create_tbl_launch_surfaces(pool: &sqlx::SqlitePool) -> Result<(), crate::Error> {
|
||||
return execute_sqlite_schema_statement(
|
||||
pool,
|
||||
|
||||
@@ -345,8 +345,12 @@ pub use db::DbRuntimeEventEntity;
|
||||
pub use db::DbRuntimeEventLevel;
|
||||
/// Application-facing decoded DEX event DTO.
|
||||
pub use db::DexDecodedEventDto;
|
||||
/// Application-facing DEX decode replay ledger DTO.
|
||||
pub use db::DexDecodeReplayLedgerDto;
|
||||
/// Persisted decoded DEX event row.
|
||||
pub use db::DexDecodedEventEntity;
|
||||
/// Persisted DEX decode replay ledger row.
|
||||
pub use db::DexDecodeReplayLedgerEntity;
|
||||
/// Application-facing normalized DEX DTO.
|
||||
pub use db::DexDto;
|
||||
/// Persisted normalized DEX row.
|
||||
@@ -597,6 +601,12 @@ pub use db::query_dex_decoded_events_get_latest_pump_fun_create_payload_by_mint;
|
||||
pub use db::query_dex_decoded_events_list_by_transaction_id;
|
||||
/// Inserts or updates one decoded DEX event row.
|
||||
pub use db::query_dex_decoded_events_upsert;
|
||||
/// Reads one DEX decode replay ledger row by signature and decoder identity.
|
||||
pub use db::query_dex_decode_replay_ledger_get_by_signature;
|
||||
/// Reads one DEX decode replay ledger row by transaction and decoder identity.
|
||||
pub use db::query_dex_decode_replay_ledger_get_by_transaction;
|
||||
/// Inserts or updates one DEX decode replay ledger row.
|
||||
pub use db::query_dex_decode_replay_ledger_upsert;
|
||||
/// Reads one normalized DEX row by code.
|
||||
pub use db::query_dexs_get_by_code;
|
||||
/// Lists normalized DEX rows.
|
||||
|
||||
@@ -6,6 +6,13 @@
|
||||
//! reuses rows already present in `k_sol_chain_transactions` and replays the
|
||||
//! deterministic local pipeline over their signatures.
|
||||
|
||||
const LOCAL_PIPELINE_DEX_DECODER_SCOPE: &str = "dex_decode.local_pipeline";
|
||||
const LOCAL_PIPELINE_DEX_DECODER_VERSION: &str = "dex_decode.v0.7.44.ledger1";
|
||||
|
||||
fn default_skip_certified_dex_decode() -> bool {
|
||||
return true;
|
||||
}
|
||||
|
||||
/// Configuration for a local pipeline replay pass.
|
||||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
@@ -18,6 +25,12 @@ pub struct LocalPipelineReplayConfig {
|
||||
pub token_metadata_limit: std::option::Option<i64>,
|
||||
/// Whether locally replayed market materialization tables are reset before replay.
|
||||
pub reset_market_materialization_before_replay: bool,
|
||||
/// Whether DEX decoding may be skipped when the replay ledger certifies it is safe.
|
||||
#[serde(default = "default_skip_certified_dex_decode")]
|
||||
pub skip_certified_dex_decode: bool,
|
||||
/// Whether DEX decoding must run even when the replay ledger certifies a safe prior pass.
|
||||
#[serde(default)]
|
||||
pub force_decode_replay: bool,
|
||||
}
|
||||
|
||||
impl Default for LocalPipelineReplayConfig {
|
||||
@@ -27,6 +40,8 @@ impl Default for LocalPipelineReplayConfig {
|
||||
refresh_missing_token_metadata: false,
|
||||
token_metadata_limit: Some(250),
|
||||
reset_market_materialization_before_replay: true,
|
||||
skip_certified_dex_decode: true,
|
||||
force_decode_replay: false,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -53,6 +68,14 @@ pub struct LocalPipelineReplayResult {
|
||||
pub analytic_signal_error_count: usize,
|
||||
/// Total decoded events returned by replayed decode calls.
|
||||
pub decoded_event_count: usize,
|
||||
/// Number of transactions where DEX decoding was skipped through the replay ledger.
|
||||
pub decode_skipped_count: usize,
|
||||
/// Number of persisted decoded events covered by skipped decode ledger rows.
|
||||
pub decode_skipped_event_count: usize,
|
||||
/// Number of replay ledger rows upserted by this replay pass.
|
||||
pub decode_ledger_upsert_count: usize,
|
||||
/// Number of replay ledger rows marked unsafe for future decode skip.
|
||||
pub decode_ledger_unsafe_count: usize,
|
||||
/// Total detection results returned by replayed detect calls.
|
||||
pub detection_count: usize,
|
||||
/// Total trade aggregation results returned by replayed aggregation calls.
|
||||
@@ -170,21 +193,135 @@ impl LocalPipelineReplayService {
|
||||
signature = %signature,
|
||||
"replaying local pipeline for persisted transaction"
|
||||
);
|
||||
let decode_result =
|
||||
dex_decode.decode_transaction_by_signature(signature.as_str()).await;
|
||||
match decode_result {
|
||||
Ok(decoded_events) => {
|
||||
result.decoded_event_count += decoded_events.len();
|
||||
let transaction_result =
|
||||
crate::query_chain_transactions_get_by_signature(self.database.as_ref(), signature.as_str())
|
||||
.await;
|
||||
let transaction = match transaction_result {
|
||||
Ok(Some(transaction)) => transaction,
|
||||
Ok(None) => {
|
||||
result.global_error_count += 1;
|
||||
tracing::warn!(
|
||||
signature = %signature,
|
||||
"local pipeline replay transaction row disappeared before replay"
|
||||
);
|
||||
continue;
|
||||
},
|
||||
Err(error) => {
|
||||
result.decode_error_count += 1;
|
||||
result.global_error_count += 1;
|
||||
tracing::warn!(
|
||||
signature = %signature,
|
||||
error = %error,
|
||||
"local pipeline replay decode step failed"
|
||||
"local pipeline replay transaction lookup failed"
|
||||
);
|
||||
continue;
|
||||
},
|
||||
};
|
||||
let transaction_id = match transaction.id {
|
||||
Some(transaction_id) => transaction_id,
|
||||
None => {
|
||||
result.global_error_count += 1;
|
||||
tracing::warn!(
|
||||
signature = %signature,
|
||||
"local pipeline replay transaction row has no persisted id"
|
||||
);
|
||||
continue;
|
||||
},
|
||||
};
|
||||
let decode_skip_ledger_result = self
|
||||
.get_certified_dex_decode_skip_ledger(config, transaction_id, signature.as_str())
|
||||
.await;
|
||||
let decode_skip_ledger = match decode_skip_ledger_result {
|
||||
Ok(decode_skip_ledger) => decode_skip_ledger,
|
||||
Err(error) => return Err(error),
|
||||
};
|
||||
match decode_skip_ledger {
|
||||
Some(ledger) => {
|
||||
result.decode_skipped_count += 1;
|
||||
let ledger_event_count = usize::try_from(ledger.event_count);
|
||||
match ledger_event_count {
|
||||
Ok(event_count) => {
|
||||
result.decode_skipped_event_count += event_count;
|
||||
},
|
||||
Err(error) => {
|
||||
result.global_error_count += 1;
|
||||
tracing::warn!(
|
||||
signature = %signature,
|
||||
event_count = ledger.event_count,
|
||||
error = %error,
|
||||
"local pipeline replay could not convert skipped event count"
|
||||
);
|
||||
},
|
||||
}
|
||||
tracing::debug!(
|
||||
signature = %signature,
|
||||
event_count = ledger.event_count,
|
||||
decoder_version = %ledger.decoder_version,
|
||||
"local pipeline replay skipped certified DEX decode step"
|
||||
);
|
||||
},
|
||||
None => {
|
||||
let decode_result = dex_decode
|
||||
.decode_transaction_by_signature(signature.as_str())
|
||||
.await;
|
||||
match decode_result {
|
||||
Ok(decoded_events) => {
|
||||
result.decoded_event_count += decoded_events.len();
|
||||
let ledger_result = self
|
||||
.record_dex_decode_replay_ledger(
|
||||
transaction_id,
|
||||
signature.as_str(),
|
||||
&decoded_events,
|
||||
)
|
||||
.await;
|
||||
match ledger_result {
|
||||
Ok(ledger) => {
|
||||
result.decode_ledger_upsert_count += 1;
|
||||
if ledger.force_replay_required {
|
||||
result.decode_ledger_unsafe_count += 1;
|
||||
}
|
||||
},
|
||||
Err(error) => {
|
||||
result.global_error_count += 1;
|
||||
tracing::warn!(
|
||||
signature = %signature,
|
||||
error = %error,
|
||||
"local pipeline replay could not record successful decode ledger row"
|
||||
);
|
||||
},
|
||||
}
|
||||
},
|
||||
Err(error) => {
|
||||
result.decode_error_count += 1;
|
||||
let ledger_result = self
|
||||
.record_failed_dex_decode_replay_ledger(
|
||||
transaction_id,
|
||||
signature.as_str(),
|
||||
error.to_string(),
|
||||
)
|
||||
.await;
|
||||
match ledger_result {
|
||||
Ok(_) => {
|
||||
result.decode_ledger_upsert_count += 1;
|
||||
result.decode_ledger_unsafe_count += 1;
|
||||
},
|
||||
Err(ledger_error) => {
|
||||
result.global_error_count += 1;
|
||||
tracing::warn!(
|
||||
signature = %signature,
|
||||
error = %ledger_error,
|
||||
"local pipeline replay could not record failed decode ledger row"
|
||||
);
|
||||
},
|
||||
}
|
||||
tracing::warn!(
|
||||
signature = %signature,
|
||||
error = %error,
|
||||
"local pipeline replay decode step failed"
|
||||
);
|
||||
continue;
|
||||
},
|
||||
}
|
||||
},
|
||||
}
|
||||
let detect_result =
|
||||
dex_detect.detect_transaction_by_signature(signature.as_str()).await;
|
||||
@@ -315,6 +452,236 @@ impl LocalPipelineReplayService {
|
||||
}
|
||||
return Ok(result);
|
||||
}
|
||||
|
||||
async fn get_certified_dex_decode_skip_ledger(
|
||||
&self,
|
||||
config: &crate::LocalPipelineReplayConfig,
|
||||
transaction_id: i64,
|
||||
signature: &str,
|
||||
) -> Result<std::option::Option<crate::DexDecodeReplayLedgerDto>, crate::Error> {
|
||||
if config.force_decode_replay {
|
||||
return Ok(None);
|
||||
}
|
||||
if !config.skip_certified_dex_decode {
|
||||
return Ok(None);
|
||||
}
|
||||
let ledger_result = crate::query_dex_decode_replay_ledger_get_by_transaction(
|
||||
self.database.as_ref(),
|
||||
transaction_id,
|
||||
LOCAL_PIPELINE_DEX_DECODER_SCOPE,
|
||||
LOCAL_PIPELINE_DEX_DECODER_VERSION,
|
||||
)
|
||||
.await;
|
||||
let ledger_option = match ledger_result {
|
||||
Ok(ledger_option) => ledger_option,
|
||||
Err(error) => return Err(error),
|
||||
};
|
||||
match ledger_option {
|
||||
Some(ledger) => {
|
||||
if ledger.can_skip_decode() {
|
||||
let persisted_count_result = self
|
||||
.count_persisted_decoded_events_for_skip(transaction_id, signature)
|
||||
.await;
|
||||
let persisted_count = match persisted_count_result {
|
||||
Ok(persisted_count) => persisted_count,
|
||||
Err(error) => return Err(error),
|
||||
};
|
||||
if persisted_count >= ledger.event_count {
|
||||
return Ok(Some(ledger));
|
||||
}
|
||||
tracing::debug!(
|
||||
signature = %signature,
|
||||
ledger_event_count = ledger.event_count,
|
||||
persisted_event_count = persisted_count,
|
||||
"local pipeline replay ledger is certified but persisted decoded events are missing"
|
||||
);
|
||||
return Ok(None);
|
||||
}
|
||||
tracing::debug!(
|
||||
signature = %signature,
|
||||
decode_status = %ledger.decode_status,
|
||||
certainty = %ledger.certainty,
|
||||
force_replay_required = ledger.force_replay_required,
|
||||
"local pipeline replay ledger requires DEX decode"
|
||||
);
|
||||
return Ok(None);
|
||||
},
|
||||
None => return Ok(None),
|
||||
}
|
||||
}
|
||||
|
||||
async fn count_persisted_decoded_events_for_skip(
|
||||
&self,
|
||||
transaction_id: i64,
|
||||
signature: &str,
|
||||
) -> Result<i64, crate::Error> {
|
||||
let events_result = crate::query_dex_decoded_events_list_by_transaction_id(
|
||||
self.database.as_ref(),
|
||||
transaction_id,
|
||||
)
|
||||
.await;
|
||||
let events = match events_result {
|
||||
Ok(events) => events,
|
||||
Err(error) => return Err(error),
|
||||
};
|
||||
let count_result = i64::try_from(events.len());
|
||||
match count_result {
|
||||
Ok(count) => return Ok(count),
|
||||
Err(error) => {
|
||||
return Err(crate::Error::Db(format!(
|
||||
"cannot convert persisted decoded event count for signature '{}' to i64: {}",
|
||||
signature, error
|
||||
)));
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
async fn record_dex_decode_replay_ledger(
|
||||
&self,
|
||||
transaction_id: i64,
|
||||
signature: &str,
|
||||
decoded_events: &[crate::DexDecodedEventDto],
|
||||
) -> Result<crate::DexDecodeReplayLedgerDto, crate::Error> {
|
||||
let ledger_result = build_success_dex_decode_replay_ledger(
|
||||
transaction_id,
|
||||
signature,
|
||||
decoded_events,
|
||||
);
|
||||
let ledger = match ledger_result {
|
||||
Ok(ledger) => ledger,
|
||||
Err(error) => return Err(error),
|
||||
};
|
||||
let upsert_result =
|
||||
crate::query_dex_decode_replay_ledger_upsert(self.database.as_ref(), &ledger).await;
|
||||
match upsert_result {
|
||||
Ok(_) => return Ok(ledger),
|
||||
Err(error) => return Err(error),
|
||||
}
|
||||
}
|
||||
|
||||
async fn record_failed_dex_decode_replay_ledger(
|
||||
&self,
|
||||
transaction_id: i64,
|
||||
signature: &str,
|
||||
error_message: std::string::String,
|
||||
) -> Result<crate::DexDecodeReplayLedgerDto, crate::Error> {
|
||||
let ledger = crate::DexDecodeReplayLedgerDto::new(
|
||||
transaction_id,
|
||||
signature.to_string(),
|
||||
LOCAL_PIPELINE_DEX_DECODER_SCOPE.to_string(),
|
||||
LOCAL_PIPELINE_DEX_DECODER_VERSION.to_string(),
|
||||
crate::DexDecodeReplayLedgerDto::STATUS_FAILED.to_string(),
|
||||
crate::DexDecodeReplayLedgerDto::CERTAINTY_UNSAFE.to_string(),
|
||||
0,
|
||||
0,
|
||||
true,
|
||||
Some(format!("decode failed: {error_message}")),
|
||||
);
|
||||
let upsert_result =
|
||||
crate::query_dex_decode_replay_ledger_upsert(self.database.as_ref(), &ledger).await;
|
||||
match upsert_result {
|
||||
Ok(_) => return Ok(ledger),
|
||||
Err(error) => return Err(error),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn build_success_dex_decode_replay_ledger(
|
||||
transaction_id: i64,
|
||||
signature: &str,
|
||||
decoded_events: &[crate::DexDecodedEventDto],
|
||||
) -> Result<crate::DexDecodeReplayLedgerDto, crate::Error> {
|
||||
let event_count_result = i64::try_from(decoded_events.len());
|
||||
let event_count = match event_count_result {
|
||||
Ok(event_count) => event_count,
|
||||
Err(error) => {
|
||||
return Err(crate::Error::Db(format!(
|
||||
"cannot convert decoded event count '{}' to i64: {}",
|
||||
decoded_events.len(), error
|
||||
)));
|
||||
},
|
||||
};
|
||||
let distinct_token_mint_count_usize = count_distinct_decoded_event_token_mints(decoded_events);
|
||||
let distinct_token_mint_count_result = i64::try_from(distinct_token_mint_count_usize);
|
||||
let distinct_token_mint_count = match distinct_token_mint_count_result {
|
||||
Ok(distinct_token_mint_count) => distinct_token_mint_count,
|
||||
Err(error) => {
|
||||
return Err(crate::Error::Db(format!(
|
||||
"cannot convert distinct token mint count '{}' to i64: {}",
|
||||
distinct_token_mint_count_usize, error
|
||||
)));
|
||||
},
|
||||
};
|
||||
let force_replay_required = event_count > 1 || distinct_token_mint_count > 2;
|
||||
let decode_status = if event_count == 0 {
|
||||
crate::DexDecodeReplayLedgerDto::STATUS_NO_EVENTS.to_string()
|
||||
} else {
|
||||
crate::DexDecodeReplayLedgerDto::STATUS_DECODED.to_string()
|
||||
};
|
||||
let certainty = if force_replay_required {
|
||||
crate::DexDecodeReplayLedgerDto::CERTAINTY_UNSAFE.to_string()
|
||||
} else {
|
||||
crate::DexDecodeReplayLedgerDto::CERTAINTY_SURE.to_string()
|
||||
};
|
||||
let status_reason = build_dex_decode_replay_ledger_status_reason(
|
||||
event_count,
|
||||
distinct_token_mint_count,
|
||||
force_replay_required,
|
||||
);
|
||||
return Ok(crate::DexDecodeReplayLedgerDto::new(
|
||||
transaction_id,
|
||||
signature.to_string(),
|
||||
LOCAL_PIPELINE_DEX_DECODER_SCOPE.to_string(),
|
||||
LOCAL_PIPELINE_DEX_DECODER_VERSION.to_string(),
|
||||
decode_status,
|
||||
certainty,
|
||||
event_count,
|
||||
distinct_token_mint_count,
|
||||
force_replay_required,
|
||||
Some(status_reason),
|
||||
));
|
||||
}
|
||||
|
||||
fn count_distinct_decoded_event_token_mints(
|
||||
decoded_events: &[crate::DexDecodedEventDto],
|
||||
) -> usize {
|
||||
let mut mints = std::collections::BTreeSet::<std::string::String>::new();
|
||||
for event in decoded_events {
|
||||
insert_optional_mint(&mut mints, &event.lp_mint);
|
||||
insert_optional_mint(&mut mints, &event.token_a_mint);
|
||||
insert_optional_mint(&mut mints, &event.token_b_mint);
|
||||
}
|
||||
return mints.len();
|
||||
}
|
||||
|
||||
fn insert_optional_mint(
|
||||
mints: &mut std::collections::BTreeSet<std::string::String>,
|
||||
mint_option: &std::option::Option<std::string::String>,
|
||||
) {
|
||||
if let Some(mint) = mint_option {
|
||||
let trimmed = mint.trim();
|
||||
if !trimmed.is_empty() {
|
||||
mints.insert(trimmed.to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn build_dex_decode_replay_ledger_status_reason(
|
||||
event_count: i64,
|
||||
distinct_token_mint_count: i64,
|
||||
force_replay_required: bool,
|
||||
) -> std::string::String {
|
||||
if event_count == 0 {
|
||||
return "decode completed with no persisted DEX event".to_string();
|
||||
}
|
||||
if force_replay_required {
|
||||
return format!(
|
||||
"decode completed but remains unsafe for skip: event_count={event_count}, distinct_token_mint_count={distinct_token_mint_count}"
|
||||
);
|
||||
}
|
||||
return format!(
|
||||
"decode completed and certified for skip: event_count={event_count}, distinct_token_mint_count={distinct_token_mint_count}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Replays the local pipeline from persisted raw chain transaction rows.
|
||||
|
||||
Reference in New Issue
Block a user