This commit is contained in:
2026-05-05 20:49:45 +02:00
parent f2c227e08f
commit 348e76660c
28 changed files with 3279 additions and 210 deletions

View File

@@ -55,4 +55,5 @@
0.7.22 - Ajout dune première fenêtre `Demo Pipeline` dans `kb_app` pour linspection en lecture seule du pipeline `0.7.x`, avec recherche par signature, token mint, pair id ou pool address, affichage structuré des transactions résolues, événements DEX décodés, pools, paires, listings, launch origins, pool origins, wallets et holdings observés, trade events, pair metrics, candles et signaux analytiques déjà persistés, ainsi que conservation dune instance partagée de la base SQLite pour éviter la réouverture et la réinitialisation du schéma à chaque commande UI
0.7.23 - Ajout du pilotage UI du backfill historique ciblé par `token mint` dans `kb_app`, avec saisie du rôle HTTP et des limites de signatures, affichage du résumé de backfill, réinspection automatique du token dans `Demo Pipeline` lorsque des objets persistés sont effectivement reconstruits, et gestion explicite du cas où le backfill réussit sans matérialiser de token exploitable dans la base locale
0.7.24 - Ajout de laffichage graphique des candles / OHLCV dans `kb_app` via `echarts`, avec sélection de paire et de timeframe, rendu chandelier + volume, et prise en charge des candles matérialisées ou régénérées à la demande depuis `Demo Pipeline`
0.7.25 - En cours : préparation de lenrichissement metadata des tokens, avec résolution locale limitée à SOL / WSOL, résolution des autres mints via comptes on-chain, Token-2022, Metaplex ou payloads DEX, et conservation explicite des cas non résolus
0.7.25 - Enrichissement metadata des tokens, avec résolution locale limitée à SOL / WSOL, résolution des autres mints via comptes on-chain, Token-2022, Metaplex ou payloads DEX, et conservation explicite des cas non résolus
0.7.26 - ???

View File

@@ -8,7 +8,7 @@ members = [
]
[workspace.package]
version = "0.7.25"
version = "0.7.26"
edition = "2024"
license = "MIT"
repository = "https://git.sasedev.com/Sasedev/khadhroony-bobobot"

View File

@@ -699,34 +699,150 @@ Réalisé :
Réalisé :
- Ajout :
- Relecture locale du pipeline à partir des transactions brutes persistantes de la chaîne.
- Actualisation optionnelle des métadonnées de jetons manquantes lors de la relecture locale.
- Reconstruction des symboles de paires à partir des métadonnées des jetons.
- Commandes d'interface utilisateur dans le pipeline de démonstration 2 pour la relecture locale.
- Flux de travail d'actualisation du catalogue de jetons/paires piloté par les métadonnées.
- relecture locale du pipeline à partir des transactions brutes persistantes de la chaîne,
- actualisation optionnelle des métadonnées de jetons manquantes lors de la relecture locale,
- reconstruction des symboles de paires à partir des métadonnées des jetons,
- commandes dinterface utilisateur dans le pipeline de démonstration 2 pour la relecture locale,
- flux de travail dactualisation du catalogue de jetons/paires piloté par les métadonnées.
- Modifications :
- Les symboles de paires sont désormais dérivés comme `BASE/QUOTE` lorsque les deux symboles de jetons sont disponibles.
- L'actualisation des métadonnées évite de nécessiter un remplissage complet de la blockchain lorsque les données de transaction brutes existent déjà localement.
- les symboles de paires sont désormais dérivés comme `BASE/QUOTE` lorsque les deux symboles de jetons sont disponibles,
- lactualisation des métadonnées évite de nécessiter un remplissage complet de la blockchain lorsque les données de transaction brutes existent déjà localement.
- Corrections :
- Suppression des cycles complets de suppression/remplissage répétés pour les métadonnées et les entités locales dérivées.
- Conservation de l'accès SQL dans les modules de requêtes de base de données au lieu du SQL brut au niveau du service.
- suppression des cycles complets de suppression/remplissage répétés pour les métadonnées et les entités locales dérivées,
- conservation de laccès SQL dans les modules de requêtes de base de données au lieu du SQL brut au niveau du service.
### 6.058. Version `0.7.26` — Validation multi-DEX et non-régression du pipeline
Objectif : vérifier que les connecteurs déjà branchés restent cohérents avant douvrir la phase danalyse `0.8.x`.
### 6.058. Version `0.7.26` — Diagnostics locaux, replay et extraction instruction-scoped
Réalisé :
- Ajout du diagnostic local complet du pipeline persisté :
- transactions OK / échouées,
- événements décodés,
- trade candidates,
- trade events,
- candles,
- tokens,
- pools,
- pairs,
- diagnostics par DEX,
- diagnostics par paire,
- samples dévénements manquants ou multi-trades.
- Ajout des compteurs de santé du pipeline :
- `diagnosticsClean`,
- `blockingIssueCount`,
- `actionableMissingTradeEventCount`,
- `ignoredFailedTransactionTradeCandidateCount`,
- `duplicateDecodedEventTradeCount`,
- `multiTradeSignaturePairCount`,
- `duplicateCandleBucketCount`.
- Correction de lagrégation des trades Raydium via extraction instruction-scoped des transferts SPL Token depuis `meta.innerInstructions`.
- Correction des cas CPMM contenant plusieurs swaps dans une même transaction, sans mélange des montants entre instructions.
- Conservation des transactions échouées comme événements décodés traçables, sans génération de `kb_trade_events`.
- Clarification des compteurs de replay :
- `pairCandleUpsertCount`,
- `analyticSignalUpsertCount`.
- Validation :
- aucun trade candidate issu dune transaction OK nest perdu,
- aucun trade event invalide nest persisté,
- aucun doublon réel par `decoded_event_id`,
- aucune candle dupliquée par bucket,
- aucune paire sans trade ni candle après replay,
- seuls les trade candidates issus de transactions échouées restent ignorés.
### 6.059. Version `0.7.27` — Validation multi-DEX des connecteurs déjà branchés
Objectif : verrouiller la non-régression du pipeline actuel avant dajouter de nouveaux DEX ou douvrir la phase danalyse `0.8.x`.
À faire :
- rejouer des bases neuves de test pour `pump_fun`, `pump_swap`, `raydium_cpmm` et `raydium_clmm`,
- ne pas ajouter de nouveau DEX dans cette version ; cette version sert uniquement à valider les connecteurs déjà branchés,
- vérifier pour chaque DEX le triptyque `decoded_event_count / trade_event_count / pair_candle_count`,
- vérifier que les compteurs `diagnosticsClean`, `blockingIssueCount`, `actionableMissingTradeEventCount`, `duplicateDecodedEventTradeCount` et `duplicateCandleBucketCount` restent cohérents après replay,
- garantir que les événements non pricés ou non candle ne produisent pas de trade event invalide,
- conserver lenrichissement `eventCategory`, `tradeCandidate`, `candleCandidate`, `liquidityCandidate`, `feeCandidate`, `rewardCandidate`, `adminCandidate` et `poolLifecycleCandidate` dans `payload_json`,
- documenter les familles dévénements utilisées pour les candles et celles conservées seulement pour lanalyse ou la traçabilité,
- ajouter ou compléter les tests unitaires sur `dex_decode`, `dex_detect`, `trade_aggregation`, `pair_candle_aggregation` et `pair_analytic_signal`,
- ajouter des requêtes SQL de diagnostic de référence pour contrôler rapidement les tables clés après backfill,
- conserver la tolérance aux événements DEX partiels tout en refusant les trades sans montant ou prix exploitable.
- documenter les familles dévénements utilisées pour les candles et celles conservées seulement pour lanalyse, la liquidité, les frais, les rewards, ladministration ou la traçabilité,
- ajouter ou compléter les tests unitaires sur `dex_decode`, `dex_detect`, `trade_aggregation`, `pair_candle_aggregation`, `pair_analytic_signal`, `local_pipeline_replay` et `local_pipeline_diagnostics`,
- ajouter des requêtes SQL de diagnostic de référence pour contrôler rapidement les tables clés après backfill ou replay local,
- conserver la tolérance aux événements DEX partiels tout en refusant les trades sans montant ou prix exploitable,
- valider que les transactions échouées restent traçables dans les événements décodés sans produire de `kb_trade_events`.
### 6.059. Version `0.7.27` — Raydium AMM v4 legacy : corpus et validation ciblée
Objectif : isoler correctement le vrai Raydium AMM v4 historique et le distinguer de `raydium_cpmm` et `raydium_clmm`.
### 6.060. Version `0.7.28` — Matérialisation des événements liquidité et cycle de vie pool
Objectif : exploiter les événements non buy/sell utiles à lanalyse et au trading semi-automatique sans les mélanger avec les trades/candles.
À faire :
- stabiliser ou ajouter la table `kb_liquidity_events`,
- stabiliser ou ajouter la table `kb_pool_lifecycle_events`,
- matérialiser les événements de type `increase_liquidity`, `decrease_liquidity`, `add_liquidity`, `remove_liquidity`, `open_position`, `close_position`, `initialize`, `create_pool`, `migrate` et assimilés,
- rattacher chaque événement métier à `dex_id`, `pool_id`, `pair_id`, `transaction_id`, `decoded_event_id`, `signature` et `slot`,
- conserver le `payload_json` source pour audit et extension future,
- alimenter les diagnostics locaux avec les compteurs liquidité et cycle de vie,
- garantir quun événement de liquidité ou de cycle de vie ne produit jamais de candle directement,
- préparer les signaux futurs liés à la liquidité initiale, à lajout/retrait brutal de liquidité, aux migrations et aux changements de statut de pool.
### 6.061. Version `0.7.29` — Matérialisation des événements fees, rewards et administration
Objectif : conserver les événements non price-action utiles au risque, à lanalyse économique et à la traçabilité opérationnelle des pools.
À faire :
- ajouter ou stabiliser `kb_fee_events`,
- ajouter ou stabiliser `kb_reward_events`,
- ajouter ou stabiliser `kb_pool_admin_events`,
- matérialiser les événements `collect_protocol_fee`, `collect_fund_fee`, `collect_creator_fee`, `collect_fee` et assimilés,
- matérialiser les événements `set_reward_params`, `initialize_reward`, `collect_reward`, `update_reward_infos` et assimilés,
- matérialiser les événements `set_config`, `update_config`, `set_authority`, `set_fee_rate`, `pause`, `resume` et assimilés,
- rattacher chaque événement à la transaction, au decoded event, au pool, à la paire et aux wallets observés lorsque les comptes sont disponibles,
- documenter clairement que ces événements sont utiles à lanalyse et au scoring mais ne sont ni des trades ni des candles.
### 6.062. Version `0.7.30` — Meteora : DBC / DAMM v1 / DAMM v2
Objectif : ajouter ou stabiliser les connecteurs Meteora avec une séparation claire entre swaps, liquidité, fees, rewards et événements pool.
À faire :
- vérifier les programmes et discriminants réellement utilisés pour `Meteora DBC`, `Meteora DAMM v1` et `Meteora DAMM v2`,
- constituer un petit corpus local de pools et signatures fiables pour chaque variante,
- décoder les créations de pool, swaps et événements de liquidité exploitables,
- alimenter `kb_dex_decoded_events`, les tables métier pool/pair/listing et les nouvelles tables dévénements non-trade,
- vérifier lidempotence du replay local sur un corpus mixte Meteora,
- documenter les limites connues des variantes insuffisamment couvertes par le corpus.
### 6.063. Version `0.7.31` — Launch DEX : LaunchLab / Fun Launch / Bags / Moonit
Objectif : couvrir les surfaces de lancement et de migration de tokens sans confondre origine de lancement et protocole DEX final.
À faire :
- ajouter ou stabiliser les mappings `LaunchLab`, `Fun Launch`, `Bags` et `Moonit`,
- distinguer clairement launch origin, pool origin et DEX effectif,
- détecter les créations de token, pools initiaux, migrations et listings dérivés,
- rattacher les launch origins aux pools et paires lorsque les comptes permettent un matching fiable,
- enrichir les diagnostics pour exposer les objets détectés par surface de lancement,
- éviter les heuristiques trop larges lorsquun suffixe de mint ou un label externe ne suffit pas à prouver lorigine.
### 6.064. Version `0.7.32` — Orca / FluxBeam / DexLab : corpus et validation ciblée
Objectif : ajouter les connecteurs restants à partir de corpus locaux vérifiables et garder les décodeurs heuristiques isolés tant quils ne sont pas prouvés.
À faire :
- constituer des corpus locaux pour `Orca Whirlpools`, `FluxBeam` et `DexLab`,
- vérifier les `program_id`, comptes, préfixes `data_json` et familles dinstructions utiles,
- stabiliser les événements `create_pool`, `swap` et liquidité réellement observés,
- alimenter les mêmes tables métier et diagnostics que les connecteurs déjà validés,
- marquer explicitement les variantes partiellement supportées ou heuristiques,
- rejouer les corpus plusieurs fois pour vérifier lidempotence et labsence de trades/candles invalides.
### 6.065. Version `0.7.33` — Validation DEX v1 consolidée
Objectif : rejouer tous les DEX supportés et valider les invariants du pipeline complet avant de traiter Raydium AMM v4 legacy séparément.
À faire :
- rejouer des bases neuves couvrant tous les connecteurs DEX supportés hors `raydium_amm_v4` legacy,
- vérifier les compteurs globaux et par DEX : decoded events, trade events, liquidity events, lifecycle events, fee events, reward events, admin events, candles et analytic signals,
- contrôler que chaque famille dévénements alimente uniquement les tables métier prévues,
- vérifier les diagnostics bloquants et les samples danomalie,
- documenter les corpus utilisés pour chaque DEX,
- conserver une matrice de support par DEX, variante, instruction et type dévénement.
### 6.066. Version `0.7.34` — Raydium AMM v4 legacy : corpus et validation ciblée
Objectif : traiter le vrai Raydium AMM v4 historique après les autres DEX, afin de lisoler correctement de `raydium_cpmm`, `raydium_clmm` et des labels Raydium génériques.
À faire :
@@ -738,7 +854,7 @@ Objectif : isoler correctement le vrai Raydium AMM v4 historique et le distingue
- renommer et stabiliser les fonctions internes autour de `raydium_amm_v4` afin déviter lambiguïté avec `raydium_cpmm` et `raydium_clmm`,
- documenter les limites connues si le corpus AMM v4 reste trop faible.
### 6.060. Version `0.7.28` — `kb_app` : overlays analytiques
### 6.067. Version `0.7.35` — `kb_app` : overlays analytiques
Objectif : rendre visibles les signaux analytiques directement sur les graphes et vues de marché.
À faire :
@@ -749,7 +865,7 @@ Objectif : rendre visibles les signaux analytiques directement sur les graphes e
- afficher un panneau latéral listant les signaux liés à une paire et à un timeframe,
- préparer lextension future vers des indicateurs Ichimoku, Kumo, projections ABCD et égalités temps/prix sans les mélanger au pipeline de décodage DEX.
### 6.061. Version `0.7.29` — `kb_app` : vues consolidées token / pair / pool
### 6.068. Version `0.7.36` — `kb_app` : vues consolidées token / pair / pool
Objectif : fournir une lecture métier plus confortable du modèle `0.7.x`.
À faire :
@@ -757,11 +873,11 @@ Objectif : fournir une lecture métier plus confortable du modèle `0.7.x`.
- ajouter une fiche token avec mint, programme token, metadata, pools, paires et historique de découverte,
- ajouter une fiche paire avec base/quote, DEX, pool, métriques, candles, signaux et derniers trades,
- ajouter une fiche pool avec composition, vaults, origine, première signature vue, programme DEX et statut de décodage,
- relier dans lUI les launch origins, pool origins, wallets observés, holdings observés, candles et analytic signals,
- relier dans lUI les launch origins, pool origins, wallets observés, holdings observés, événements de liquidité, événements lifecycle, fees, rewards, admin, candles et analytic signals,
- préparer une navigation transversale entre objets techniques et objets métier,
- rendre explicites les cas `tradeCount = null`, `lastPriceQuotePerBase = null` et tokens non enrichis.
- rendre explicites les cas `tradeCount = null`, `lastPriceQuotePerBase = null`, tokens non enrichis et événements conservés uniquement pour analyse.
### 6.062. Version `0.7.30` — Finition UI `0.7.x`
### 6.069. Version `0.7.37` — Finition UI `0.7.x`
Objectif : stabiliser la couche desktop de validation avant louverture de `0.8.x`.
À faire :
@@ -772,25 +888,25 @@ Objectif : stabiliser la couche desktop de validation avant louverture de `0.
- préparer une base UI suffisamment stable pour la future phase danalyse et filtrage `0.8.x`,
- vérifier que les commandes Tauri restent de simples façades vers `kb_lib` et ne récupèrent pas de logique métier.
### 6.063. Version `0.7.x` — Couverture DEX v1
### 6.070. Version `0.7.x` — Couverture DEX v1
Objectif : structurer les connecteurs DEX autour dun pipeline complet de résolution, décodage et normalisation métier.
Protocoles cibles :
- Pump.fun
- PumpSwap
- Raydium CPMM
- Raydium CLMM
- Meteora DBC
- Meteora DAMM v2
- Meteora DAMM v1
- LaunchLab / Fun Launch
- Pump.fun
- PumpSwap
- Raydium AMM v4 legacy
- Raydium CPMM
- Raydium CLMM
- Orca
- Bags
- Moonit
- Orca
- FluxBeam
- DexLab
- Moonit
- Raydium AMM v4 legacy
Résultat attendu :
@@ -799,11 +915,12 @@ Résultat attendu :
- décodage des transactions utiles,
- création dobjets métier riches pour tokens, pools, paires, listings, participants et holdings observés,
- enrichissement metadata des tokens découverts,
- séparation claire entre événements candle/trade et événements utiles seulement à lanalyse, aux frais, à la liquidité, aux rewards ou à ladministration,
- séparation claire entre événements candle/trade et événements utiles seulement à lanalyse, aux frais, à la liquidité, aux rewards, à ladministration ou au cycle de vie des pools,
- matérialisation progressive des événements non-trade dans des tables métier dédiées,
- préparation dune détection temps réel hybride et dun backfill ciblé compatible avec les mêmes objets métier,
- préparation dagrégats DEX plus riches, de candles / OHLCV et dune UI dinspection du pipeline `0.7.x`.
### 6.064. Version `0.8.x` — Analyse et filtrage
### 6.071. Version `0.8.x` — Analyse et filtrage
Objectif : transformer les événements bruts en signaux exploitables.
À faire :
@@ -818,7 +935,7 @@ Objectif : transformer les événements bruts en signaux exploitables.
- outils de sélection manuelle de points ABC et projection dun point D selon des règles temps/prix explicites,
- séparation stricte entre signaux analytiques observés, projections hypothétiques et décisions de trading.
### 6.065. Version `1.x.y` — Wallets et swap préparatoire
### 6.072. Version `1.x.y` — Wallets et swap préparatoire
Objectif : préparer la couche daction.
À faire :
@@ -829,7 +946,7 @@ Objectif : préparer la couche daction.
- préparation dordres et de swaps,
- simulation et garde-fous.
### 6.066. Version `2.x.y` — Trading semi-automatisé
### 6.073. Version `2.x.y` — Trading semi-automatisé
Objectif : brancher lanalyse à laction tout en gardant des garde-fous explicites.
À faire :
@@ -840,7 +957,7 @@ Objectif : brancher lanalyse à laction tout en gardant des garde-fous exp
- confirmations explicites ou semi-automatiques,
- journaux dexécution.
### 6.067. Version `3.x.y` — Yellowstone gRPC
### 6.074. Version `3.x.y` — Yellowstone gRPC
Objectif : ajouter le connecteur gRPC dédié.
À faire :
@@ -873,6 +990,11 @@ Modules cibles à court terme :
- `pair_candle_aggregation.rs`
- `pair_analytic_signal.rs`
- `token_metadata.rs`
- `local_pipeline_replay.rs`
- `local_pipeline_diagnostics.rs`
- `db/entities/*`
- `db/dtos/*`
- `db/queries/*`
### 7.2. `kb_app`
Responsabilités cibles :
@@ -934,12 +1056,16 @@ Le projet doit maintenir au minimum :
La priorité immédiate est désormais la suivante :
1. ajouter lenrichissement metadata des tokens afin que le catalogue affiche au minimum les symboles/noms résolus, sans hardcoder de mints hors SOL / WSOL,
2. rejouer une campagne de validation multi-DEX sur bases neuves pour `pump_fun`, `pump_swap`, `raydium_cpmm` et `raydium_clmm`,
3. constituer un corpus ciblé pour `raydium_amm_v4` legacy au lieu de sappuyer sur des labels Raydium trop génériques,
4. conserver les événements non-candle enrichis en payload pour lanalyse future, sans créer de trades invalides,
5. ajouter les overlays des signaux analytiques sur les candles,
6. consolider les vues métier `token / pair / pool` dans `kb_app`,
7. stabiliser lergonomie, les filtres et la navigation de lUI dinspection,
8. préparer ensuite louverture de `0.8.x` pour lanalyse, les filtres, les patterns et les projections graphiques,
9. préparer enfin Yellowstone gRPC comme extension de capacité, et non comme remplacement du socle HTTP / WS existant.
1. terminer la validation `0.7.27` sur les connecteurs déjà branchés : `pump_fun`, `pump_swap`, `raydium_cpmm` et `raydium_clmm`, sans ajouter de nouveau DEX dans cette étape,
2. vérifier sur bases neuves et après replay local les invariants bloquants du pipeline : `diagnosticsClean = true`, `blockingIssueCount = 0`, aucun trade candidate exploitable perdu, aucun trade event invalide, aucun doublon réel par `decoded_event_id`, aucune candle dupliquée par bucket,
3. documenter les requêtes SQL de diagnostic de référence et les résultats attendus pour les tables clés du pipeline,
4. matérialiser ensuite les événements non buy/sell utiles au trading et à lanalyse : liquidité, cycle de vie des pools, fees, rewards et administration,
5. garantir que ces événements non-trade restent séparés des `kb_trade_events` et des candles tout en restant rattachés aux transactions, decoded events, pools, pairs et wallets observés,
6. ajouter ou stabiliser les autres DEX par lots vérifiables : Meteora, surfaces de lancement, Orca, FluxBeam et DexLab,
7. traiter `raydium_amm_v4` legacy seulement après les autres DEX, avec un corpus dédié prouvant le programme `675kPX9MHTjS2zt1qfr1NYHuzeLXfQM9H24wFSUt1Mp8`,
8. effectuer une validation DEX v1 consolidée sur tous les connecteurs supportés avant de considérer la couche DEX `0.7.x` comme stable,
9. ajouter ensuite les overlays des signaux analytiques sur les candles,
10. consolider les vues métier `token / pair / pool` dans `kb_app`, y compris les événements liquidité, lifecycle, fees, rewards et admin,
11. stabiliser lergonomie, les filtres, la pagination et la navigation de lUI dinspection,
12. préparer ensuite louverture de `0.8.x` pour lanalyse, les filtres, les patterns et les projections graphiques,
13. préparer enfin Yellowstone gRPC comme extension de capacité, et non comme remplacement du socle HTTP / WS existant.

View File

@@ -151,6 +151,27 @@
</div>
</div>
<div class="accordion-item border-0 shadow-sm mb-3">
<h2 class="accordion-header" id="demoPipeline2DiagnosticsHeading">
<button class="accordion-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#demoPipeline2DiagnosticsCollapse" aria-expanded="false" aria-controls="demoPipeline2DiagnosticsCollapse">
Diagnostics locaux
</button>
</h2>
<div id="demoPipeline2DiagnosticsCollapse" class="accordion-collapse collapse" aria-labelledby="demoPipeline2DiagnosticsHeading" data-bs-parent="#demoPipeline2LeftAccordion">
<div class="accordion-body">
<p class="small text-body-secondary mb-3">
Analyse les données déjà persistées : transactions, decoded events, trades, candles, tokens, pools et pairs.
</p>
<div class="d-flex gap-2">
<button id="demoPipeline2DiagnoseLocalPipelineButton" type="button" class="btn btn-outline-primary">
Diagnose local pipeline
</button>
</div>
</div>
</div>
</div>
<div class="accordion-item border-0 shadow-sm">
<h2 class="accordion-header" id="demoPipeline2CandlesControlHeading">
<button class="accordion-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#demoPipeline2CandlesControlCollapse" aria-expanded="false" aria-controls="demoPipeline2CandlesControlCollapse">
@@ -215,6 +236,19 @@
</div>
</div>
<div class="accordion-item border-0 shadow-sm mb-3">
<h2 class="accordion-header" id="demoPipeline2LocalDiagnosticsHeading">
<button class="accordion-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#demoPipeline2LocalDiagnosticsCollapse" aria-expanded="false" aria-controls="demoPipeline2LocalDiagnosticsCollapse">
Local pipeline diagnostics
</button>
</h2>
<div id="demoPipeline2LocalDiagnosticsCollapse" class="accordion-collapse collapse" aria-labelledby="demoPipeline2LocalDiagnosticsHeading" data-bs-parent="#demoPipeline2ContentAccordion">
<div class="accordion-body">
<textarea id="demoPipeline2LocalDiagnosticsTextarea" class="form-control font-monospace" rows="18" readonly spellcheck="false"></textarea>
</div>
</div>
</div>
<div class="accordion-item border-0 shadow-sm mb-3">
<h2 class="accordion-header" id="demoPipeline2ChartHeading">
<button class="accordion-button" type="button" data-bs-toggle="collapse" data-bs-target="#demoPipeline2ChartCollapse" aria-expanded="true" aria-controls="demoPipeline2ChartCollapse">

View File

@@ -0,0 +1,34 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* Local decoded-event diagnostics summary for the UI.
*/
export type KbDemoPipeline2LocalDecodedEventDiagnosticSummary = {
/**
* Protocol name.
*/
protocolName: string,
/**
* Event kind.
*/
eventKind: string,
/**
* Event category.
*/
eventCategory: string | null,
/**
* Trade candidate flag.
*/
tradeCandidate: boolean | null,
/**
* Candle candidate flag.
*/
candleCandidate: boolean | null,
/**
* Event count.
*/
eventCount: number,
/**
* Linked trade-event count.
*/
tradeEventCount: number, };

View File

@@ -0,0 +1,38 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* Local DEX diagnostics summary for the UI.
*/
export type KbDemoPipeline2LocalDexDiagnosticSummary = {
/**
* DEX code.
*/
dexCode: string,
/**
* Pool count.
*/
poolCount: number,
/**
* Pair count.
*/
pairCount: number,
/**
* Decoded event count.
*/
decodedEventCount: number,
/**
* Decoded trade candidate count.
*/
decodedTradeCandidateCount: number,
/**
* Decoded candle candidate count.
*/
decodedCandleCandidateCount: number,
/**
* Trade event count.
*/
tradeEventCount: number,
/**
* Pair candle count.
*/
pairCandleCount: number, };

View File

@@ -0,0 +1,19 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { KbDemoPipeline2LocalPipelineDiagnosticSummary } from "./KbDemoPipeline2LocalPipelineDiagnosticSummary";
/**
* Local diagnostics payload returned to the UI.
*/
export type KbDemoPipeline2LocalDiagnosticsPayload = {
/**
* Open database URL.
*/
databaseUrl: string,
/**
* Pretty JSON diagnostics summary.
*/
summaryJson: string,
/**
* Structured diagnostics summary.
*/
summary: KbDemoPipeline2LocalPipelineDiagnosticSummary, };

View File

@@ -0,0 +1,34 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* Local duplicate decoded-event trade diagnostic sample for the UI.
*/
export type KbDemoPipeline2LocalDuplicateDecodedEventTradeDiagnosticSample = {
/**
* Decoded event id.
*/
decodedEventId: number,
/**
* Protocol name.
*/
protocolName: string | null,
/**
* Event kind.
*/
eventKind: string | null,
/**
* Pool account.
*/
poolAccount: string | null,
/**
* Trade event count.
*/
tradeEventCount: number,
/**
* Trade event ids.
*/
tradeEventIds: string | null,
/**
* Signatures.
*/
signatures: string | null, };

View File

@@ -0,0 +1,50 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* Local missing-trade-event diagnostic sample for the UI.
*/
export type KbDemoPipeline2LocalMissingTradeEventDiagnosticSample = {
/**
* Decoded event id.
*/
decodedEventId: number,
/**
* Chain transaction id.
*/
transactionId: number | null,
/**
* Transaction signature.
*/
signature: string | null,
/**
* Protocol name.
*/
protocolName: string,
/**
* Event kind.
*/
eventKind: string,
/**
* Pool account.
*/
poolAccount: string | null,
/**
* Whether the source transaction failed.
*/
transactionFailed: boolean,
/**
* Diagnostic reason explaining why no trade event was linked.
*/
reason: string,
/**
* Whether payload has an explicit base amount.
*/
hasBaseAmountPayload: boolean,
/**
* Whether payload has an explicit quote amount.
*/
hasQuoteAmountPayload: boolean,
/**
* Whether payload has an explicit price.
*/
hasPricePayload: boolean, };

View File

@@ -0,0 +1,38 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* Local multi-trade signature/pair diagnostic sample for the UI.
*/
export type KbDemoPipeline2LocalMultiTradeSignaturePairDiagnosticSample = {
/**
* Transaction signature.
*/
signature: string,
/**
* Pair id.
*/
pairId: number,
/**
* Pool address.
*/
poolAddress: string | null,
/**
* DEX code.
*/
dexCode: string | null,
/**
* Trade event count.
*/
tradeEventCount: number,
/**
* Distinct decoded event count.
*/
decodedEventCount: number,
/**
* Trade event ids.
*/
tradeEventIds: string | null,
/**
* Decoded event ids.
*/
decodedEventIds: string | null, };

View File

@@ -0,0 +1,66 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* Local pair diagnostics summary for the UI.
*/
export type KbDemoPipeline2LocalPairDiagnosticSummary = {
/**
* Pair id.
*/
pairId: number,
/**
* Pool address.
*/
poolAddress: string,
/**
* DEX code.
*/
dexCode: string,
/**
* Base mint.
*/
baseMint: string,
/**
* Base symbol.
*/
baseSymbol: string | null,
/**
* Quote mint.
*/
quoteMint: string,
/**
* Quote symbol.
*/
quoteSymbol: string | null,
/**
* Pair symbol.
*/
pairSymbol: string | null,
/**
* Decoded event count.
*/
decodedEventCount: number,
/**
* Decoded trade candidate count.
*/
decodedTradeCandidateCount: number,
/**
* Decoded candle candidate count.
*/
decodedCandleCandidateCount: number,
/**
* Trade event count.
*/
tradeEventCount: number,
/**
* Invalid trade event count.
*/
invalidTradeEventCount: number,
/**
* Pair candle count.
*/
pairCandleCount: number,
/**
* Last known price.
*/
lastPriceQuotePerBase: number | null, };

View File

@@ -0,0 +1,54 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* Local pair gap diagnostic sample for the UI.
*/
export type KbDemoPipeline2LocalPairGapDiagnosticSample = {
/**
* Pair id.
*/
pairId: number,
/**
* Pool address.
*/
poolAddress: string,
/**
* DEX code.
*/
dexCode: string,
/**
* Base mint.
*/
baseMint: string,
/**
* Base symbol.
*/
baseSymbol: string | null,
/**
* Quote mint.
*/
quoteMint: string,
/**
* Quote symbol.
*/
quoteSymbol: string | null,
/**
* Pair symbol.
*/
pairSymbol: string | null,
/**
* Decoded event count.
*/
decodedEventCount: number,
/**
* Decoded trade candidate count.
*/
decodedTradeCandidateCount: number,
/**
* Trade event count.
*/
tradeEventCount: number,
/**
* Pair candle count.
*/
pairCandleCount: number, };

View File

@@ -0,0 +1,150 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { KbDemoPipeline2LocalDecodedEventDiagnosticSummary } from "./KbDemoPipeline2LocalDecodedEventDiagnosticSummary";
import type { KbDemoPipeline2LocalDexDiagnosticSummary } from "./KbDemoPipeline2LocalDexDiagnosticSummary";
import type { KbDemoPipeline2LocalDuplicateDecodedEventTradeDiagnosticSample } from "./KbDemoPipeline2LocalDuplicateDecodedEventTradeDiagnosticSample";
import type { KbDemoPipeline2LocalMissingTradeEventDiagnosticSample } from "./KbDemoPipeline2LocalMissingTradeEventDiagnosticSample";
import type { KbDemoPipeline2LocalMultiTradeSignaturePairDiagnosticSample } from "./KbDemoPipeline2LocalMultiTradeSignaturePairDiagnosticSample";
import type { KbDemoPipeline2LocalPairDiagnosticSummary } from "./KbDemoPipeline2LocalPairDiagnosticSummary";
import type { KbDemoPipeline2LocalPairGapDiagnosticSample } from "./KbDemoPipeline2LocalPairGapDiagnosticSample";
/**
* Local pipeline diagnostics summary for the UI.
*/
export type KbDemoPipeline2LocalPipelineDiagnosticSummary = {
/**
* Total persisted chain transactions.
*/
transactionCount: number,
/**
* Total successful chain transactions.
*/
okTransactionCount: number,
/**
* Total failed chain transactions.
*/
failedTransactionCount: number,
/**
* Total decoded DEX events.
*/
decodedEventCount: number,
/**
* Total decoded DEX trade candidates.
*/
decodedTradeCandidateCount: number,
/**
* Total decoded DEX candle candidates.
*/
decodedCandleCandidateCount: number,
/**
* Whether the local persisted pipeline has no blocking diagnostic issue.
*/
diagnosticsClean: boolean,
/**
* Number of blocking diagnostic issues.
*/
blockingIssueCount: number,
/**
* Total trade candidates without trade event, including ignored failed transactions.
*/
missingTradeEventCount: number,
/**
* Explicit alias for decoded trade candidates without linked trade event.
*/
decodedTradeCandidateWithoutTradeEventCount: number,
/**
* Trade candidates without linked trade event on successful transactions.
*/
decodedTradeCandidateWithoutTradeEventOnOkTransactionCount: number,
/**
* Trade candidates without linked trade event on failed transactions.
*/
decodedTradeCandidateWithoutTradeEventOnFailedTransactionCount: number,
/**
* Trade candidates without linked trade event and without explicit base/quote payload amounts.
* Actionable missing trade events on successful transactions.
*/
actionableMissingTradeEventCount: number,
/**
* Ignored missing trade events caused by failed transactions.
*/
ignoredFailedTransactionTradeCandidateCount: number, decodedTradeCandidateWithoutAmountPayloadCount: number,
/**
* Total persisted trade events.
*/
tradeEventCount: number,
/**
* Total invalid trade events.
*/
invalidTradeEventCount: number,
/**
* Total persisted pair candles.
*/
pairCandleCount: number,
/**
* Real duplicate trade rows grouped by decoded event id.
*/
duplicateDecodedEventTradeCount: number,
/**
* Multi-trade groups sharing the same signature and pair id.
*/
multiTradeSignaturePairCount: number,
/**
* Total duplicate candle buckets.
*/
duplicateCandleBucketCount: number,
/**
* Total known tokens.
*/
tokenCount: number,
/**
* Total tokens missing symbol or name.
*/
tokenMetadataMissingCount: number,
/**
* Total known pools.
*/
poolCount: number,
/**
* Total known pairs.
*/
pairCount: number,
/**
* Total pairs without trade.
*/
pairWithoutTradeCount: number,
/**
* Total pairs without candle.
*/
pairWithoutCandleCount: number,
/**
* Diagnostics grouped by DEX.
*/
dexSummaries: Array<KbDemoPipeline2LocalDexDiagnosticSummary>,
/**
* Diagnostics grouped by pair.
*/
pairSummaries: Array<KbDemoPipeline2LocalPairDiagnosticSummary>,
/**
* Diagnostics grouped by decoded event kind.
*/
decodedEventSummaries: Array<KbDemoPipeline2LocalDecodedEventDiagnosticSummary>,
/**
* Samples of decoded trade candidates without linked trade event.
*/
missingTradeEventSamples: Array<KbDemoPipeline2LocalMissingTradeEventDiagnosticSample>,
/**
* Samples of duplicated trade rows by decoded event id.
*/
duplicateDecodedEventTradeSamples: Array<KbDemoPipeline2LocalDuplicateDecodedEventTradeDiagnosticSample>,
/**
* Samples of multi-trade signature/pair groups.
*/
multiTradeSignaturePairSamples: Array<KbDemoPipeline2LocalMultiTradeSignaturePairDiagnosticSample>,
/**
* Samples of pairs without trade.
*/
pairWithoutTradeSamples: Array<KbDemoPipeline2LocalPairGapDiagnosticSample>,
/**
* Samples of pairs without candle.
*/
pairWithoutCandleSamples: Array<KbDemoPipeline2LocalPairGapDiagnosticSample>, };

View File

@@ -13,6 +13,7 @@ import type { KbDemoPipeline2BackfillPoolRequest } from "./bindings/KbDemoPipeli
import type { KbDemoPipeline2BackfillPayload } from "./bindings/KbDemoPipeline2BackfillPayload.ts";
import type { KbDemoPipeline2PairCandlesRequest } from "./bindings/KbDemoPipeline2PairCandlesRequest.ts";
import type { KbDemoPipeline2PairCandlesPayload } from "./bindings/KbDemoPipeline2PairCandlesPayload.ts";
import type { KbDemoPipeline2LocalDiagnosticsPayload } from "./bindings/KbDemoPipeline2LocalDiagnosticsPayload.ts";
(window as Window & typeof globalThis & { bootstrap?: typeof bootstrap }).bootstrap = bootstrap;
(window as Window & typeof globalThis & { ResizeObserver?: typeof ResizeObserver }).ResizeObserver = ResizeObserver;
@@ -50,8 +51,8 @@ interface KbLocalPipelineReplayResult {
decodedEventCount: number;
detectionCount: number;
tradeEventCount: number;
pairCandleCount: number;
analyticSignalCount: number;
pairCandleUpsertCount: number;
analyticSignalUpsertCount: number;
tokenMetadataUpdatedCount: number;
pairSymbolUpdatedCount: number;
globalErrorCount: number;
@@ -349,6 +350,7 @@ document.addEventListener("DOMContentLoaded", async () => {
const replayMetadataCheckbox = document.querySelector<HTMLInputElement>("#demoPipeline2ReplayMetadataCheckbox");
const replayMetadataLimitInput = document.querySelector<HTMLInputElement>("#demoPipeline2ReplayMetadataLimitInput");
const replayLocalPipelineButton = document.querySelector<HTMLButtonElement>("#demoPipeline2ReplayLocalPipelineButton");
const diagnoseLocalPipelineButton = document.querySelector<HTMLButtonElement>("#demoPipeline2DiagnoseLocalPipelineButton");
const pairSelect = document.querySelector<HTMLSelectElement>("#demoPipeline2PairSelect");
const timeframeSelect = document.querySelector<HTMLSelectElement>("#demoPipeline2TimeframeSelect");
@@ -359,6 +361,7 @@ document.addEventListener("DOMContentLoaded", async () => {
const backfillSummaryTextarea = document.querySelector<HTMLTextAreaElement>("#demoPipeline2BackfillSummaryTextarea");
const chartElement = document.querySelector<HTMLDivElement>("#demoPipeline2Chart");
const chartMeta = document.querySelector<HTMLDivElement>("#demoPipeline2ChartMeta");
const localDiagnosticsTextarea = document.querySelector<HTMLTextAreaElement>("#demoPipeline2LocalDiagnosticsTextarea");
const clearLogButton = document.querySelector<HTMLButtonElement>("#demoPipeline2ClearLogButton");
const logTextarea = document.querySelector<HTMLTextAreaElement>("#demoPipeline2LogTextarea");
@@ -380,12 +383,14 @@ document.addEventListener("DOMContentLoaded", async () => {
!replayMetadataCheckbox ||
!replayMetadataLimitInput ||
!replayLocalPipelineButton ||
!diagnoseLocalPipelineButton ||
!pairSelect ||
!timeframeSelect ||
!customTimeframeInput ||
!preferMaterializedInput ||
!loadCandlesButton ||
!backfillSummaryTextarea ||
!localDiagnosticsTextarea ||
!chartElement ||
!chartMeta ||
!clearLogButton ||
@@ -405,6 +410,7 @@ document.addEventListener("DOMContentLoaded", async () => {
const safeChartMeta = chartMeta;
const safeLogTextarea = logTextarea;
const safeLocalDiagnosticsTextarea = localDiagnosticsTextarea;
const chart = echarts.init(safeChartElement);
setEmptyChart(chart, safeChartMeta, "Aucune candle disponible.");
@@ -581,7 +587,7 @@ document.addEventListener("DOMContentLoaded", async () => {
appendLogLine(
logTextarea,
`[ui] local pipeline replay completed: ${result.replayedTransactionCount.toString()} replayed, ${result.tradeEventCount.toString()} trades, ${result.pairCandleCount.toString()} candles`,
`[ui] local pipeline replay completed: ${result.replayedTransactionCount.toString()} replayed, ${result.tradeEventCount.toString()} trades, ${result.pairCandleUpsertCount.toString()} candle upserts`,
);
await refreshCatalog();
@@ -590,6 +596,29 @@ document.addEventListener("DOMContentLoaded", async () => {
}
});
diagnoseLocalPipelineButton.addEventListener("click", async () => {
appendLogLine(logTextarea, "[ui] diagnosing local pipeline");
diagnoseLocalPipelineButton.disabled = true;
try {
const payload = await invoke<KbDemoPipeline2LocalDiagnosticsPayload>(
"demo_pipeline2_diagnose_local_pipeline",
);
safeLocalDiagnosticsTextarea.value = payload.summaryJson;
appendLogLine(
logTextarea,
`[ui] local pipeline diagnostics completed: ${payload.summary.decodedEventCount.toString()} decoded, ${payload.summary.tradeEventCount.toString()} trades, ${payload.summary.pairCandleCount.toString()} candles`,
);
} catch (error) {
appendLogLine(logTextarea, `[ui] local pipeline diagnostics error: ${String(error)}`);
} finally {
diagnoseLocalPipelineButton.disabled = false;
}
});
loadCandlesButton.addEventListener("click", async () => {
const pairIdText = pairSelect.value.trim();
if (pairIdText === "") {

View File

@@ -1,7 +1,7 @@
{
"name": "kb-app",
"private": true,
"version": "0.7.24",
"version": "0.7.26",
"type": "module",
"scripts": {
"dev": "vite",

View File

@@ -10,13 +10,370 @@
use tauri::Manager;
use ts_rs::TS;
/// One token item for the local catalog.
/// Local diagnostics payload returned to the UI.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/KbDemoPipeline2TokenItem.ts"
export_to = "../frontend/ts/bindings/KbDemoPipeline2LocalDiagnosticsPayload.ts"
)]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2LocalDiagnosticsPayload {
/// Open database URL.
pub database_url: std::string::String,
/// Pretty JSON diagnostics summary.
pub summary_json: std::string::String,
/// Structured diagnostics summary.
pub summary: KbDemoPipeline2LocalPipelineDiagnosticSummary,
}
/// Local pipeline diagnostics summary for the UI.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/KbDemoPipeline2LocalPipelineDiagnosticSummary.ts"
)]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2LocalPipelineDiagnosticSummary {
/// Total persisted chain transactions.
#[ts(type = "number")]
pub transaction_count: i64,
/// Total successful chain transactions.
#[ts(type = "number")]
pub ok_transaction_count: i64,
/// Total failed chain transactions.
#[ts(type = "number")]
pub failed_transaction_count: i64,
/// Total decoded DEX events.
#[ts(type = "number")]
pub decoded_event_count: i64,
/// Total decoded DEX trade candidates.
#[ts(type = "number")]
pub decoded_trade_candidate_count: i64,
/// Total decoded DEX candle candidates.
#[ts(type = "number")]
pub decoded_candle_candidate_count: i64,
/// Whether the local persisted pipeline has no blocking diagnostic issue.
pub diagnostics_clean: bool,
/// Number of blocking diagnostic issues.
#[ts(type = "number")]
pub blocking_issue_count: i64,
/// Total trade candidates without trade event, including ignored failed transactions.
#[ts(type = "number")]
pub missing_trade_event_count: i64,
/// Explicit alias for decoded trade candidates without linked trade event.
#[ts(type = "number")]
pub decoded_trade_candidate_without_trade_event_count: i64,
/// Trade candidates without linked trade event on successful transactions.
#[ts(type = "number")]
pub decoded_trade_candidate_without_trade_event_on_ok_transaction_count: i64,
/// Trade candidates without linked trade event on failed transactions.
#[ts(type = "number")]
pub decoded_trade_candidate_without_trade_event_on_failed_transaction_count: i64,
/// Trade candidates without linked trade event and without explicit base/quote payload amounts.
/// Actionable missing trade events on successful transactions.
#[ts(type = "number")]
pub actionable_missing_trade_event_count: i64,
/// Ignored missing trade events caused by failed transactions.
#[ts(type = "number")]
pub ignored_failed_transaction_trade_candidate_count: i64,
#[ts(type = "number")]
pub decoded_trade_candidate_without_amount_payload_count: i64,
/// Total persisted trade events.
#[ts(type = "number")]
pub trade_event_count: i64,
/// Total invalid trade events.
#[ts(type = "number")]
pub invalid_trade_event_count: i64,
/// Total persisted pair candles.
#[ts(type = "number")]
pub pair_candle_count: i64,
/// Real duplicate trade rows grouped by decoded event id.
#[ts(type = "number")]
pub duplicate_decoded_event_trade_count: i64,
/// Multi-trade groups sharing the same signature and pair id.
#[ts(type = "number")]
pub multi_trade_signature_pair_count: i64,
/// Total duplicate candle buckets.
#[ts(type = "number")]
pub duplicate_candle_bucket_count: i64,
/// Total known tokens.
#[ts(type = "number")]
pub token_count: i64,
/// Total tokens missing symbol or name.
#[ts(type = "number")]
pub token_metadata_missing_count: i64,
/// Total known pools.
#[ts(type = "number")]
pub pool_count: i64,
/// Total known pairs.
#[ts(type = "number")]
pub pair_count: i64,
/// Total pairs without trade.
#[ts(type = "number")]
pub pair_without_trade_count: i64,
/// Total pairs without candle.
#[ts(type = "number")]
pub pair_without_candle_count: i64,
/// Diagnostics grouped by DEX.
pub dex_summaries: std::vec::Vec<KbDemoPipeline2LocalDexDiagnosticSummary>,
/// Diagnostics grouped by pair.
pub pair_summaries: std::vec::Vec<KbDemoPipeline2LocalPairDiagnosticSummary>,
/// Diagnostics grouped by decoded event kind.
pub decoded_event_summaries: std::vec::Vec<KbDemoPipeline2LocalDecodedEventDiagnosticSummary>,
/// Samples of decoded trade candidates without linked trade event.
pub missing_trade_event_samples:
std::vec::Vec<KbDemoPipeline2LocalMissingTradeEventDiagnosticSample>,
/// Samples of duplicated trade rows by decoded event id.
pub duplicate_decoded_event_trade_samples:
std::vec::Vec<KbDemoPipeline2LocalDuplicateDecodedEventTradeDiagnosticSample>,
/// Samples of multi-trade signature/pair groups.
pub multi_trade_signature_pair_samples:
std::vec::Vec<KbDemoPipeline2LocalMultiTradeSignaturePairDiagnosticSample>,
/// Samples of pairs without trade.
pub pair_without_trade_samples: std::vec::Vec<KbDemoPipeline2LocalPairGapDiagnosticSample>,
/// Samples of pairs without candle.
pub pair_without_candle_samples: std::vec::Vec<KbDemoPipeline2LocalPairGapDiagnosticSample>,
}
/// Local DEX diagnostics summary for the UI.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/KbDemoPipeline2LocalDexDiagnosticSummary.ts"
)]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2LocalDexDiagnosticSummary {
/// DEX code.
pub dex_code: std::string::String,
/// Pool count.
#[ts(type = "number")]
pub pool_count: i64,
/// Pair count.
#[ts(type = "number")]
pub pair_count: i64,
/// Decoded event count.
#[ts(type = "number")]
pub decoded_event_count: i64,
/// Decoded trade candidate count.
#[ts(type = "number")]
pub decoded_trade_candidate_count: i64,
/// Decoded candle candidate count.
#[ts(type = "number")]
pub decoded_candle_candidate_count: i64,
/// Trade event count.
#[ts(type = "number")]
pub trade_event_count: i64,
/// Pair candle count.
#[ts(type = "number")]
pub pair_candle_count: i64,
}
/// Local pair diagnostics summary for the UI.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/KbDemoPipeline2LocalPairDiagnosticSummary.ts"
)]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2LocalPairDiagnosticSummary {
/// Pair id.
#[ts(type = "number")]
pub pair_id: i64,
/// Pool address.
pub pool_address: std::string::String,
/// DEX code.
pub dex_code: std::string::String,
/// Base mint.
pub base_mint: std::string::String,
/// Base symbol.
pub base_symbol: std::option::Option<std::string::String>,
/// Quote mint.
pub quote_mint: std::string::String,
/// Quote symbol.
pub quote_symbol: std::option::Option<std::string::String>,
/// Pair symbol.
pub pair_symbol: std::option::Option<std::string::String>,
/// Decoded event count.
#[ts(type = "number")]
pub decoded_event_count: i64,
/// Decoded trade candidate count.
#[ts(type = "number")]
pub decoded_trade_candidate_count: i64,
/// Decoded candle candidate count.
#[ts(type = "number")]
pub decoded_candle_candidate_count: i64,
/// Trade event count.
#[ts(type = "number")]
pub trade_event_count: i64,
/// Invalid trade event count.
#[ts(type = "number")]
pub invalid_trade_event_count: i64,
/// Pair candle count.
#[ts(type = "number")]
pub pair_candle_count: i64,
/// Last known price.
#[ts(type = "number | null")]
pub last_price_quote_per_base: std::option::Option<f64>,
}
/// Local decoded-event diagnostics summary for the UI.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/KbDemoPipeline2LocalDecodedEventDiagnosticSummary.ts"
)]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2LocalDecodedEventDiagnosticSummary {
/// Protocol name.
pub protocol_name: std::string::String,
/// Event kind.
pub event_kind: std::string::String,
/// Event category.
pub event_category: std::option::Option<std::string::String>,
/// Trade candidate flag.
pub trade_candidate: std::option::Option<bool>,
/// Candle candidate flag.
pub candle_candidate: std::option::Option<bool>,
/// Event count.
#[ts(type = "number")]
pub event_count: i64,
/// Linked trade-event count.
#[ts(type = "number")]
pub trade_event_count: i64,
}
/// Local missing-trade-event diagnostic sample for the UI.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/KbDemoPipeline2LocalMissingTradeEventDiagnosticSample.ts"
)]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2LocalMissingTradeEventDiagnosticSample {
/// Decoded event id.
#[ts(type = "number")]
pub decoded_event_id: i64,
/// Chain transaction id.
#[ts(type = "number | null")]
pub transaction_id: std::option::Option<i64>,
/// Transaction signature.
pub signature: std::option::Option<std::string::String>,
/// Protocol name.
pub protocol_name: std::string::String,
/// Event kind.
pub event_kind: std::string::String,
/// Pool account.
pub pool_account: std::option::Option<std::string::String>,
/// Whether the source transaction failed.
pub transaction_failed: bool,
/// Diagnostic reason explaining why no trade event was linked.
pub reason: std::string::String,
/// Whether payload has an explicit base amount.
pub has_base_amount_payload: bool,
/// Whether payload has an explicit quote amount.
pub has_quote_amount_payload: bool,
/// Whether payload has an explicit price.
pub has_price_payload: bool,
}
/// Local duplicate decoded-event trade diagnostic sample for the UI.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/KbDemoPipeline2LocalDuplicateDecodedEventTradeDiagnosticSample.ts"
)]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2LocalDuplicateDecodedEventTradeDiagnosticSample {
/// Decoded event id.
#[ts(type = "number")]
pub decoded_event_id: i64,
/// Protocol name.
pub protocol_name: std::option::Option<std::string::String>,
/// Event kind.
pub event_kind: std::option::Option<std::string::String>,
/// Pool account.
pub pool_account: std::option::Option<std::string::String>,
/// Trade event count.
#[ts(type = "number")]
pub trade_event_count: i64,
/// Trade event ids.
pub trade_event_ids: std::option::Option<std::string::String>,
/// Signatures.
pub signatures: std::option::Option<std::string::String>,
}
/// Local multi-trade signature/pair diagnostic sample for the UI.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/KbDemoPipeline2LocalMultiTradeSignaturePairDiagnosticSample.ts"
)]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2LocalMultiTradeSignaturePairDiagnosticSample {
/// Transaction signature.
pub signature: std::string::String,
/// Pair id.
#[ts(type = "number")]
pub pair_id: i64,
/// Pool address.
pub pool_address: std::option::Option<std::string::String>,
/// DEX code.
pub dex_code: std::option::Option<std::string::String>,
/// Trade event count.
#[ts(type = "number")]
pub trade_event_count: i64,
/// Distinct decoded event count.
#[ts(type = "number")]
pub decoded_event_count: i64,
/// Trade event ids.
pub trade_event_ids: std::option::Option<std::string::String>,
/// Decoded event ids.
pub decoded_event_ids: std::option::Option<std::string::String>,
}
/// Local pair gap diagnostic sample for the UI.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/KbDemoPipeline2LocalPairGapDiagnosticSample.ts"
)]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2LocalPairGapDiagnosticSample {
/// Pair id.
#[ts(type = "number")]
pub pair_id: i64,
/// Pool address.
pub pool_address: std::string::String,
/// DEX code.
pub dex_code: std::string::String,
/// Base mint.
pub base_mint: std::string::String,
/// Base symbol.
pub base_symbol: std::option::Option<std::string::String>,
/// Quote mint.
pub quote_mint: std::string::String,
/// Quote symbol.
pub quote_symbol: std::option::Option<std::string::String>,
/// Pair symbol.
pub pair_symbol: std::option::Option<std::string::String>,
/// Decoded event count.
#[ts(type = "number")]
pub decoded_event_count: i64,
/// Decoded trade candidate count.
#[ts(type = "number")]
pub decoded_trade_candidate_count: i64,
/// Trade event count.
#[ts(type = "number")]
pub trade_event_count: i64,
/// Pair candle count.
#[ts(type = "number")]
pub pair_candle_count: i64,
}
/// One token item for the local catalog.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(export, export_to = "../frontend/ts/bindings/KbDemoPipeline2TokenItem.ts")]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2TokenItem {
/// Token mint.
pub mint: std::string::String,
@@ -28,10 +385,7 @@ pub(crate) struct KbDemoPipeline2TokenItem {
/// One pool item for the local catalog.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/KbDemoPipeline2PoolItem.ts"
)]
#[ts(export, export_to = "../frontend/ts/bindings/KbDemoPipeline2PoolItem.ts")]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2PoolItem {
/// Pool address.
@@ -45,10 +399,7 @@ pub(crate) struct KbDemoPipeline2PoolItem {
/// One pair item for the local catalog.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/KbDemoPipeline2PairItem.ts"
)]
#[ts(export, export_to = "../frontend/ts/bindings/KbDemoPipeline2PairItem.ts")]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2PairItem {
/// Internal pair id.
@@ -70,10 +421,7 @@ pub(crate) struct KbDemoPipeline2PairItem {
/// Full local catalog payload.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/KbDemoPipeline2CatalogPayload.ts"
)]
#[ts(export, export_to = "../frontend/ts/bindings/KbDemoPipeline2CatalogPayload.ts")]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2CatalogPayload {
/// Open database URL.
@@ -122,10 +470,7 @@ pub(crate) struct KbDemoPipeline2BackfillPoolRequest {
/// Shared backfill response payload.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/KbDemoPipeline2BackfillPayload.ts"
)]
#[ts(export, export_to = "../frontend/ts/bindings/KbDemoPipeline2BackfillPayload.ts")]
#[serde(rename_all = "camelCase")]
pub(crate) struct KbDemoPipeline2BackfillPayload {
/// Object key used by the backfill.
@@ -176,6 +521,35 @@ pub(crate) struct KbDemoPipeline2PairCandlesPayload {
pub candles_json: std::string::String,
}
/// Runs local pipeline diagnostics from persisted data only.
#[tauri::command]
pub(crate) async fn demo_pipeline2_diagnose_local_pipeline(
state: tauri::State<'_, crate::KbAppState>,
) -> Result<KbDemoPipeline2LocalDiagnosticsPayload, std::string::String> {
let database = state.database.clone();
let service = kb_lib::KbLocalPipelineDiagnosticsService::new(database.clone());
let summary_result = service.diagnose().await;
let summary = match summary_result {
Ok(summary) => summary,
Err(error) => {
return Err(format!("local pipeline diagnostics failed: {}", error));
},
};
let ui_summary = kb_demo_pipeline2_map_local_diagnostics_summary(summary);
let summary_json_result = serde_json::to_string_pretty(&ui_summary);
let summary_json = match summary_json_result {
Ok(summary_json) => summary_json,
Err(error) => {
return Err(format!("cannot serialize local pipeline diagnostics: {}", error));
},
};
Ok(KbDemoPipeline2LocalDiagnosticsPayload {
database_url: database.database_url().to_string(),
summary_json,
summary: ui_summary,
})
}
/// Opens the `Demo Pipeline 2` window.
#[tauri::command]
pub(crate) fn open_demo_pipeline2_window(
@@ -203,9 +577,9 @@ pub(crate) fn open_demo_pipeline2_window(
Ok(window) => window,
Err(error) => {
return Err(format!("cannot create demo_pipeline2 window: {error:?}"));
}
},
}
}
},
};
let show_result = demo_window.show();
if let Err(error) = show_result {
@@ -261,7 +635,7 @@ pub(crate) async fn demo_pipeline2_backfill_token_mint(
"cannot backfill token mint '{}' with role '{}': {}",
token_mint, http_role, error
));
}
},
};
let summary_json_result = serde_json::to_string_pretty(&backfill);
let summary_json = match summary_json_result {
@@ -271,7 +645,7 @@ pub(crate) async fn demo_pipeline2_backfill_token_mint(
"cannot serialize token backfill result for '{}': {}",
token_mint, error
));
}
},
};
let catalog = kb_demo_pipeline2_build_catalog(database).await?;
Ok(KbDemoPipeline2BackfillPayload {
@@ -311,7 +685,7 @@ pub(crate) async fn demo_pipeline2_backfill_pool_address(
"cannot backfill pool address '{}' with role '{}': {}",
pool_address, http_role, error
));
}
},
};
let summary_json_result = serde_json::to_string_pretty(&backfill);
let summary_json = match summary_json_result {
@@ -321,7 +695,7 @@ pub(crate) async fn demo_pipeline2_backfill_pool_address(
"cannot serialize pool backfill result for '{}': {}",
pool_address, error
));
}
},
};
let catalog = kb_demo_pipeline2_build_catalog(database).await?;
Ok(KbDemoPipeline2BackfillPayload {
@@ -362,7 +736,7 @@ pub(crate) async fn demo_pipeline2_get_pair_candles(
"cannot load candles for pair '{}' timeframe '{}': {}",
request.pair_id, request.timeframe_seconds, error
));
}
},
};
let candles_json_result = serde_json::to_string_pretty(&candles);
let candles_json = match candles_json_result {
@@ -372,7 +746,7 @@ pub(crate) async fn demo_pipeline2_get_pair_candles(
"cannot serialize candles for pair '{}' timeframe '{}': {}",
request.pair_id, request.timeframe_seconds, error
));
}
},
};
Ok(KbDemoPipeline2PairCandlesPayload {
pair_id: request.pair_id,
@@ -389,7 +763,7 @@ async fn kb_demo_pipeline2_build_catalog(
Ok(dexes) => dexes,
Err(error) => {
return Err(format!("cannot list DEXes: {}", error));
}
},
};
let mut dex_code_by_id = std::collections::BTreeMap::<i64, std::string::String>::new();
for dex in dexes {
@@ -402,9 +776,8 @@ async fn kb_demo_pipeline2_build_catalog(
Ok(db_tokens) => db_tokens,
Err(error) => {
return Err(format!("cannot list tokens: {}", error));
}
},
};
let mut tokens = std::vec::Vec::<KbDemoPipeline2TokenItem>::new();
for token in db_tokens {
tokens.push(KbDemoPipeline2TokenItem {
@@ -418,14 +791,14 @@ async fn kb_demo_pipeline2_build_catalog(
Ok(pools) => pools,
Err(error) => {
return Err(format!("cannot list pools: {}", error));
}
},
};
let pairs_result = kb_lib::list_pairs(database.as_ref()).await;
let pairs = match pairs_result {
Ok(pairs) => pairs,
Err(error) => {
return Err(format!("cannot list pairs: {}", error));
}
},
};
let mut pair_by_pool_id = std::collections::BTreeMap::<i64, kb_lib::KbPairDto>::new();
for pair in &pairs {
@@ -445,7 +818,7 @@ async fn kb_demo_pipeline2_build_catalog(
Ok(all_pools) => all_pools,
Err(error) => {
return Err(format!("cannot reload pools for pair catalog: {}", error));
}
},
};
let mut found_address = std::string::String::new();
for pool in all_pools {
@@ -465,11 +838,8 @@ async fn kb_demo_pipeline2_build_catalog(
let pair_metric_option = match pair_metric_result {
Ok(pair_metric_option) => pair_metric_option,
Err(error) => {
return Err(format!(
"cannot fetch pair metric for pair '{}': {}",
pair_id, error
));
}
return Err(format!("cannot fetch pair metric for pair '{}': {}", pair_id, error));
},
};
let trade_count = pair_metric_option.as_ref().map(|metric| metric.trade_count);
let last_price_quote_per_base =
@@ -537,18 +907,215 @@ pub(crate) async fn demo_pipeline2_replay_local_pipeline(
}
}
fn kb_demo_pipeline2_map_local_diagnostics_summary(
summary: kb_lib::KbLocalPipelineDiagnosticSummaryDto,
) -> KbDemoPipeline2LocalPipelineDiagnosticSummary {
let mut dex_summaries = std::vec::Vec::new();
for dex_summary in summary.dex_summaries {
dex_summaries.push(kb_demo_pipeline2_map_local_dex_diagnostic_summary(dex_summary));
}
let mut pair_summaries = std::vec::Vec::new();
for pair_summary in summary.pair_summaries {
pair_summaries.push(kb_demo_pipeline2_map_local_pair_diagnostic_summary(pair_summary));
}
let mut decoded_event_summaries = std::vec::Vec::new();
for decoded_event_summary in summary.decoded_event_summaries {
decoded_event_summaries.push(kb_demo_pipeline2_map_local_decoded_event_diagnostic_summary(
decoded_event_summary,
));
}
let mut missing_trade_event_samples = std::vec::Vec::new();
for sample in summary.missing_trade_event_samples {
missing_trade_event_samples.push(kb_demo_pipeline2_map_missing_trade_event_sample(sample));
}
let mut duplicate_decoded_event_trade_samples = std::vec::Vec::new();
for sample in summary.duplicate_decoded_event_trade_samples {
duplicate_decoded_event_trade_samples
.push(kb_demo_pipeline2_map_duplicate_decoded_event_trade_sample(sample));
}
let mut multi_trade_signature_pair_samples = std::vec::Vec::new();
for sample in summary.multi_trade_signature_pair_samples {
multi_trade_signature_pair_samples
.push(kb_demo_pipeline2_map_multi_trade_signature_pair_sample(sample));
}
let mut pair_without_trade_samples = std::vec::Vec::new();
for sample in summary.pair_without_trade_samples {
pair_without_trade_samples.push(kb_demo_pipeline2_map_pair_gap_sample(sample));
}
let mut pair_without_candle_samples = std::vec::Vec::new();
for sample in summary.pair_without_candle_samples {
pair_without_candle_samples.push(kb_demo_pipeline2_map_pair_gap_sample(sample));
}
KbDemoPipeline2LocalPipelineDiagnosticSummary {
transaction_count: summary.transaction_count,
ok_transaction_count: summary.ok_transaction_count,
failed_transaction_count: summary.failed_transaction_count,
decoded_event_count: summary.decoded_event_count,
decoded_trade_candidate_count: summary.decoded_trade_candidate_count,
decoded_candle_candidate_count: summary.decoded_candle_candidate_count,
diagnostics_clean: summary.diagnostics_clean,
blocking_issue_count: summary.blocking_issue_count,
missing_trade_event_count: summary.missing_trade_event_count,
decoded_trade_candidate_without_trade_event_count: summary
.decoded_trade_candidate_without_trade_event_count,
decoded_trade_candidate_without_trade_event_on_ok_transaction_count: summary
.decoded_trade_candidate_without_trade_event_on_ok_transaction_count,
decoded_trade_candidate_without_trade_event_on_failed_transaction_count: summary
.decoded_trade_candidate_without_trade_event_on_failed_transaction_count,
actionable_missing_trade_event_count: summary.actionable_missing_trade_event_count,
ignored_failed_transaction_trade_candidate_count: summary
.ignored_failed_transaction_trade_candidate_count,
decoded_trade_candidate_without_amount_payload_count: summary
.decoded_trade_candidate_without_amount_payload_count,
trade_event_count: summary.trade_event_count,
invalid_trade_event_count: summary.invalid_trade_event_count,
pair_candle_count: summary.pair_candle_count,
duplicate_decoded_event_trade_count: summary.duplicate_decoded_event_trade_count,
multi_trade_signature_pair_count: summary.multi_trade_signature_pair_count,
duplicate_candle_bucket_count: summary.duplicate_candle_bucket_count,
token_count: summary.token_count,
token_metadata_missing_count: summary.token_metadata_missing_count,
pool_count: summary.pool_count,
pair_count: summary.pair_count,
pair_without_trade_count: summary.pair_without_trade_count,
pair_without_candle_count: summary.pair_without_candle_count,
dex_summaries,
pair_summaries,
decoded_event_summaries,
missing_trade_event_samples,
duplicate_decoded_event_trade_samples,
multi_trade_signature_pair_samples,
pair_without_trade_samples,
pair_without_candle_samples,
}
}
fn kb_demo_pipeline2_map_local_dex_diagnostic_summary(
summary: kb_lib::KbLocalDexDiagnosticSummaryDto,
) -> KbDemoPipeline2LocalDexDiagnosticSummary {
KbDemoPipeline2LocalDexDiagnosticSummary {
dex_code: summary.dex_code,
pool_count: summary.pool_count,
pair_count: summary.pair_count,
decoded_event_count: summary.decoded_event_count,
decoded_trade_candidate_count: summary.decoded_trade_candidate_count,
decoded_candle_candidate_count: summary.decoded_candle_candidate_count,
trade_event_count: summary.trade_event_count,
pair_candle_count: summary.pair_candle_count,
}
}
fn kb_demo_pipeline2_map_local_pair_diagnostic_summary(
summary: kb_lib::KbLocalPairDiagnosticSummaryDto,
) -> KbDemoPipeline2LocalPairDiagnosticSummary {
KbDemoPipeline2LocalPairDiagnosticSummary {
pair_id: summary.pair_id,
pool_address: summary.pool_address,
dex_code: summary.dex_code,
base_mint: summary.base_mint,
base_symbol: summary.base_symbol,
quote_mint: summary.quote_mint,
quote_symbol: summary.quote_symbol,
pair_symbol: summary.pair_symbol,
decoded_event_count: summary.decoded_event_count,
decoded_trade_candidate_count: summary.decoded_trade_candidate_count,
decoded_candle_candidate_count: summary.decoded_candle_candidate_count,
trade_event_count: summary.trade_event_count,
invalid_trade_event_count: summary.invalid_trade_event_count,
pair_candle_count: summary.pair_candle_count,
last_price_quote_per_base: summary.last_price_quote_per_base,
}
}
fn kb_demo_pipeline2_map_local_decoded_event_diagnostic_summary(
summary: kb_lib::KbLocalDecodedEventDiagnosticSummaryDto,
) -> KbDemoPipeline2LocalDecodedEventDiagnosticSummary {
KbDemoPipeline2LocalDecodedEventDiagnosticSummary {
protocol_name: summary.protocol_name,
event_kind: summary.event_kind,
event_category: summary.event_category,
trade_candidate: summary.trade_candidate,
candle_candidate: summary.candle_candidate,
event_count: summary.event_count,
trade_event_count: summary.trade_event_count,
}
}
fn kb_demo_pipeline2_map_missing_trade_event_sample(
sample: kb_lib::KbLocalMissingTradeEventDiagnosticSampleDto,
) -> KbDemoPipeline2LocalMissingTradeEventDiagnosticSample {
return KbDemoPipeline2LocalMissingTradeEventDiagnosticSample {
decoded_event_id: sample.decoded_event_id,
transaction_id: sample.transaction_id,
signature: sample.signature,
protocol_name: sample.protocol_name,
event_kind: sample.event_kind,
pool_account: sample.pool_account,
transaction_failed: sample.transaction_failed,
reason: sample.reason,
has_base_amount_payload: sample.has_base_amount_payload,
has_quote_amount_payload: sample.has_quote_amount_payload,
has_price_payload: sample.has_price_payload,
};
}
fn kb_demo_pipeline2_map_duplicate_decoded_event_trade_sample(
sample: kb_lib::KbLocalDuplicateDecodedEventTradeDiagnosticSampleDto,
) -> KbDemoPipeline2LocalDuplicateDecodedEventTradeDiagnosticSample {
return KbDemoPipeline2LocalDuplicateDecodedEventTradeDiagnosticSample {
decoded_event_id: sample.decoded_event_id,
protocol_name: sample.protocol_name,
event_kind: sample.event_kind,
pool_account: sample.pool_account,
trade_event_count: sample.trade_event_count,
trade_event_ids: sample.trade_event_ids,
signatures: sample.signatures,
};
}
fn kb_demo_pipeline2_map_multi_trade_signature_pair_sample(
sample: kb_lib::KbLocalMultiTradeSignaturePairDiagnosticSampleDto,
) -> KbDemoPipeline2LocalMultiTradeSignaturePairDiagnosticSample {
return KbDemoPipeline2LocalMultiTradeSignaturePairDiagnosticSample {
signature: sample.signature,
pair_id: sample.pair_id,
pool_address: sample.pool_address,
dex_code: sample.dex_code,
trade_event_count: sample.trade_event_count,
decoded_event_count: sample.decoded_event_count,
trade_event_ids: sample.trade_event_ids,
decoded_event_ids: sample.decoded_event_ids,
};
}
fn kb_demo_pipeline2_map_pair_gap_sample(
sample: kb_lib::KbLocalPairGapDiagnosticSampleDto,
) -> KbDemoPipeline2LocalPairGapDiagnosticSample {
return KbDemoPipeline2LocalPairGapDiagnosticSample {
pair_id: sample.pair_id,
pool_address: sample.pool_address,
dex_code: sample.dex_code,
base_mint: sample.base_mint,
base_symbol: sample.base_symbol,
quote_mint: sample.quote_mint,
quote_symbol: sample.quote_symbol,
pair_symbol: sample.pair_symbol,
decoded_event_count: sample.decoded_event_count,
decoded_trade_candidate_count: sample.decoded_trade_candidate_count,
trade_event_count: sample.trade_event_count,
pair_candle_count: sample.pair_candle_count,
};
}
fn kb_demo_pipeline2_normalize_http_role(
role: std::option::Option<std::string::String>,
) -> std::string::String {
match role {
Some(role) => {
let trimmed = role.trim().to_string();
if trimmed.is_empty() {
"history_backfill".to_string()
} else {
trimmed
}
}
if trimmed.is_empty() { "history_backfill".to_string() } else { trimmed }
},
None => "history_backfill".to_string(),
}
}

View File

@@ -61,7 +61,7 @@ pub async fn run() -> Result<(), kb_lib::KbError> {
error
);
return Err(error);
}
},
};
let prepare_result = config.prepare_filesystem();
if let Err(error) = prepare_result {
@@ -74,7 +74,7 @@ pub async fn run() -> Result<(), kb_lib::KbError> {
Err(error) => {
eprintln!("kb_app tracing initialization error: {error}");
return Err(error);
}
},
};
tracing::info!(
app_name = %config.app.name,
@@ -92,7 +92,7 @@ pub async fn run() -> Result<(), kb_lib::KbError> {
Err(error) => {
tracing::error!("cannot create http endpoint pool: {}", error);
panic!("cannot create http endpoint pool: {}", error);
}
},
};
let ws_manager_result = kb_lib::WsManager::from_config(&config);
let ws_manager = match ws_manager_result {
@@ -100,7 +100,7 @@ pub async fn run() -> Result<(), kb_lib::KbError> {
Err(error) => {
tracing::error!("cannot create websocket manager: {}", error);
panic!("cannot create websocket manager: {}", error);
}
},
};
let app_state = KbAppState {
config: config.clone(),
@@ -151,6 +151,7 @@ pub async fn run() -> Result<(), kb_lib::KbError> {
crate::demo_pipeline2::demo_pipeline2_backfill_pool_address,
crate::demo_pipeline2::demo_pipeline2_get_pair_candles,
crate::demo_pipeline2::demo_pipeline2_replay_local_pipeline,
crate::demo_pipeline2::demo_pipeline2_diagnose_local_pipeline,
]);
tauri_builder = tauri_builder.plugin(tracing_builder.build::<tauri::Wry>());
tauri_builder = tauri_builder.setup(|app| {
@@ -162,7 +163,7 @@ pub async fn run() -> Result<(), kb_lib::KbError> {
None => {
tracing::error!("splash window not found");
return;
}
},
};
let main_window_option = app_handle.get_webview_window("main");
let main_window = match main_window_option {
@@ -170,7 +171,7 @@ pub async fn run() -> Result<(), kb_lib::KbError> {
None => {
tracing::error!("main window not found");
return;
}
},
};
let is_debug = cfg!(debug_assertions);
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
@@ -178,12 +179,7 @@ pub async fn run() -> Result<(), kb_lib::KbError> {
emit_splash_order(&splash_window, "add_log", Some("Start Fade-In"), None);
}
emit_splash_order(&splash_window, "fadein", None, None);
emit_splash_order(
&splash_window,
"add_msg",
Some("Initialisation..."),
Some("info"),
);
emit_splash_order(&splash_window, "add_msg", Some("Initialisation..."), Some("info"));
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
emit_splash_order(
&splash_window,
@@ -273,20 +269,14 @@ async fn start_ws_clients(
}
kb_emit_app_log(
&app_handle,
&format!(
"[app] starting {} websocket client(s)",
enabled_endpoints.len()
),
&format!("[app] starting {} websocket client(s)", enabled_endpoints.len()),
);
let mut started_clients: std::vec::Vec<kb_lib::WsClient> = std::vec::Vec::new();
let mut relay_tasks: std::vec::Vec<tauri::async_runtime::JoinHandle<()>> = std::vec::Vec::new();
for endpoint in enabled_endpoints {
kb_emit_app_log(
&app_handle,
&format!(
"[app] preparing websocket endpoint '{}' ({})",
endpoint.name, endpoint.url
),
&format!("[app] preparing websocket endpoint '{}' ({})", endpoint.name, endpoint.url),
);
let client_result = kb_lib::WsClient::new(endpoint.clone());
let client = match client_result {
@@ -297,7 +287,7 @@ async fn start_ws_clients(
"cannot create websocket client for endpoint '{}': {}",
endpoint.name, error
));
}
},
};
let mut event_receiver = client.subscribe_events();
let relay_app_handle = app_handle.clone();
@@ -308,7 +298,7 @@ async fn start_ws_clients(
Ok(event) => {
let line = kb_format_ws_event(&event);
kb_emit_app_log(&relay_app_handle, &line);
}
},
Err(tokio::sync::broadcast::error::RecvError::Lagged(skipped)) => {
kb_emit_app_log(
&relay_app_handle,
@@ -317,10 +307,10 @@ async fn start_ws_clients(
skipped
),
);
}
},
Err(tokio::sync::broadcast::error::RecvError::Closed) => {
break;
}
},
}
}
});
@@ -349,10 +339,7 @@ async fn start_ws_clients(
let runtime_guard = state.ws_runtime.lock().await;
runtime_guard.clients.len()
};
kb_emit_app_log(
&app_handle,
&format!("[app] {} websocket client(s) started", started_count),
);
kb_emit_app_log(&app_handle, &format!("[app] {} websocket client(s) started", started_count));
Ok(started_count)
}
@@ -372,10 +359,7 @@ async fn stop_ws_clients(
kb_emit_app_log(&app_handle, "[app] websocket clients are already stopped");
return Ok(0);
}
kb_emit_app_log(
&app_handle,
&format!("[app] stopping {} websocket client(s)", clients.len()),
);
kb_emit_app_log(&app_handle, &format!("[app] stopping {} websocket client(s)", clients.len()));
let stopped_count = clients.len();
for client in &clients {
let disconnect_result = client.disconnect().await;
@@ -393,10 +377,7 @@ async fn stop_ws_clients(
for relay_task in relay_tasks.drain(..) {
relay_task.abort();
}
kb_emit_app_log(
&app_handle,
&format!("[app] {} websocket client(s) stopped", stopped_count),
);
kb_emit_app_log(&app_handle, &format!("[app] {} websocket client(s) stopped", stopped_count));
Ok(stopped_count)
}
@@ -409,34 +390,25 @@ fn kb_emit_app_log(app_handle: &tauri::AppHandle, message: &str) {
fn kb_format_ws_event(event: &kb_lib::WsEvent) -> std::string::String {
match event {
kb_lib::WsEvent::Connected {
endpoint_name,
endpoint_url,
} => {
kb_lib::WsEvent::Connected { endpoint_name, endpoint_url } => {
format!("[ws:{endpoint_name}] connected to {endpoint_url}")
}
kb_lib::WsEvent::TextMessage {
endpoint_name,
text,
} => {
},
kb_lib::WsEvent::TextMessage { endpoint_name, text } => {
format!("[ws:{endpoint_name}] text: {text}")
}
kb_lib::WsEvent::JsonRpcMessage {
endpoint_name,
message,
} => match message {
},
kb_lib::WsEvent::JsonRpcMessage { endpoint_name, message } => match message {
kb_lib::KbJsonRpcWsIncomingMessage::SuccessResponse(response) => {
format!(
"[ws:{endpoint_name}] json-rpc success id={} result={}",
response.id, response.result
)
}
},
kb_lib::KbJsonRpcWsIncomingMessage::ErrorResponse(response) => {
format!(
"[ws:{endpoint_name}] json-rpc error id={} code={} message={}",
response.id, response.error.code, response.error.message
)
}
},
kb_lib::KbJsonRpcWsIncomingMessage::Notification(notification) => {
format!(
"[ws:{endpoint_name}] json-rpc notification method={} subscription={} result={}",
@@ -444,22 +416,12 @@ fn kb_format_ws_event(event: &kb_lib::WsEvent) -> std::string::String {
notification.params.subscription,
notification.params.result
)
}
},
},
kb_lib::WsEvent::JsonRpcParseError {
endpoint_name,
text,
error,
} => {
format!(
"[ws:{endpoint_name}] json-rpc parse error: {} | raw={}",
error, text
)
}
kb_lib::WsEvent::SubscriptionRegistered {
endpoint_name,
subscription,
} => {
kb_lib::WsEvent::JsonRpcParseError { endpoint_name, text, error } => {
format!("[ws:{endpoint_name}] json-rpc parse error: {} | raw={}", error, text)
},
kb_lib::WsEvent::SubscriptionRegistered { endpoint_name, subscription } => {
format!(
"[ws:{endpoint_name}] subscription registered subscribe_method={} unsubscribe_method={} notification_method={} request_id={} subscription_id={}",
subscription.subscribe_method,
@@ -468,7 +430,7 @@ fn kb_format_ws_event(event: &kb_lib::WsEvent) -> std::string::String {
subscription.request_id,
subscription.subscription_id
)
}
},
kb_lib::WsEvent::SubscriptionNotification {
endpoint_name,
subscription,
@@ -483,16 +445,13 @@ fn kb_format_ws_event(event: &kb_lib::WsEvent) -> std::string::String {
subscription.subscription_id,
notification.params.result
)
}
kb_lib::WsEvent::JsonRpcNotificationWithoutSubscription {
endpoint_name,
notification,
} => {
},
kb_lib::WsEvent::JsonRpcNotificationWithoutSubscription { endpoint_name, notification } => {
format!(
"[ws:{endpoint_name}] untracked notification method={} subscription={} result={}",
notification.method, notification.params.subscription, notification.params.result
)
}
},
kb_lib::WsEvent::SubscriptionUnregistered {
endpoint_name,
subscription_id,
@@ -503,44 +462,25 @@ fn kb_format_ws_event(event: &kb_lib::WsEvent) -> std::string::String {
"[ws:{endpoint_name}] subscription unregistered subscription_id={} unsubscribe_method={} was_active={}",
subscription_id, unsubscribe_method, was_active
)
}
kb_lib::WsEvent::BinaryMessage {
endpoint_name,
data,
} => {
},
kb_lib::WsEvent::BinaryMessage { endpoint_name, data } => {
format!("[ws:{endpoint_name}] binary message ({} bytes)", data.len())
}
kb_lib::WsEvent::Ping {
endpoint_name,
data,
} => {
},
kb_lib::WsEvent::Ping { endpoint_name, data } => {
format!("[ws:{endpoint_name}] ping ({} bytes)", data.len())
}
kb_lib::WsEvent::Pong {
endpoint_name,
data,
} => {
},
kb_lib::WsEvent::Pong { endpoint_name, data } => {
format!("[ws:{endpoint_name}] pong ({} bytes)", data.len())
}
kb_lib::WsEvent::CloseReceived {
endpoint_name,
code,
reason,
} => {
format!(
"[ws:{endpoint_name}] close received code={:?} reason={:?}",
code, reason
)
}
},
kb_lib::WsEvent::CloseReceived { endpoint_name, code, reason } => {
format!("[ws:{endpoint_name}] close received code={:?} reason={:?}", code, reason)
},
kb_lib::WsEvent::Disconnected { endpoint_name } => {
format!("[ws:{endpoint_name}] disconnected")
}
kb_lib::WsEvent::Error {
endpoint_name,
error,
} => {
},
kb_lib::WsEvent::Error { endpoint_name, error } => {
format!("[ws:{endpoint_name}] error: {error}")
}
},
}
}

View File

@@ -1,7 +1,7 @@
{
"$schema": "https://schema.tauri.app/config/2",
"productName": "kb-bapp",
"version": "0.7.24",
"version": "0.7.26",
"identifier": "com.sasedev.kb-app",
"build": {
"beforeDevCommand": "npm run dev",

View File

@@ -29,6 +29,23 @@ pub use dtos::KbLaunchAttributionDto;
pub use dtos::KbLaunchSurfaceDto;
pub use dtos::KbLaunchSurfaceKeyDto;
pub use dtos::KbLiquidityEventDto;
pub use dtos::KbLocalDecodedEventDiagnosticSummaryDto;
pub(crate) use dtos::KbLocalDecodedEventDiagnosticSummaryRow;
pub use dtos::KbLocalDexDiagnosticSummaryDto;
pub(crate) use dtos::KbLocalDexDiagnosticSummaryRow;
pub use dtos::KbLocalDuplicateDecodedEventTradeDiagnosticSampleDto;
pub(crate) use dtos::KbLocalDuplicateDecodedEventTradeDiagnosticSampleRow;
pub use dtos::KbLocalMissingTradeEventDiagnosticSampleDto;
pub(crate) use dtos::KbLocalMissingTradeEventDiagnosticSampleRow;
pub use dtos::KbLocalMultiTradeSignaturePairDiagnosticSampleDto;
pub(crate) use dtos::KbLocalMultiTradeSignaturePairDiagnosticSampleRow;
pub use dtos::KbLocalPairDiagnosticSummaryDto;
pub(crate) use dtos::KbLocalPairDiagnosticSummaryRow;
pub use dtos::KbLocalPairGapDiagnosticSampleDto;
pub(crate) use dtos::KbLocalPairGapDiagnosticSampleRow;
pub use dtos::KbLocalPipelineDiagnosticCountersDto;
pub(crate) use dtos::KbLocalPipelineDiagnosticCountersRow;
pub use dtos::KbLocalPipelineDiagnosticSummaryDto;
pub use dtos::KbObservedTokenDto;
pub use dtos::KbOnchainObservationDto;
pub use dtos::KbPairAnalyticSignalDto;
@@ -80,6 +97,7 @@ pub use entities::KbWalletEntity;
pub use entities::KbWalletHoldingEntity;
pub use entities::KbWalletParticipationEntity;
pub use queries::delete_chain_instructions_by_transaction_id;
pub use queries::get_chain_instruction_by_id;
pub use queries::get_chain_slot;
pub use queries::get_chain_transaction_by_signature;
pub use queries::get_db_metadata;
@@ -91,6 +109,7 @@ pub use queries::get_latest_pump_fun_create_payload_by_mint;
pub use queries::get_launch_attribution_by_decoded_event_id;
pub use queries::get_launch_surface_by_code;
pub use queries::get_launch_surface_key_by_match;
pub use queries::get_local_pipeline_diagnostic_counters;
pub use queries::get_observed_token_by_mint;
pub use queries::get_pair_analytic_signal_by_key;
pub use queries::get_pair_by_pool_id;
@@ -119,6 +138,14 @@ pub use queries::list_known_ws_endpoints;
pub use queries::list_launch_attributions_by_pool_id;
pub use queries::list_launch_surface_keys_by_surface_id;
pub use queries::list_launch_surfaces;
pub use queries::list_local_decoded_event_diagnostic_summaries;
pub use queries::list_local_dex_diagnostic_summaries;
pub use queries::list_local_duplicate_decoded_event_trade_diagnostic_samples;
pub use queries::list_local_missing_trade_event_diagnostic_samples;
pub use queries::list_local_multi_trade_signature_pair_diagnostic_samples;
pub use queries::list_local_pair_diagnostic_summaries;
pub use queries::list_local_pair_without_candle_diagnostic_samples;
pub use queries::list_local_pair_without_trade_diagnostic_samples;
pub use queries::list_observed_tokens;
pub use queries::list_pair_analytic_signals_by_pair_id;
pub use queries::list_pair_candles_by_pair_and_timeframe;

View File

@@ -16,6 +16,7 @@ mod launch_attribution;
mod launch_surface;
mod launch_surface_key;
mod liquidity_event;
mod local_pipeline_diagnostics;
mod observed_token;
mod onchain_observation;
mod pair;
@@ -49,6 +50,23 @@ pub use launch_attribution::KbLaunchAttributionDto;
pub use launch_surface::KbLaunchSurfaceDto;
pub use launch_surface_key::KbLaunchSurfaceKeyDto;
pub use liquidity_event::KbLiquidityEventDto;
pub use local_pipeline_diagnostics::KbLocalDecodedEventDiagnosticSummaryDto;
pub(crate) use local_pipeline_diagnostics::KbLocalDecodedEventDiagnosticSummaryRow;
pub use local_pipeline_diagnostics::KbLocalDexDiagnosticSummaryDto;
pub(crate) use local_pipeline_diagnostics::KbLocalDexDiagnosticSummaryRow;
pub use local_pipeline_diagnostics::KbLocalDuplicateDecodedEventTradeDiagnosticSampleDto;
pub(crate) use local_pipeline_diagnostics::KbLocalDuplicateDecodedEventTradeDiagnosticSampleRow;
pub use local_pipeline_diagnostics::KbLocalMissingTradeEventDiagnosticSampleDto;
pub(crate) use local_pipeline_diagnostics::KbLocalMissingTradeEventDiagnosticSampleRow;
pub use local_pipeline_diagnostics::KbLocalMultiTradeSignaturePairDiagnosticSampleDto;
pub(crate) use local_pipeline_diagnostics::KbLocalMultiTradeSignaturePairDiagnosticSampleRow;
pub use local_pipeline_diagnostics::KbLocalPairDiagnosticSummaryDto;
pub(crate) use local_pipeline_diagnostics::KbLocalPairDiagnosticSummaryRow;
pub use local_pipeline_diagnostics::KbLocalPairGapDiagnosticSampleDto;
pub(crate) use local_pipeline_diagnostics::KbLocalPairGapDiagnosticSampleRow;
pub use local_pipeline_diagnostics::KbLocalPipelineDiagnosticCountersDto;
pub(crate) use local_pipeline_diagnostics::KbLocalPipelineDiagnosticCountersRow;
pub use local_pipeline_diagnostics::KbLocalPipelineDiagnosticSummaryDto;
pub use observed_token::KbObservedTokenDto;
pub use onchain_observation::KbOnchainObservationDto;
pub use pair::KbPairDto;

View File

@@ -0,0 +1,444 @@
// file: kb_lib/src/db/dtos/local_pipeline_diagnostics.rs
//! DTOs for local pipeline diagnostics.
/// Local pipeline diagnostics summary.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct KbLocalPipelineDiagnosticSummaryDto {
/// Total persisted chain transactions.
pub transaction_count: i64,
/// Total successful chain transactions.
pub ok_transaction_count: i64,
/// Total failed chain transactions.
pub failed_transaction_count: i64,
/// Total decoded DEX events.
pub decoded_event_count: i64,
/// Total decoded DEX trade candidates.
pub decoded_trade_candidate_count: i64,
/// Total decoded DEX candle candidates.
pub decoded_candle_candidate_count: i64,
/// Whether the local persisted pipeline has no blocking diagnostic issue.
pub diagnostics_clean: bool,
/// Number of blocking diagnostic issues.
///
/// This currently includes actionable missing trade events, invalid trade
/// events, duplicate decoded-event trade rows, and duplicate candle buckets.
pub blocking_issue_count: i64,
/// Total decoded DEX events that are trade candidates but have no trade event,
/// including intentionally ignored failed transactions.
pub missing_trade_event_count: i64,
/// Total persisted trade events.
pub trade_event_count: i64,
/// Explicit alias for decoded trade candidates without linked trade event.
pub decoded_trade_candidate_without_trade_event_count: i64,
/// Trade candidates without linked trade event on successful transactions.
pub decoded_trade_candidate_without_trade_event_on_ok_transaction_count: i64,
/// Trade candidates without linked trade event on failed transactions.
pub decoded_trade_candidate_without_trade_event_on_failed_transaction_count: i64,
/// Actionable missing trade events on successful transactions.
pub actionable_missing_trade_event_count: i64,
/// Ignored missing trade events caused by failed transactions.
pub ignored_failed_transaction_trade_candidate_count: i64,
/// Trade candidates without linked trade event and without explicit base/quote payload amounts.
pub decoded_trade_candidate_without_amount_payload_count: i64,
/// Total trade events with missing or invalid pricing fields.
pub invalid_trade_event_count: i64,
/// Total persisted pair candles.
pub pair_candle_count: i64,
/// Real duplicate trade rows grouped by decoded event id.
pub duplicate_decoded_event_trade_count: i64,
/// Multi-trade groups sharing the same signature and pair id.
pub multi_trade_signature_pair_count: i64,
/// Total duplicate candle buckets.
pub duplicate_candle_bucket_count: i64,
/// Total known tokens.
pub token_count: i64,
/// Total tokens missing symbol or name.
pub token_metadata_missing_count: i64,
/// Total known pools.
pub pool_count: i64,
/// Total known pairs.
pub pair_count: i64,
/// Total pairs without trade event.
pub pair_without_trade_count: i64,
/// Total pairs without candle.
pub pair_without_candle_count: i64,
/// Diagnostics grouped by DEX.
pub dex_summaries: std::vec::Vec<crate::KbLocalDexDiagnosticSummaryDto>,
/// Diagnostics grouped by pair.
pub pair_summaries: std::vec::Vec<crate::KbLocalPairDiagnosticSummaryDto>,
/// Diagnostics grouped by decoded event kind.
pub decoded_event_summaries: std::vec::Vec<crate::KbLocalDecodedEventDiagnosticSummaryDto>,
/// Samples of decoded trade candidates without linked trade event.
pub missing_trade_event_samples:
std::vec::Vec<crate::KbLocalMissingTradeEventDiagnosticSampleDto>,
/// Samples of duplicated trade rows by decoded event id.
pub duplicate_decoded_event_trade_samples:
std::vec::Vec<crate::KbLocalDuplicateDecodedEventTradeDiagnosticSampleDto>,
/// Samples of multi-trade signature/pair groups.
pub multi_trade_signature_pair_samples:
std::vec::Vec<crate::KbLocalMultiTradeSignaturePairDiagnosticSampleDto>,
/// Samples of pairs without trade.
pub pair_without_trade_samples: std::vec::Vec<crate::KbLocalPairGapDiagnosticSampleDto>,
/// Samples of pairs without candle.
pub pair_without_candle_samples: std::vec::Vec<crate::KbLocalPairGapDiagnosticSampleDto>,
}
/// Local DEX diagnostics summary.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct KbLocalDexDiagnosticSummaryDto {
/// DEX code or protocol name.
pub dex_code: std::string::String,
/// Total known pools for this DEX.
pub pool_count: i64,
/// Total known pairs for this DEX.
pub pair_count: i64,
/// Total decoded events for this DEX.
pub decoded_event_count: i64,
/// Total decoded trade candidates for this DEX.
pub decoded_trade_candidate_count: i64,
/// Total decoded candle candidates for this DEX.
pub decoded_candle_candidate_count: i64,
/// Total persisted trade events for this DEX.
pub trade_event_count: i64,
/// Total persisted candles for this DEX.
pub pair_candle_count: i64,
}
/// Local pair diagnostics summary.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct KbLocalPairDiagnosticSummaryDto {
/// Pair id.
pub pair_id: i64,
/// Pool address.
pub pool_address: std::string::String,
/// DEX code.
pub dex_code: std::string::String,
/// Base token mint.
pub base_mint: std::string::String,
/// Base token symbol.
pub base_symbol: std::option::Option<std::string::String>,
/// Quote token mint.
pub quote_mint: std::string::String,
/// Quote token symbol.
pub quote_symbol: std::option::Option<std::string::String>,
/// Pair symbol.
pub pair_symbol: std::option::Option<std::string::String>,
/// Total decoded events attached to the pool.
pub decoded_event_count: i64,
/// Total decoded trade candidates attached to the pool.
pub decoded_trade_candidate_count: i64,
/// Total decoded candle candidates attached to the pool.
pub decoded_candle_candidate_count: i64,
/// Total trade events attached to the pair.
pub trade_event_count: i64,
/// Total invalid trade events attached to the pair.
pub invalid_trade_event_count: i64,
/// Total candle buckets attached to the pair.
pub pair_candle_count: i64,
/// Last known price.
pub last_price_quote_per_base: std::option::Option<f64>,
}
/// Local decoded-event diagnostics summary.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct KbLocalDecodedEventDiagnosticSummaryDto {
/// Protocol name.
pub protocol_name: std::string::String,
/// Event kind.
pub event_kind: std::string::String,
/// Event category.
pub event_category: std::option::Option<std::string::String>,
/// Whether payload says this event is a trade candidate.
pub trade_candidate: std::option::Option<bool>,
/// Whether payload says this event is a candle candidate.
pub candle_candidate: std::option::Option<bool>,
/// Event count.
pub event_count: i64,
/// Trade-event count linked to these decoded events.
pub trade_event_count: i64,
}
/// Internal flat counter row for local diagnostics.
#[derive(Debug, Clone)]
pub struct KbLocalPipelineDiagnosticCountersDto {
/// Total persisted chain transactions.
pub transaction_count: i64,
/// Total successful chain transactions.
pub ok_transaction_count: i64,
/// Total failed chain transactions.
pub failed_transaction_count: i64,
/// Total decoded DEX events.
pub decoded_event_count: i64,
/// Total decoded DEX trade candidates.
pub decoded_trade_candidate_count: i64,
/// Total decoded DEX candle candidates.
pub decoded_candle_candidate_count: i64,
/// Total decoded trade candidates without trade event, including ignored failed transactions.
pub missing_trade_event_count: i64,
/// Explicit alias for decoded trade candidates without linked trade event.
pub decoded_trade_candidate_without_trade_event_count: i64,
/// Trade candidates without linked trade event on successful transactions.
pub decoded_trade_candidate_without_trade_event_on_ok_transaction_count: i64,
/// Trade candidates without linked trade event on failed transactions.
pub decoded_trade_candidate_without_trade_event_on_failed_transaction_count: i64,
/// Actionable missing trade events on successful transactions.
pub actionable_missing_trade_event_count: i64,
/// Ignored missing trade events caused by failed transactions.
pub ignored_failed_transaction_trade_candidate_count: i64,
/// Trade candidates without linked trade event and without explicit base/quote payload amounts.
pub decoded_trade_candidate_without_amount_payload_count: i64,
/// Total persisted trade events.
pub trade_event_count: i64,
/// Total invalid trade events.
pub invalid_trade_event_count: i64,
/// Total persisted candles.
pub pair_candle_count: i64,
/// Real duplicate trade rows grouped by decoded event id.
pub duplicate_decoded_event_trade_count: i64,
/// Multi-trade groups sharing the same signature and pair id.
pub multi_trade_signature_pair_count: i64,
/// Total duplicate candle groups.
pub duplicate_candle_bucket_count: i64,
/// Total known tokens.
pub token_count: i64,
/// Total tokens missing metadata.
pub token_metadata_missing_count: i64,
/// Total known pools.
pub pool_count: i64,
/// Total known pairs.
pub pair_count: i64,
/// Total pairs without trade.
pub pair_without_trade_count: i64,
/// Total pairs without candle.
pub pair_without_candle_count: i64,
}
/// SQL row for global local pipeline diagnostic counters.
#[derive(Debug, Clone, sqlx::FromRow)]
pub(crate) struct KbLocalPipelineDiagnosticCountersRow {
pub(crate) transaction_count: i64,
pub(crate) ok_transaction_count: i64,
pub(crate) failed_transaction_count: i64,
pub(crate) decoded_event_count: i64,
pub(crate) decoded_trade_candidate_count: i64,
pub(crate) decoded_candle_candidate_count: i64,
pub(crate) missing_trade_event_count: i64,
pub(crate) decoded_trade_candidate_without_trade_event_count: i64,
pub(crate) decoded_trade_candidate_without_trade_event_on_ok_transaction_count: i64,
pub(crate) decoded_trade_candidate_without_trade_event_on_failed_transaction_count: i64,
pub(crate) actionable_missing_trade_event_count: i64,
pub(crate) ignored_failed_transaction_trade_candidate_count: i64,
pub(crate) decoded_trade_candidate_without_amount_payload_count: i64,
pub(crate) trade_event_count: i64,
pub(crate) invalid_trade_event_count: i64,
pub(crate) pair_candle_count: i64,
pub(crate) duplicate_decoded_event_trade_count: i64,
pub(crate) multi_trade_signature_pair_count: i64,
pub(crate) duplicate_candle_bucket_count: i64,
pub(crate) token_count: i64,
pub(crate) token_metadata_missing_count: i64,
pub(crate) pool_count: i64,
pub(crate) pair_count: i64,
pub(crate) pair_without_trade_count: i64,
pub(crate) pair_without_candle_count: i64,
}
/// SQL row for local DEX diagnostics.
#[derive(Debug, Clone, sqlx::FromRow)]
pub(crate) struct KbLocalDexDiagnosticSummaryRow {
pub(crate) dex_code: std::string::String,
pub(crate) pool_count: i64,
pub(crate) pair_count: i64,
pub(crate) decoded_event_count: i64,
pub(crate) decoded_trade_candidate_count: i64,
pub(crate) decoded_candle_candidate_count: i64,
pub(crate) trade_event_count: i64,
pub(crate) pair_candle_count: i64,
}
/// SQL row for local pair diagnostics.
#[derive(Debug, Clone, sqlx::FromRow)]
pub(crate) struct KbLocalPairDiagnosticSummaryRow {
pub(crate) pair_id: i64,
pub(crate) pool_address: std::string::String,
pub(crate) dex_code: std::string::String,
pub(crate) base_mint: std::string::String,
pub(crate) base_symbol: std::option::Option<std::string::String>,
pub(crate) quote_mint: std::string::String,
pub(crate) quote_symbol: std::option::Option<std::string::String>,
pub(crate) pair_symbol: std::option::Option<std::string::String>,
pub(crate) decoded_event_count: i64,
pub(crate) decoded_trade_candidate_count: i64,
pub(crate) decoded_candle_candidate_count: i64,
pub(crate) trade_event_count: i64,
pub(crate) invalid_trade_event_count: i64,
pub(crate) pair_candle_count: i64,
pub(crate) last_price_quote_per_base: std::option::Option<f64>,
}
/// SQL row for local decoded-event diagnostics.
#[derive(Debug, Clone, sqlx::FromRow)]
pub(crate) struct KbLocalDecodedEventDiagnosticSummaryRow {
pub(crate) protocol_name: std::string::String,
pub(crate) event_kind: std::string::String,
pub(crate) event_category: std::option::Option<std::string::String>,
pub(crate) trade_candidate: std::option::Option<i64>,
pub(crate) candle_candidate: std::option::Option<i64>,
pub(crate) event_count: i64,
pub(crate) trade_event_count: i64,
}
/// Sample of a decoded trade candidate without linked trade event.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct KbLocalMissingTradeEventDiagnosticSampleDto {
/// Decoded event id.
pub decoded_event_id: i64,
/// Chain transaction id.
pub transaction_id: std::option::Option<i64>,
/// Transaction signature.
pub signature: std::option::Option<std::string::String>,
/// Protocol name.
pub protocol_name: std::string::String,
/// Event kind.
pub event_kind: std::string::String,
/// Pool account.
pub pool_account: std::option::Option<std::string::String>,
/// Whether the source transaction failed.
pub transaction_failed: bool,
/// Diagnostic reason explaining why no trade event was linked.
pub reason: std::string::String,
/// Whether payload has an explicit base amount.
pub has_base_amount_payload: bool,
/// Whether payload has an explicit quote amount.
pub has_quote_amount_payload: bool,
/// Whether payload has an explicit price.
pub has_price_payload: bool,
}
/// Sample of duplicated trade rows grouped by decoded event id.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct KbLocalDuplicateDecodedEventTradeDiagnosticSampleDto {
/// Decoded event id.
pub decoded_event_id: i64,
/// Protocol name.
pub protocol_name: std::option::Option<std::string::String>,
/// Event kind.
pub event_kind: std::option::Option<std::string::String>,
/// Pool account.
pub pool_account: std::option::Option<std::string::String>,
/// Duplicate trade row count.
pub trade_event_count: i64,
/// Trade event ids.
pub trade_event_ids: std::option::Option<std::string::String>,
/// Signatures.
pub signatures: std::option::Option<std::string::String>,
}
/// Sample of multi-trade groups sharing the same signature and pair id.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct KbLocalMultiTradeSignaturePairDiagnosticSampleDto {
/// Transaction signature.
pub signature: std::string::String,
/// Pair id.
pub pair_id: i64,
/// Pool address.
pub pool_address: std::option::Option<std::string::String>,
/// DEX code.
pub dex_code: std::option::Option<std::string::String>,
/// Trade event count.
pub trade_event_count: i64,
/// Distinct decoded event count.
pub decoded_event_count: i64,
/// Trade event ids.
pub trade_event_ids: std::option::Option<std::string::String>,
/// Decoded event ids.
pub decoded_event_ids: std::option::Option<std::string::String>,
}
/// Sample of a pair gap.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct KbLocalPairGapDiagnosticSampleDto {
/// Pair id.
pub pair_id: i64,
/// Pool address.
pub pool_address: std::string::String,
/// DEX code.
pub dex_code: std::string::String,
/// Base mint.
pub base_mint: std::string::String,
/// Base symbol.
pub base_symbol: std::option::Option<std::string::String>,
/// Quote mint.
pub quote_mint: std::string::String,
/// Quote symbol.
pub quote_symbol: std::option::Option<std::string::String>,
/// Pair symbol.
pub pair_symbol: std::option::Option<std::string::String>,
/// Decoded event count.
pub decoded_event_count: i64,
/// Decoded trade candidate count.
pub decoded_trade_candidate_count: i64,
/// Trade event count.
pub trade_event_count: i64,
/// Pair candle count.
pub pair_candle_count: i64,
}
/// SQL row for missing trade event samples.
#[derive(Debug, Clone, sqlx::FromRow)]
pub(crate) struct KbLocalMissingTradeEventDiagnosticSampleRow {
pub(crate) decoded_event_id: i64,
pub(crate) transaction_id: std::option::Option<i64>,
pub(crate) signature: std::option::Option<std::string::String>,
pub(crate) protocol_name: std::string::String,
pub(crate) event_kind: std::string::String,
pub(crate) pool_account: std::option::Option<std::string::String>,
pub(crate) transaction_failed: i64,
pub(crate) reason: std::string::String,
pub(crate) has_base_amount_payload: i64,
pub(crate) has_quote_amount_payload: i64,
pub(crate) has_price_payload: i64,
}
/// SQL row for duplicated decoded event trade samples.
#[derive(Debug, Clone, sqlx::FromRow)]
pub(crate) struct KbLocalDuplicateDecodedEventTradeDiagnosticSampleRow {
pub(crate) decoded_event_id: i64,
pub(crate) protocol_name: std::option::Option<std::string::String>,
pub(crate) event_kind: std::option::Option<std::string::String>,
pub(crate) pool_account: std::option::Option<std::string::String>,
pub(crate) trade_event_count: i64,
pub(crate) trade_event_ids: std::option::Option<std::string::String>,
pub(crate) signatures: std::option::Option<std::string::String>,
}
/// SQL row for multi-trade signature/pair samples.
#[derive(Debug, Clone, sqlx::FromRow)]
pub(crate) struct KbLocalMultiTradeSignaturePairDiagnosticSampleRow {
pub(crate) signature: std::string::String,
pub(crate) pair_id: i64,
pub(crate) pool_address: std::option::Option<std::string::String>,
pub(crate) dex_code: std::option::Option<std::string::String>,
pub(crate) trade_event_count: i64,
pub(crate) decoded_event_count: i64,
pub(crate) trade_event_ids: std::option::Option<std::string::String>,
pub(crate) decoded_event_ids: std::option::Option<std::string::String>,
}
/// SQL row for pair gap samples.
#[derive(Debug, Clone, sqlx::FromRow)]
pub(crate) struct KbLocalPairGapDiagnosticSampleRow {
pub(crate) pair_id: i64,
pub(crate) pool_address: std::string::String,
pub(crate) dex_code: std::string::String,
pub(crate) base_mint: std::string::String,
pub(crate) base_symbol: std::option::Option<std::string::String>,
pub(crate) quote_mint: std::string::String,
pub(crate) quote_symbol: std::option::Option<std::string::String>,
pub(crate) pair_symbol: std::option::Option<std::string::String>,
pub(crate) decoded_event_count: i64,
pub(crate) decoded_trade_candidate_count: i64,
pub(crate) trade_event_count: i64,
pub(crate) pair_candle_count: i64,
}

View File

@@ -20,6 +20,7 @@ mod launch_attribution;
mod launch_surface;
mod launch_surface_key;
mod liquidity_event;
mod local_pipeline_diagnostics;
mod observed_token;
mod onchain_observation;
mod pair;
@@ -42,6 +43,7 @@ mod wallet_participation;
pub use analysis_signal::insert_analysis_signal;
pub use analysis_signal::list_recent_analysis_signals;
pub use chain_instruction::delete_chain_instructions_by_transaction_id;
pub use chain_instruction::get_chain_instruction_by_id;
pub use chain_instruction::insert_chain_instruction;
pub use chain_instruction::list_chain_instructions_by_transaction_id;
pub use chain_slot::get_chain_slot;
@@ -80,6 +82,15 @@ pub use launch_surface_key::list_launch_surface_keys_by_surface_id;
pub use launch_surface_key::upsert_launch_surface_key;
pub use liquidity_event::list_recent_liquidity_events;
pub use liquidity_event::upsert_liquidity_event;
pub use local_pipeline_diagnostics::get_local_pipeline_diagnostic_counters;
pub use local_pipeline_diagnostics::list_local_decoded_event_diagnostic_summaries;
pub use local_pipeline_diagnostics::list_local_dex_diagnostic_summaries;
pub use local_pipeline_diagnostics::list_local_duplicate_decoded_event_trade_diagnostic_samples;
pub use local_pipeline_diagnostics::list_local_missing_trade_event_diagnostic_samples;
pub use local_pipeline_diagnostics::list_local_multi_trade_signature_pair_diagnostic_samples;
pub use local_pipeline_diagnostics::list_local_pair_diagnostic_summaries;
pub use local_pipeline_diagnostics::list_local_pair_without_candle_diagnostic_samples;
pub use local_pipeline_diagnostics::list_local_pair_without_trade_diagnostic_samples;
pub use observed_token::get_observed_token_by_mint;
pub use observed_token::list_observed_tokens;
pub use observed_token::upsert_observed_token;

View File

@@ -65,6 +65,60 @@ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
}
}
/// Reads one chain instruction by its internal id.
pub async fn get_chain_instruction_by_id(
database: &crate::KbDatabase,
instruction_id: i64,
) -> Result<std::option::Option<crate::KbChainInstructionDto>, crate::KbError> {
match database.connection() {
crate::KbDatabaseConnection::Sqlite(pool) => {
let query_result = sqlx::query_as::<sqlx::Sqlite, crate::KbChainInstructionEntity>(
r#"
SELECT
id,
transaction_id,
parent_instruction_id,
instruction_index,
inner_instruction_index,
program_id,
program_name,
stack_height,
accounts_json,
data_json,
parsed_type,
parsed_json,
created_at
FROM kb_chain_instructions
WHERE id = ?
LIMIT 1
"#,
)
.bind(instruction_id)
.fetch_optional(pool)
.await;
let entity_option = match query_result {
Ok(entity_option) => entity_option,
Err(error) => {
return Err(crate::KbError::Db(format!(
"cannot fetch kb_chain_instructions id '{}' on sqlite: {}",
instruction_id, error
)));
},
};
match entity_option {
Some(entity) => {
let dto_result = crate::KbChainInstructionDto::try_from(entity);
match dto_result {
Ok(dto) => return Ok(Some(dto)),
Err(error) => return Err(error),
}
},
None => return Ok(None),
}
},
}
}
/// Lists instructions for one transaction ordered from outer to inner.
pub async fn list_chain_instructions_by_transaction_id(
database: &crate::KbDatabase,

View File

@@ -0,0 +1,836 @@
// file: kb_lib/src/db/queries/local_pipeline_diagnostics.rs
//! Local pipeline diagnostics SQL queries.
/// Returns global local-pipeline diagnostic counters.
pub async fn get_local_pipeline_diagnostic_counters(
database: &crate::KbDatabase,
) -> Result<crate::KbLocalPipelineDiagnosticCountersDto, crate::KbError> {
match database.connection() {
crate::KbDatabaseConnection::Sqlite(pool) => {
let row_result =
sqlx::query_as::<sqlx::Sqlite, crate::KbLocalPipelineDiagnosticCountersRow>(
r#"
SELECT
(SELECT COUNT(*) FROM kb_chain_transactions) AS transaction_count,
(SELECT COUNT(*) FROM kb_chain_transactions WHERE err_json IS NULL) AS ok_transaction_count,
(SELECT COUNT(*) FROM kb_chain_transactions WHERE err_json IS NOT NULL) AS failed_transaction_count,
(SELECT COUNT(*) FROM kb_dex_decoded_events) AS decoded_event_count,
(
SELECT COUNT(*)
FROM kb_dex_decoded_events
WHERE json_extract(payload_json, '$.tradeCandidate') = 1
) AS decoded_trade_candidate_count,
(
SELECT COUNT(*)
FROM kb_dex_decoded_events
WHERE json_extract(payload_json, '$.candleCandidate') = 1
) AS decoded_candle_candidate_count,
(
SELECT COUNT(*)
FROM kb_dex_decoded_events dde
LEFT JOIN kb_trade_events te ON te.decoded_event_id = dde.id
WHERE json_extract(dde.payload_json, '$.tradeCandidate') = 1
AND te.id IS NULL
) AS missing_trade_event_count,
(
SELECT COUNT(*)
FROM kb_dex_decoded_events dde
LEFT JOIN kb_trade_events te ON te.decoded_event_id = dde.id
WHERE json_extract(dde.payload_json, '$.tradeCandidate') = 1
AND te.id IS NULL
) AS decoded_trade_candidate_without_trade_event_count,
(
SELECT COUNT(*)
FROM kb_dex_decoded_events dde
LEFT JOIN kb_trade_events te ON te.decoded_event_id = dde.id
LEFT JOIN kb_chain_transactions ct ON ct.id = dde.transaction_id
WHERE json_extract(dde.payload_json, '$.tradeCandidate') = 1
AND te.id IS NULL
AND ct.id IS NOT NULL
AND ct.err_json IS NULL
) AS decoded_trade_candidate_without_trade_event_on_ok_transaction_count,
(
SELECT COUNT(*)
FROM kb_dex_decoded_events dde
LEFT JOIN kb_trade_events te ON te.decoded_event_id = dde.id
LEFT JOIN kb_chain_transactions ct ON ct.id = dde.transaction_id
WHERE json_extract(dde.payload_json, '$.tradeCandidate') = 1
AND te.id IS NULL
AND ct.id IS NOT NULL
AND ct.err_json IS NOT NULL
) AS decoded_trade_candidate_without_trade_event_on_failed_transaction_count,
(
SELECT COUNT(*)
FROM kb_dex_decoded_events dde
LEFT JOIN kb_trade_events te ON te.decoded_event_id = dde.id
LEFT JOIN kb_chain_transactions ct ON ct.id = dde.transaction_id
WHERE json_extract(dde.payload_json, '$.tradeCandidate') = 1
AND te.id IS NULL
AND ct.id IS NOT NULL
AND ct.err_json IS NULL
) AS actionable_missing_trade_event_count,
(
SELECT COUNT(*)
FROM kb_dex_decoded_events dde
LEFT JOIN kb_trade_events te ON te.decoded_event_id = dde.id
LEFT JOIN kb_chain_transactions ct ON ct.id = dde.transaction_id
WHERE json_extract(dde.payload_json, '$.tradeCandidate') = 1
AND te.id IS NULL
AND ct.id IS NOT NULL
AND ct.err_json IS NOT NULL
) AS ignored_failed_transaction_trade_candidate_count,
(
SELECT COUNT(*)
FROM kb_dex_decoded_events dde
LEFT JOIN kb_trade_events te ON te.decoded_event_id = dde.id
WHERE json_extract(dde.payload_json, '$.tradeCandidate') = 1
AND te.id IS NULL
AND (
(
json_extract(dde.payload_json, '$.baseAmountRaw') IS NULL
AND json_extract(dde.payload_json, '$.base_amount_raw') IS NULL
)
OR (
json_extract(dde.payload_json, '$.quoteAmountRaw') IS NULL
AND json_extract(dde.payload_json, '$.quote_amount_raw') IS NULL
)
)
) AS decoded_trade_candidate_without_amount_payload_count,
(SELECT COUNT(*) FROM kb_trade_events) AS trade_event_count,
(
SELECT COUNT(*)
FROM kb_trade_events
WHERE base_amount_raw IS NULL
OR quote_amount_raw IS NULL
OR price_quote_per_base IS NULL
OR CAST(base_amount_raw AS INTEGER) <= 0
OR CAST(quote_amount_raw AS INTEGER) <= 0
OR price_quote_per_base <= 0
) AS invalid_trade_event_count,
(SELECT COUNT(*) FROM kb_pair_candles) AS pair_candle_count,
(
SELECT COUNT(*)
FROM (
SELECT decoded_event_id
FROM kb_trade_events
WHERE decoded_event_id IS NOT NULL
GROUP BY decoded_event_id
HAVING COUNT(*) > 1
)
) AS duplicate_decoded_event_trade_count,
(
SELECT COUNT(*)
FROM (
SELECT signature, pair_id
FROM kb_trade_events
GROUP BY signature, pair_id
HAVING COUNT(*) > 1
)
) AS multi_trade_signature_pair_count,
(
SELECT COUNT(*)
FROM (
SELECT pair_id, timeframe_seconds, bucket_start_unix
FROM kb_pair_candles
GROUP BY pair_id, timeframe_seconds, bucket_start_unix
HAVING COUNT(*) > 1
)
) AS duplicate_candle_bucket_count,
(SELECT COUNT(*) FROM kb_tokens) AS token_count,
(
SELECT COUNT(*)
FROM kb_tokens
WHERE symbol IS NULL
OR symbol = ''
OR name IS NULL
OR name = ''
) AS token_metadata_missing_count,
(SELECT COUNT(*) FROM kb_pools) AS pool_count,
(SELECT COUNT(*) FROM kb_pairs) AS pair_count,
(
SELECT COUNT(*)
FROM (
SELECT pair.id
FROM kb_pairs pair
LEFT JOIN kb_trade_events te ON te.pair_id = pair.id
GROUP BY pair.id
HAVING COUNT(te.id) = 0
)
) AS pair_without_trade_count,
(
SELECT COUNT(*)
FROM (
SELECT pair.id
FROM kb_pairs pair
LEFT JOIN kb_pair_candles pc ON pc.pair_id = pair.id
GROUP BY pair.id
HAVING COUNT(pc.pair_id) = 0
)
) AS pair_without_candle_count
"#,
)
.fetch_one(pool)
.await;
let row = match row_result {
Ok(row) => row,
Err(error) => {
return Err(crate::KbError::Db(format!(
"cannot read local pipeline diagnostic counters on sqlite: {}",
error
)));
},
};
return Ok(crate::KbLocalPipelineDiagnosticCountersDto {
transaction_count: row.transaction_count,
ok_transaction_count: row.ok_transaction_count,
failed_transaction_count: row.failed_transaction_count,
decoded_event_count: row.decoded_event_count,
decoded_trade_candidate_count: row.decoded_trade_candidate_count,
decoded_candle_candidate_count: row.decoded_candle_candidate_count,
missing_trade_event_count: row.missing_trade_event_count,
decoded_trade_candidate_without_trade_event_count: row
.decoded_trade_candidate_without_trade_event_count,
decoded_trade_candidate_without_trade_event_on_ok_transaction_count: row
.decoded_trade_candidate_without_trade_event_on_ok_transaction_count,
decoded_trade_candidate_without_trade_event_on_failed_transaction_count: row
.decoded_trade_candidate_without_trade_event_on_failed_transaction_count,
decoded_trade_candidate_without_amount_payload_count: row
.decoded_trade_candidate_without_amount_payload_count,
trade_event_count: row.trade_event_count,
invalid_trade_event_count: row.invalid_trade_event_count,
pair_candle_count: row.pair_candle_count,
duplicate_decoded_event_trade_count: row.duplicate_decoded_event_trade_count,
multi_trade_signature_pair_count: row.multi_trade_signature_pair_count,
duplicate_candle_bucket_count: row.duplicate_candle_bucket_count,
token_count: row.token_count,
token_metadata_missing_count: row.token_metadata_missing_count,
pool_count: row.pool_count,
pair_count: row.pair_count,
pair_without_trade_count: row.pair_without_trade_count,
pair_without_candle_count: row.pair_without_candle_count,
actionable_missing_trade_event_count: row.actionable_missing_trade_event_count,
ignored_failed_transaction_trade_candidate_count: row
.ignored_failed_transaction_trade_candidate_count,
});
},
}
}
/// Lists local DEX diagnostic summaries.
pub async fn list_local_dex_diagnostic_summaries(
database: &crate::KbDatabase,
) -> Result<std::vec::Vec<crate::KbLocalDexDiagnosticSummaryDto>, crate::KbError> {
match database.connection() {
crate::KbDatabaseConnection::Sqlite(pool) => {
let rows_result = sqlx::query_as::<sqlx::Sqlite, crate::KbLocalDexDiagnosticSummaryRow>(
r#"
WITH decoded AS (
SELECT
protocol_name AS dex_code,
COUNT(*) AS decoded_event_count,
SUM(CASE WHEN json_extract(payload_json, '$.tradeCandidate') = 1 THEN 1 ELSE 0 END) AS decoded_trade_candidate_count,
SUM(CASE WHEN json_extract(payload_json, '$.candleCandidate') = 1 THEN 1 ELSE 0 END) AS decoded_candle_candidate_count
FROM kb_dex_decoded_events
GROUP BY protocol_name
),
dex_pool_pairs AS (
SELECT
d.code AS dex_code,
COUNT(DISTINCT p.id) AS pool_count,
COUNT(DISTINCT pair.id) AS pair_count
FROM kb_dexes d
LEFT JOIN kb_pools p ON p.dex_id = d.id
LEFT JOIN kb_pairs pair ON pair.pool_id = p.id
GROUP BY d.code
),
trades AS (
SELECT
d.code AS dex_code,
COUNT(te.id) AS trade_event_count
FROM kb_trade_events te
JOIN kb_pairs pair ON pair.id = te.pair_id
JOIN kb_pools p ON p.id = pair.pool_id
JOIN kb_dexes d ON d.id = p.dex_id
GROUP BY d.code
),
candles AS (
SELECT
d.code AS dex_code,
COUNT(pc.pair_id) AS pair_candle_count
FROM kb_pair_candles pc
JOIN kb_pairs pair ON pair.id = pc.pair_id
JOIN kb_pools p ON p.id = pair.pool_id
JOIN kb_dexes d ON d.id = p.dex_id
GROUP BY d.code
)
SELECT
COALESCE(dex_pool_pairs.dex_code, decoded.dex_code) AS dex_code,
COALESCE(dex_pool_pairs.pool_count, 0) AS pool_count,
COALESCE(dex_pool_pairs.pair_count, 0) AS pair_count,
COALESCE(decoded.decoded_event_count, 0) AS decoded_event_count,
COALESCE(decoded.decoded_trade_candidate_count, 0) AS decoded_trade_candidate_count,
COALESCE(decoded.decoded_candle_candidate_count, 0) AS decoded_candle_candidate_count,
COALESCE(trades.trade_event_count, 0) AS trade_event_count,
COALESCE(candles.pair_candle_count, 0) AS pair_candle_count
FROM dex_pool_pairs
LEFT JOIN decoded ON decoded.dex_code = dex_pool_pairs.dex_code
LEFT JOIN trades ON trades.dex_code = dex_pool_pairs.dex_code
LEFT JOIN candles ON candles.dex_code = dex_pool_pairs.dex_code
UNION
SELECT
decoded.dex_code AS dex_code,
0 AS pool_count,
0 AS pair_count,
decoded.decoded_event_count AS decoded_event_count,
decoded.decoded_trade_candidate_count AS decoded_trade_candidate_count,
decoded.decoded_candle_candidate_count AS decoded_candle_candidate_count,
0 AS trade_event_count,
0 AS pair_candle_count
FROM decoded
LEFT JOIN dex_pool_pairs ON dex_pool_pairs.dex_code = decoded.dex_code
WHERE dex_pool_pairs.dex_code IS NULL
ORDER BY dex_code
"#,
)
.fetch_all(pool)
.await;
let rows = match rows_result {
Ok(rows) => rows,
Err(error) => {
return Err(crate::KbError::Db(format!(
"cannot list local dex diagnostic summaries on sqlite: {}",
error
)));
},
};
let mut summaries = std::vec::Vec::new();
for row in rows {
summaries.push(crate::KbLocalDexDiagnosticSummaryDto {
dex_code: row.dex_code,
pool_count: row.pool_count,
pair_count: row.pair_count,
decoded_event_count: row.decoded_event_count,
decoded_trade_candidate_count: row.decoded_trade_candidate_count,
decoded_candle_candidate_count: row.decoded_candle_candidate_count,
trade_event_count: row.trade_event_count,
pair_candle_count: row.pair_candle_count,
});
}
return Ok(summaries);
},
}
}
/// Lists local pair diagnostic summaries.
pub async fn list_local_pair_diagnostic_summaries(
database: &crate::KbDatabase,
) -> Result<std::vec::Vec<crate::KbLocalPairDiagnosticSummaryDto>, crate::KbError> {
match database.connection() {
crate::KbDatabaseConnection::Sqlite(pool) => {
let rows_result = sqlx::query_as::<sqlx::Sqlite, crate::KbLocalPairDiagnosticSummaryRow>(
r#"
SELECT
pair.id AS pair_id,
p.address AS pool_address,
d.code AS dex_code,
base_token.mint AS base_mint,
base_token.symbol AS base_symbol,
quote_token.mint AS quote_mint,
quote_token.symbol AS quote_symbol,
pair.symbol AS pair_symbol,
COUNT(DISTINCT dde.id) AS decoded_event_count,
COUNT(DISTINCT CASE WHEN json_extract(dde.payload_json, '$.tradeCandidate') = 1 THEN dde.id END) AS decoded_trade_candidate_count,
COUNT(DISTINCT CASE WHEN json_extract(dde.payload_json, '$.candleCandidate') = 1 THEN dde.id END) AS decoded_candle_candidate_count,
COUNT(DISTINCT te.id) AS trade_event_count,
COUNT(
DISTINCT CASE
WHEN te.id IS NOT NULL
AND (
te.base_amount_raw IS NULL
OR te.quote_amount_raw IS NULL
OR te.price_quote_per_base IS NULL
OR CAST(te.base_amount_raw AS INTEGER) <= 0
OR CAST(te.quote_amount_raw AS INTEGER) <= 0
OR te.price_quote_per_base <= 0
)
THEN te.id
END
) AS invalid_trade_event_count,
COUNT(DISTINCT pc.bucket_start_unix || ':' || pc.timeframe_seconds) AS pair_candle_count,
(
SELECT te_last.price_quote_per_base
FROM kb_trade_events te_last
WHERE te_last.pair_id = pair.id
ORDER BY te_last.id DESC
LIMIT 1
) AS last_price_quote_per_base
FROM kb_pairs pair
JOIN kb_pools p ON p.id = pair.pool_id
JOIN kb_dexes d ON d.id = p.dex_id
JOIN kb_tokens base_token ON base_token.id = pair.base_token_id
JOIN kb_tokens quote_token ON quote_token.id = pair.quote_token_id
LEFT JOIN kb_dex_decoded_events dde ON dde.pool_account = p.address
LEFT JOIN kb_trade_events te ON te.pair_id = pair.id
LEFT JOIN kb_pair_candles pc ON pc.pair_id = pair.id
GROUP BY
pair.id,
p.address,
d.code,
base_token.mint,
base_token.symbol,
quote_token.mint,
quote_token.symbol,
pair.symbol
ORDER BY pair.id
"#,
)
.fetch_all(pool)
.await;
let rows = match rows_result {
Ok(rows) => rows,
Err(error) => {
return Err(crate::KbError::Db(format!(
"cannot list local pair diagnostic summaries on sqlite: {}",
error
)));
},
};
let mut summaries = std::vec::Vec::new();
for row in rows {
summaries.push(crate::KbLocalPairDiagnosticSummaryDto {
pair_id: row.pair_id,
pool_address: row.pool_address,
dex_code: row.dex_code,
base_mint: row.base_mint,
base_symbol: row.base_symbol,
quote_mint: row.quote_mint,
quote_symbol: row.quote_symbol,
pair_symbol: row.pair_symbol,
decoded_event_count: row.decoded_event_count,
decoded_trade_candidate_count: row.decoded_trade_candidate_count,
decoded_candle_candidate_count: row.decoded_candle_candidate_count,
trade_event_count: row.trade_event_count,
invalid_trade_event_count: row.invalid_trade_event_count,
pair_candle_count: row.pair_candle_count,
last_price_quote_per_base: row.last_price_quote_per_base,
});
}
return Ok(summaries);
},
}
}
/// Lists local decoded-event diagnostic summaries.
pub async fn list_local_decoded_event_diagnostic_summaries(
database: &crate::KbDatabase,
) -> Result<std::vec::Vec<crate::KbLocalDecodedEventDiagnosticSummaryDto>, crate::KbError> {
match database.connection() {
crate::KbDatabaseConnection::Sqlite(pool) => {
let rows_result =
sqlx::query_as::<sqlx::Sqlite, crate::KbLocalDecodedEventDiagnosticSummaryRow>(
r#"
SELECT
dde.protocol_name AS protocol_name,
dde.event_kind AS event_kind,
json_extract(dde.payload_json, '$.eventCategory') AS event_category,
json_extract(dde.payload_json, '$.tradeCandidate') AS trade_candidate,
json_extract(dde.payload_json, '$.candleCandidate') AS candle_candidate,
COUNT(dde.id) AS event_count,
COUNT(te.id) AS trade_event_count
FROM kb_dex_decoded_events dde
LEFT JOIN kb_trade_events te ON te.decoded_event_id = dde.id
GROUP BY
dde.protocol_name,
dde.event_kind,
event_category,
trade_candidate,
candle_candidate
ORDER BY
dde.protocol_name,
dde.event_kind
"#,
)
.fetch_all(pool)
.await;
let rows = match rows_result {
Ok(rows) => rows,
Err(error) => {
return Err(crate::KbError::Db(format!(
"cannot list local decoded event diagnostic summaries on sqlite: {}",
error
)));
},
};
let mut summaries = std::vec::Vec::new();
for row in rows {
summaries.push(crate::KbLocalDecodedEventDiagnosticSummaryDto {
protocol_name: row.protocol_name,
event_kind: row.event_kind,
event_category: row.event_category,
trade_candidate: kb_sqlite_bool_to_option(row.trade_candidate),
candle_candidate: kb_sqlite_bool_to_option(row.candle_candidate),
event_count: row.event_count,
trade_event_count: row.trade_event_count,
});
}
return Ok(summaries);
},
}
}
fn kb_sqlite_bool_to_option(value: std::option::Option<i64>) -> std::option::Option<bool> {
match value {
Some(0) => return Some(false),
Some(_) => return Some(true),
None => return None,
}
}
/// Lists samples of decoded trade candidates without linked trade event.
pub async fn list_local_missing_trade_event_diagnostic_samples(
database: &crate::KbDatabase,
limit: i64,
) -> Result<std::vec::Vec<crate::KbLocalMissingTradeEventDiagnosticSampleDto>, crate::KbError> {
match database.connection() {
crate::KbDatabaseConnection::Sqlite(pool) => {
let rows_result =
sqlx::query_as::<sqlx::Sqlite, crate::KbLocalMissingTradeEventDiagnosticSampleRow>(
r#"
SELECT
dde.id AS decoded_event_id,
dde.transaction_id AS transaction_id,
ct.signature AS signature,
dde.protocol_name AS protocol_name,
dde.event_kind AS event_kind,
dde.pool_account AS pool_account,
CASE WHEN ct.err_json IS NOT NULL THEN 1 ELSE 0 END AS transaction_failed,
CASE
WHEN ct.err_json IS NOT NULL THEN 'failed_transaction'
WHEN (
json_extract(dde.payload_json, '$.baseAmountRaw') IS NULL
AND json_extract(dde.payload_json, '$.base_amount_raw') IS NULL
AND json_extract(dde.payload_json, '$.quoteAmountRaw') IS NULL
AND json_extract(dde.payload_json, '$.quote_amount_raw') IS NULL
AND json_extract(dde.payload_json, '$.priceQuotePerBase') IS NULL
AND json_extract(dde.payload_json, '$.price_quote_per_base') IS NULL
) THEN 'ok_transaction_without_amount_payload'
WHEN (
json_extract(dde.payload_json, '$.baseAmountRaw') IS NULL
AND json_extract(dde.payload_json, '$.base_amount_raw') IS NULL
) THEN 'ok_transaction_without_base_amount_payload'
WHEN (
json_extract(dde.payload_json, '$.quoteAmountRaw') IS NULL
AND json_extract(dde.payload_json, '$.quote_amount_raw') IS NULL
) THEN 'ok_transaction_without_quote_amount_payload'
WHEN (
json_extract(dde.payload_json, '$.priceQuotePerBase') IS NULL
AND json_extract(dde.payload_json, '$.price_quote_per_base') IS NULL
) THEN 'ok_transaction_without_price_payload'
ELSE 'ok_transaction_unclassified'
END AS reason,
CASE
WHEN json_extract(dde.payload_json, '$.baseAmountRaw') IS NOT NULL
OR json_extract(dde.payload_json, '$.base_amount_raw') IS NOT NULL
THEN 1
ELSE 0
END AS has_base_amount_payload,
CASE
WHEN json_extract(dde.payload_json, '$.quoteAmountRaw') IS NOT NULL
OR json_extract(dde.payload_json, '$.quote_amount_raw') IS NOT NULL
THEN 1
ELSE 0
END AS has_quote_amount_payload,
CASE
WHEN json_extract(dde.payload_json, '$.priceQuotePerBase') IS NOT NULL
OR json_extract(dde.payload_json, '$.price_quote_per_base') IS NOT NULL
THEN 1
ELSE 0
END AS has_price_payload
FROM kb_dex_decoded_events dde
LEFT JOIN kb_trade_events te ON te.decoded_event_id = dde.id
LEFT JOIN kb_chain_transactions ct ON ct.id = dde.transaction_id
WHERE json_extract(dde.payload_json, '$.tradeCandidate') = 1
AND te.id IS NULL
ORDER BY
transaction_failed ASC,
dde.protocol_name,
dde.event_kind,
dde.id
LIMIT ?
"#,
)
.bind(limit)
.fetch_all(pool)
.await;
let rows = match rows_result {
Ok(rows) => rows,
Err(error) => {
return Err(crate::KbError::Db(format!(
"cannot list missing trade event diagnostic samples on sqlite: {}",
error
)));
},
};
let mut samples = std::vec::Vec::new();
for row in rows {
samples.push(crate::KbLocalMissingTradeEventDiagnosticSampleDto {
decoded_event_id: row.decoded_event_id,
transaction_id: row.transaction_id,
signature: row.signature,
protocol_name: row.protocol_name,
event_kind: row.event_kind,
pool_account: row.pool_account,
transaction_failed: kb_sqlite_i64_to_bool(row.transaction_failed),
reason: row.reason,
has_base_amount_payload: kb_sqlite_i64_to_bool(row.has_base_amount_payload),
has_quote_amount_payload: kb_sqlite_i64_to_bool(row.has_quote_amount_payload),
has_price_payload: kb_sqlite_i64_to_bool(row.has_price_payload),
});
}
return Ok(samples);
},
}
}
/// Lists samples of duplicated trade rows by decoded event id.
pub async fn list_local_duplicate_decoded_event_trade_diagnostic_samples(
database: &crate::KbDatabase,
limit: i64,
) -> Result<
std::vec::Vec<crate::KbLocalDuplicateDecodedEventTradeDiagnosticSampleDto>,
crate::KbError,
> {
match database.connection() {
crate::KbDatabaseConnection::Sqlite(pool) => {
let rows_result = sqlx::query_as::<
sqlx::Sqlite,
crate::KbLocalDuplicateDecodedEventTradeDiagnosticSampleRow,
>(
r#"
SELECT
te.decoded_event_id AS decoded_event_id,
dde.protocol_name AS protocol_name,
dde.event_kind AS event_kind,
dde.pool_account AS pool_account,
COUNT(te.id) AS trade_event_count,
GROUP_CONCAT(te.id) AS trade_event_ids,
GROUP_CONCAT(te.signature) AS signatures
FROM kb_trade_events te
LEFT JOIN kb_dex_decoded_events dde ON dde.id = te.decoded_event_id
WHERE te.decoded_event_id IS NOT NULL
GROUP BY
te.decoded_event_id,
dde.protocol_name,
dde.event_kind,
dde.pool_account
HAVING COUNT(te.id) > 1
ORDER BY trade_event_count DESC, te.decoded_event_id
LIMIT ?
"#,
)
.bind(limit)
.fetch_all(pool)
.await;
let rows = match rows_result {
Ok(rows) => rows,
Err(error) => {
return Err(crate::KbError::Db(format!(
"cannot list duplicate decoded event trade diagnostic samples on sqlite: {}",
error
)));
},
};
let mut samples = std::vec::Vec::new();
for row in rows {
samples.push(crate::KbLocalDuplicateDecodedEventTradeDiagnosticSampleDto {
decoded_event_id: row.decoded_event_id,
protocol_name: row.protocol_name,
event_kind: row.event_kind,
pool_account: row.pool_account,
trade_event_count: row.trade_event_count,
trade_event_ids: row.trade_event_ids,
signatures: row.signatures,
});
}
return Ok(samples);
},
}
}
/// Lists samples of multi-trade signature/pair groups.
pub async fn list_local_multi_trade_signature_pair_diagnostic_samples(
database: &crate::KbDatabase,
limit: i64,
) -> Result<std::vec::Vec<crate::KbLocalMultiTradeSignaturePairDiagnosticSampleDto>, crate::KbError>
{
match database.connection() {
crate::KbDatabaseConnection::Sqlite(pool) => {
let rows_result = sqlx::query_as::<
sqlx::Sqlite,
crate::KbLocalMultiTradeSignaturePairDiagnosticSampleRow,
>(
r#"
SELECT
te.signature AS signature,
te.pair_id AS pair_id,
p.address AS pool_address,
d.code AS dex_code,
COUNT(te.id) AS trade_event_count,
COUNT(DISTINCT te.decoded_event_id) AS decoded_event_count,
GROUP_CONCAT(te.id) AS trade_event_ids,
GROUP_CONCAT(te.decoded_event_id) AS decoded_event_ids
FROM kb_trade_events te
LEFT JOIN kb_pairs pair ON pair.id = te.pair_id
LEFT JOIN kb_pools p ON p.id = pair.pool_id
LEFT JOIN kb_dexes d ON d.id = p.dex_id
GROUP BY
te.signature,
te.pair_id,
p.address,
d.code
HAVING COUNT(te.id) > 1
ORDER BY trade_event_count DESC, te.signature, te.pair_id
LIMIT ?
"#,
)
.bind(limit)
.fetch_all(pool)
.await;
let rows = match rows_result {
Ok(rows) => rows,
Err(error) => {
return Err(crate::KbError::Db(format!(
"cannot list multi-trade signature/pair diagnostic samples on sqlite: {}",
error
)));
},
};
let mut samples = std::vec::Vec::new();
for row in rows {
samples.push(crate::KbLocalMultiTradeSignaturePairDiagnosticSampleDto {
signature: row.signature,
pair_id: row.pair_id,
pool_address: row.pool_address,
dex_code: row.dex_code,
trade_event_count: row.trade_event_count,
decoded_event_count: row.decoded_event_count,
trade_event_ids: row.trade_event_ids,
decoded_event_ids: row.decoded_event_ids,
});
}
return Ok(samples);
},
}
}
/// Lists samples of pairs without trade events.
pub async fn list_local_pair_without_trade_diagnostic_samples(
database: &crate::KbDatabase,
limit: i64,
) -> Result<std::vec::Vec<crate::KbLocalPairGapDiagnosticSampleDto>, crate::KbError> {
return kb_list_local_pair_gap_diagnostic_samples(database, limit, true).await;
}
/// Lists samples of pairs without candles.
pub async fn list_local_pair_without_candle_diagnostic_samples(
database: &crate::KbDatabase,
limit: i64,
) -> Result<std::vec::Vec<crate::KbLocalPairGapDiagnosticSampleDto>, crate::KbError> {
return kb_list_local_pair_gap_diagnostic_samples(database, limit, false).await;
}
async fn kb_list_local_pair_gap_diagnostic_samples(
database: &crate::KbDatabase,
limit: i64,
without_trade: bool,
) -> Result<std::vec::Vec<crate::KbLocalPairGapDiagnosticSampleDto>, crate::KbError> {
match database.connection() {
crate::KbDatabaseConnection::Sqlite(pool) => {
let having_clause = if without_trade {
"HAVING COUNT(DISTINCT te.id) = 0"
} else {
"HAVING COUNT(DISTINCT pc.bucket_start_unix || ':' || pc.timeframe_seconds) = 0"
};
let sql = format!(
r#"
SELECT
pair.id AS pair_id,
p.address AS pool_address,
d.code AS dex_code,
base_token.mint AS base_mint,
base_token.symbol AS base_symbol,
quote_token.mint AS quote_mint,
quote_token.symbol AS quote_symbol,
pair.symbol AS pair_symbol,
COUNT(DISTINCT dde.id) AS decoded_event_count,
COUNT(DISTINCT CASE WHEN json_extract(dde.payload_json, '$.tradeCandidate') = 1 THEN dde.id END) AS decoded_trade_candidate_count,
COUNT(DISTINCT te.id) AS trade_event_count,
COUNT(DISTINCT pc.bucket_start_unix || ':' || pc.timeframe_seconds) AS pair_candle_count
FROM kb_pairs pair
JOIN kb_pools p ON p.id = pair.pool_id
JOIN kb_dexes d ON d.id = p.dex_id
JOIN kb_tokens base_token ON base_token.id = pair.base_token_id
JOIN kb_tokens quote_token ON quote_token.id = pair.quote_token_id
LEFT JOIN kb_dex_decoded_events dde ON dde.pool_account = p.address
LEFT JOIN kb_trade_events te ON te.pair_id = pair.id
LEFT JOIN kb_pair_candles pc ON pc.pair_id = pair.id
GROUP BY
pair.id,
p.address,
d.code,
base_token.mint,
base_token.symbol,
quote_token.mint,
quote_token.symbol,
pair.symbol
{}
ORDER BY decoded_trade_candidate_count DESC, pair.id
LIMIT ?
"#,
having_clause
);
let rows_result = sqlx::query_as::<
sqlx::Sqlite,
crate::KbLocalPairGapDiagnosticSampleRow,
>(sql.as_str())
.bind(limit)
.fetch_all(pool)
.await;
let rows = match rows_result {
Ok(rows) => rows,
Err(error) => {
return Err(crate::KbError::Db(format!(
"cannot list pair gap diagnostic samples on sqlite: {}",
error
)));
},
};
let mut samples = std::vec::Vec::new();
for row in rows {
samples.push(crate::KbLocalPairGapDiagnosticSampleDto {
pair_id: row.pair_id,
pool_address: row.pool_address,
dex_code: row.dex_code,
base_mint: row.base_mint,
base_symbol: row.base_symbol,
quote_mint: row.quote_mint,
quote_symbol: row.quote_symbol,
pair_symbol: row.pair_symbol,
decoded_event_count: row.decoded_event_count,
decoded_trade_candidate_count: row.decoded_trade_candidate_count,
trade_event_count: row.trade_event_count,
pair_candle_count: row.pair_candle_count,
});
}
return Ok(samples);
},
}
}
fn kb_sqlite_i64_to_bool(value: i64) -> bool {
return value != 0;
}

View File

@@ -20,6 +20,7 @@ mod http_client;
mod http_pool;
mod json_rpc_ws;
mod launch_origin;
mod local_pipeline_diagnostics;
mod local_pipeline_replay;
mod pair_analytic_signal;
mod pair_candle_aggregation;
@@ -86,6 +87,23 @@ pub use db::KbLaunchSurfaceKeyEntity;
pub use db::KbLiquidityEventDto;
pub use db::KbLiquidityEventEntity;
pub use db::KbLiquidityEventKind;
pub use db::KbLocalDecodedEventDiagnosticSummaryDto;
pub(crate) use db::KbLocalDecodedEventDiagnosticSummaryRow;
pub use db::KbLocalDexDiagnosticSummaryDto;
pub(crate) use db::KbLocalDexDiagnosticSummaryRow;
pub use db::KbLocalDuplicateDecodedEventTradeDiagnosticSampleDto;
pub(crate) use db::KbLocalDuplicateDecodedEventTradeDiagnosticSampleRow;
pub use db::KbLocalMissingTradeEventDiagnosticSampleDto;
pub(crate) use db::KbLocalMissingTradeEventDiagnosticSampleRow;
pub use db::KbLocalMultiTradeSignaturePairDiagnosticSampleDto;
pub(crate) use db::KbLocalMultiTradeSignaturePairDiagnosticSampleRow;
pub use db::KbLocalPairDiagnosticSummaryDto;
pub(crate) use db::KbLocalPairDiagnosticSummaryRow;
pub use db::KbLocalPairGapDiagnosticSampleDto;
pub(crate) use db::KbLocalPairGapDiagnosticSampleRow;
pub use db::KbLocalPipelineDiagnosticCountersDto;
pub(crate) use db::KbLocalPipelineDiagnosticCountersRow;
pub use db::KbLocalPipelineDiagnosticSummaryDto;
pub use db::KbObservationSourceKind;
pub use db::KbObservedTokenDto;
pub use db::KbObservedTokenEntity;
@@ -129,6 +147,7 @@ pub use db::KbWalletHoldingEntity;
pub use db::KbWalletParticipationDto;
pub use db::KbWalletParticipationEntity;
pub use db::delete_chain_instructions_by_transaction_id;
pub use db::get_chain_instruction_by_id;
pub use db::get_chain_slot;
pub use db::get_chain_transaction_by_signature;
pub use db::get_db_metadata;
@@ -140,6 +159,7 @@ pub use db::get_latest_pump_fun_create_payload_by_mint;
pub use db::get_launch_attribution_by_decoded_event_id;
pub use db::get_launch_surface_by_code;
pub use db::get_launch_surface_key_by_match;
pub use db::get_local_pipeline_diagnostic_counters;
pub use db::get_observed_token_by_mint;
pub use db::get_pair_analytic_signal_by_key;
pub use db::get_pair_by_pool_id;
@@ -168,6 +188,14 @@ pub use db::list_known_ws_endpoints;
pub use db::list_launch_attributions_by_pool_id;
pub use db::list_launch_surface_keys_by_surface_id;
pub use db::list_launch_surfaces;
pub use db::list_local_decoded_event_diagnostic_summaries;
pub use db::list_local_dex_diagnostic_summaries;
pub use db::list_local_duplicate_decoded_event_trade_diagnostic_samples;
pub use db::list_local_missing_trade_event_diagnostic_samples;
pub use db::list_local_multi_trade_signature_pair_diagnostic_samples;
pub use db::list_local_pair_diagnostic_summaries;
pub use db::list_local_pair_without_candle_diagnostic_samples;
pub use db::list_local_pair_without_trade_diagnostic_samples;
pub use db::list_observed_tokens;
pub use db::list_pair_analytic_signals_by_pair_id;
pub use db::list_pair_candles_by_pair_and_timeframe;
@@ -315,6 +343,7 @@ pub use json_rpc_ws::parse_kb_json_rpc_ws_incoming_text;
pub use json_rpc_ws::parse_kb_json_rpc_ws_incoming_value;
pub use launch_origin::KbLaunchAttributionResult;
pub use launch_origin::KbLaunchOriginService;
pub use local_pipeline_diagnostics::KbLocalPipelineDiagnosticsService;
pub use local_pipeline_replay::KbLocalPipelineReplayConfig;
pub use local_pipeline_replay::KbLocalPipelineReplayResult;
pub use local_pipeline_replay::KbLocalPipelineReplayService;

View File

@@ -0,0 +1,145 @@
// file: kb_lib/src/local_pipeline_diagnostics.rs
//! Local pipeline diagnostics service.
/// Local pipeline diagnostics service.
#[derive(Debug, Clone)]
pub struct KbLocalPipelineDiagnosticsService {
database: std::sync::Arc<crate::KbDatabase>,
}
impl KbLocalPipelineDiagnosticsService {
/// Creates a new local pipeline diagnostics service.
pub fn new(database: std::sync::Arc<crate::KbDatabase>) -> Self {
return Self { database };
}
/// Builds a local pipeline diagnostics summary from already persisted data.
pub async fn diagnose(
&self,
) -> Result<crate::KbLocalPipelineDiagnosticSummaryDto, crate::KbError> {
let sample_limit = 25_i64;
let counters_result =
crate::get_local_pipeline_diagnostic_counters(self.database.as_ref()).await;
let counters = match counters_result {
Ok(counters) => counters,
Err(error) => return Err(error),
};
let dex_summaries_result =
crate::list_local_dex_diagnostic_summaries(self.database.as_ref()).await;
let dex_summaries = match dex_summaries_result {
Ok(dex_summaries) => dex_summaries,
Err(error) => return Err(error),
};
let pair_summaries_result =
crate::list_local_pair_diagnostic_summaries(self.database.as_ref()).await;
let pair_summaries = match pair_summaries_result {
Ok(pair_summaries) => pair_summaries,
Err(error) => return Err(error),
};
let decoded_event_summaries_result =
crate::list_local_decoded_event_diagnostic_summaries(self.database.as_ref()).await;
let decoded_event_summaries = match decoded_event_summaries_result {
Ok(decoded_event_summaries) => decoded_event_summaries,
Err(error) => return Err(error),
};
let missing_trade_event_samples_result =
crate::list_local_missing_trade_event_diagnostic_samples(
self.database.as_ref(),
sample_limit,
)
.await;
let missing_trade_event_samples = match missing_trade_event_samples_result {
Ok(samples) => samples,
Err(error) => return Err(error),
};
let duplicate_decoded_event_trade_samples_result =
crate::list_local_duplicate_decoded_event_trade_diagnostic_samples(
self.database.as_ref(),
sample_limit,
)
.await;
let duplicate_decoded_event_trade_samples =
match duplicate_decoded_event_trade_samples_result {
Ok(samples) => samples,
Err(error) => return Err(error),
};
let multi_trade_signature_pair_samples_result =
crate::list_local_multi_trade_signature_pair_diagnostic_samples(
self.database.as_ref(),
sample_limit,
)
.await;
let multi_trade_signature_pair_samples = match multi_trade_signature_pair_samples_result {
Ok(samples) => samples,
Err(error) => return Err(error),
};
let pair_without_trade_samples_result =
crate::list_local_pair_without_trade_diagnostic_samples(
self.database.as_ref(),
sample_limit,
)
.await;
let pair_without_trade_samples = match pair_without_trade_samples_result {
Ok(samples) => samples,
Err(error) => return Err(error),
};
let pair_without_candle_samples_result =
crate::list_local_pair_without_candle_diagnostic_samples(
self.database.as_ref(),
sample_limit,
)
.await;
let pair_without_candle_samples = match pair_without_candle_samples_result {
Ok(samples) => samples,
Err(error) => return Err(error),
};
let blocking_issue_count = counters.actionable_missing_trade_event_count
+ counters.invalid_trade_event_count
+ counters.duplicate_decoded_event_trade_count
+ counters.duplicate_candle_bucket_count;
let diagnostics_clean = blocking_issue_count == 0;
return Ok(crate::KbLocalPipelineDiagnosticSummaryDto {
transaction_count: counters.transaction_count,
ok_transaction_count: counters.ok_transaction_count,
failed_transaction_count: counters.failed_transaction_count,
decoded_event_count: counters.decoded_event_count,
decoded_trade_candidate_count: counters.decoded_trade_candidate_count,
decoded_candle_candidate_count: counters.decoded_candle_candidate_count,
diagnostics_clean,
blocking_issue_count,
missing_trade_event_count: counters.missing_trade_event_count,
decoded_trade_candidate_without_trade_event_count: counters
.decoded_trade_candidate_without_trade_event_count,
decoded_trade_candidate_without_trade_event_on_ok_transaction_count: counters
.decoded_trade_candidate_without_trade_event_on_ok_transaction_count,
decoded_trade_candidate_without_trade_event_on_failed_transaction_count: counters
.decoded_trade_candidate_without_trade_event_on_failed_transaction_count,
actionable_missing_trade_event_count: counters.actionable_missing_trade_event_count,
ignored_failed_transaction_trade_candidate_count: counters
.ignored_failed_transaction_trade_candidate_count,
decoded_trade_candidate_without_amount_payload_count: counters
.decoded_trade_candidate_without_amount_payload_count,
trade_event_count: counters.trade_event_count,
invalid_trade_event_count: counters.invalid_trade_event_count,
pair_candle_count: counters.pair_candle_count,
duplicate_decoded_event_trade_count: counters.duplicate_decoded_event_trade_count,
multi_trade_signature_pair_count: counters.multi_trade_signature_pair_count,
duplicate_candle_bucket_count: counters.duplicate_candle_bucket_count,
token_count: counters.token_count,
token_metadata_missing_count: counters.token_metadata_missing_count,
pool_count: counters.pool_count,
pair_count: counters.pair_count,
pair_without_trade_count: counters.pair_without_trade_count,
pair_without_candle_count: counters.pair_without_candle_count,
dex_summaries,
pair_summaries,
decoded_event_summaries,
missing_trade_event_samples,
duplicate_decoded_event_trade_samples,
multi_trade_signature_pair_samples,
pair_without_trade_samples,
pair_without_candle_samples,
});
}
}

View File

@@ -52,10 +52,17 @@ pub struct KbLocalPipelineReplayResult {
pub detection_count: usize,
/// Total trade aggregation results returned by replayed aggregation calls.
pub trade_event_count: usize,
/// Total candle aggregation results returned by replayed candle calls.
pub pair_candle_count: usize,
/// Total analytic signal results returned by replayed analytic calls.
pub analytic_signal_count: usize,
/// Total candle upsert results returned by replayed candle calls.
///
/// This is a replay write/result counter, not the number of distinct rows
/// currently persisted in `kb_pair_candles`. Use local diagnostics for the
/// persisted row count.
pub pair_candle_upsert_count: usize,
/// Total analytic signal upsert results returned by replayed analytic calls.
///
/// This is a replay write/result counter, not the number of distinct rows
/// currently persisted in the analytic signal table.
pub analytic_signal_upsert_count: usize,
/// Number of token metadata rows updated after replay.
pub token_metadata_updated_count: usize,
/// Number of pair symbols updated after replay.
@@ -177,7 +184,7 @@ impl KbLocalPipelineReplayService {
.await;
match candle_result {
Ok(candle_results) => {
result.pair_candle_count += candle_results.len();
result.pair_candle_upsert_count += candle_results.len();
},
Err(error) => {
result.pair_candle_error_count += 1;
@@ -192,7 +199,7 @@ impl KbLocalPipelineReplayService {
pair_analytic_signal.record_transaction_by_signature(signature.as_str()).await;
match analytic_result {
Ok(analytic_results) => {
result.analytic_signal_count += analytic_results.len();
result.analytic_signal_upsert_count += analytic_results.len();
},
Err(error) => {
result.analytic_signal_error_count += 1;

View File

@@ -138,6 +138,20 @@ impl KbTradeAggregationService {
)));
},
};
let base_token_result =
crate::get_token_by_id(self.database.as_ref(), pair.base_token_id).await;
let base_token_decimals = match base_token_result {
Ok(Some(token)) => token.decimals,
Ok(None) => None,
Err(error) => return Err(error),
};
let quote_token_result =
crate::get_token_by_id(self.database.as_ref(), pair.quote_token_id).await;
let quote_token_decimals = match quote_token_result {
Ok(Some(token)) => token.decimals,
Ok(None) => None,
Err(error) => return Err(error),
};
let pool_tokens_result =
crate::list_pool_tokens_by_pool_id(self.database.as_ref(), pool_id).await;
let pool_tokens = match pool_tokens_result {
@@ -247,10 +261,80 @@ impl KbTradeAggregationService {
price_quote_per_base = inferred.2;
}
}
if decoded_event.event_kind.starts_with("raydium_cpmm.")
if (decoded_event.event_kind.starts_with("raydium_cpmm.")
|| decoded_event.event_kind.starts_with("raydium_clmm."))
&& (base_amount_raw.is_none()
|| quote_amount_raw.is_none()
|| price_quote_per_base.is_none())
{
let decoded_instruction_index = match decoded_event.instruction_id {
Some(instruction_id) => {
let instruction_result = crate::get_chain_instruction_by_id(
self.database.as_ref(),
instruction_id,
)
.await;
let instruction_option = match instruction_result {
Ok(instruction_option) => instruction_option,
Err(error) => return Err(error),
};
match instruction_option {
Some(instruction) => Some(instruction.instruction_index),
None => None,
}
},
None => None,
};
let payload_input_vault_address =
kb_extract_string_by_candidate_keys(&payload, &["inputVault", "input_vault"]);
let payload_output_vault_address =
kb_extract_string_by_candidate_keys(&payload, &["outputVault", "output_vault"]);
let payload_input_token_account = kb_extract_string_by_candidate_keys(
&payload,
&["inputTokenAccount", "input_token_account"],
);
let payload_output_token_account = kb_extract_string_by_candidate_keys(
&payload,
&["outputTokenAccount", "output_token_account"],
);
let payload_base_vault_address =
kb_extract_string_by_candidate_keys(&payload, &["baseVault", "base_vault"]);
let payload_quote_vault_address =
kb_extract_string_by_candidate_keys(&payload, &["quoteVault", "quote_vault"]);
let effective_base_vault_address = match base_vault_address.as_deref() {
Some(base_vault_address) => Some(base_vault_address),
None => payload_base_vault_address.as_deref(),
};
let effective_quote_vault_address = match quote_vault_address.as_deref() {
Some(quote_vault_address) => Some(quote_vault_address),
None => payload_quote_vault_address.as_deref(),
};
let inferred_result = kb_extract_trade_amounts_from_instruction_token_transfers(
transaction.meta_json.as_deref(),
decoded_instruction_index,
payload_input_vault_address.as_deref(),
payload_output_vault_address.as_deref(),
payload_input_token_account.as_deref(),
payload_output_token_account.as_deref(),
effective_base_vault_address,
effective_quote_vault_address,
);
let inferred = match inferred_result {
Ok(inferred) => inferred,
Err(error) => return Err(error),
};
if base_amount_raw.is_none() {
base_amount_raw = inferred.0;
}
if quote_amount_raw.is_none() {
quote_amount_raw = inferred.1;
}
if price_quote_per_base.is_none() {
price_quote_per_base = inferred.2;
}
}
if decoded_event.event_kind.starts_with("raydium_cpmm.")
&& (base_amount_raw.is_none() || quote_amount_raw.is_none())
{
let inferred_result = kb_extract_trade_amounts_from_vault_balance_deltas(
transaction.transaction_json.as_str(),
@@ -273,9 +357,7 @@ impl KbTradeAggregationService {
}
}
if decoded_event.event_kind.starts_with("raydium_clmm.")
&& (base_amount_raw.is_none()
|| quote_amount_raw.is_none()
|| price_quote_per_base.is_none())
&& (base_amount_raw.is_none() || quote_amount_raw.is_none())
{
let inferred_result = kb_extract_trade_amounts_from_vault_balance_deltas(
transaction.transaction_json.as_str(),
@@ -297,6 +379,15 @@ impl KbTradeAggregationService {
price_quote_per_base = inferred.2;
}
}
if price_quote_per_base.is_none() {
price_quote_per_base =
kb_compute_price_quote_per_base_from_raw_amounts_with_decimals(
base_amount_raw.as_deref(),
quote_amount_raw.as_deref(),
base_token_decimals,
quote_token_decimals,
);
}
if price_quote_per_base.is_none() {
price_quote_per_base = kb_compute_price_quote_per_base_with_decimals(
transaction.meta_json.as_deref(),
@@ -884,6 +975,182 @@ fn kb_extract_trade_amounts_from_vault_balance_deltas(
return Ok((base_amount_raw, quote_amount_raw, price_quote_per_base));
}
fn kb_extract_trade_amounts_from_instruction_token_transfers(
meta_json: std::option::Option<&str>,
instruction_index: std::option::Option<u32>,
input_vault_address: std::option::Option<&str>,
output_vault_address: std::option::Option<&str>,
input_token_account: std::option::Option<&str>,
output_token_account: std::option::Option<&str>,
base_vault_address: std::option::Option<&str>,
quote_vault_address: std::option::Option<&str>,
) -> Result<KbExtractedTradeAmounts, crate::KbError> {
let meta_json = match meta_json {
Some(meta_json) => meta_json,
None => return Ok((None, None, None)),
};
let instruction_index = match instruction_index {
Some(instruction_index) => u64::from(instruction_index),
None => return Ok((None, None, None)),
};
let input_vault_address = match input_vault_address {
Some(input_vault_address) => input_vault_address.trim(),
None => return Ok((None, None, None)),
};
let output_vault_address = match output_vault_address {
Some(output_vault_address) => output_vault_address.trim(),
None => return Ok((None, None, None)),
};
let input_token_account = match input_token_account {
Some(input_token_account) => input_token_account.trim(),
None => return Ok((None, None, None)),
};
let output_token_account = match output_token_account {
Some(output_token_account) => output_token_account.trim(),
None => return Ok((None, None, None)),
};
let base_vault_address = match base_vault_address {
Some(base_vault_address) => base_vault_address.trim(),
None => return Ok((None, None, None)),
};
let quote_vault_address = match quote_vault_address {
Some(quote_vault_address) => quote_vault_address.trim(),
None => return Ok((None, None, None)),
};
if input_vault_address.is_empty()
|| output_vault_address.is_empty()
|| input_token_account.is_empty()
|| output_token_account.is_empty()
|| base_vault_address.is_empty()
|| quote_vault_address.is_empty()
{
return Ok((None, None, None));
}
let meta_value_result = serde_json::from_str::<serde_json::Value>(meta_json);
let meta_value = match meta_value_result {
Ok(meta_value) => meta_value,
Err(error) => {
return Err(crate::KbError::Json(format!(
"cannot parse meta_json for instruction-scoped token transfer amount extraction: {}",
error
)));
},
};
let inner_groups_option =
meta_value.get("innerInstructions").and_then(|value| return value.as_array());
let inner_groups = match inner_groups_option {
Some(inner_groups) => inner_groups,
None => return Ok((None, None, None)),
};
let mut input_amount_raw = None;
let mut output_amount_raw = None;
for inner_group in inner_groups {
let group_index_option = inner_group.get("index").and_then(|value| return value.as_u64());
let group_index = match group_index_option {
Some(group_index) => group_index,
None => continue,
};
if group_index != instruction_index {
continue;
}
let instructions_option =
inner_group.get("instructions").and_then(|value| return value.as_array());
let instructions = match instructions_option {
Some(instructions) => instructions,
None => continue,
};
for instruction in instructions {
if !kb_is_spl_token_transfer_instruction(instruction) {
continue;
}
let parsed_option = instruction.get("parsed");
let parsed = match parsed_option {
Some(parsed) => parsed,
None => continue,
};
let info_option = parsed.get("info");
let info = match info_option {
Some(info) => info,
None => continue,
};
let source_option = kb_extract_string_by_candidate_keys(info, &["source"]);
let source = match source_option {
Some(source) => source,
None => continue,
};
let destination_option = kb_extract_string_by_candidate_keys(info, &["destination"]);
let destination = match destination_option {
Some(destination) => destination,
None => continue,
};
let amount_option = kb_extract_scalar_as_string_by_candidate_keys(info, &["amount"]);
let amount = match amount_option {
Some(amount) => amount,
None => continue,
};
if input_amount_raw.is_none()
&& kb_account_equals(source.as_str(), input_token_account)
&& kb_account_equals(destination.as_str(), input_vault_address)
{
input_amount_raw = Some(amount.clone());
continue;
}
if output_amount_raw.is_none()
&& kb_account_equals(source.as_str(), output_vault_address)
&& kb_account_equals(destination.as_str(), output_token_account)
{
output_amount_raw = Some(amount);
continue;
}
}
}
if input_amount_raw.is_none() && output_amount_raw.is_none() {
return Ok((None, None, None));
}
if kb_account_equals(input_vault_address, base_vault_address)
&& kb_account_equals(output_vault_address, quote_vault_address)
{
return Ok((input_amount_raw, output_amount_raw, None));
}
if kb_account_equals(input_vault_address, quote_vault_address)
&& kb_account_equals(output_vault_address, base_vault_address)
{
return Ok((output_amount_raw, input_amount_raw, None));
}
return Ok((None, None, None));
}
fn kb_is_spl_token_transfer_instruction(instruction: &serde_json::Value) -> bool {
let program_id_option = instruction.get("programId").and_then(|value| return value.as_str());
if let Some(program_id) = program_id_option {
let spl_token_program_id = crate::SPL_TOKEN_PROGRAM_ID.to_string();
let spl_token_2022_program_id = crate::SPL_TOKEN_2022_PROGRAM_ID.to_string();
if program_id != spl_token_program_id.as_str()
&& program_id != spl_token_2022_program_id.as_str()
{
return false;
}
}
let parsed_type_option = instruction
.get("parsed")
.and_then(|parsed| return parsed.get("type"))
.and_then(|value| return value.as_str());
match parsed_type_option {
Some("transfer") => return true,
Some("transferChecked") => return true,
_ => return false,
}
}
fn kb_account_equals(left: &str, right: &str) -> bool {
let left = left.trim();
let right = right.trim();
if left.is_empty() || right.is_empty() {
return false;
}
return left == right;
}
fn kb_extract_pump_fun_amounts_from_transaction(
transaction_json: &str,
meta_json: std::option::Option<&str>,
@@ -1167,6 +1434,57 @@ fn kb_compute_ui_delta_abs(
return Some(delta);
}
fn kb_compute_price_quote_per_base_from_raw_amounts_with_decimals(
base_amount_raw: std::option::Option<&str>,
quote_amount_raw: std::option::Option<&str>,
base_decimals: std::option::Option<u8>,
quote_decimals: std::option::Option<u8>,
) -> std::option::Option<f64> {
let base_decimals = match base_decimals {
Some(base_decimals) => base_decimals,
None => return None,
};
let quote_decimals = match quote_decimals {
Some(quote_decimals) => quote_decimals,
None => return None,
};
let base_amount_raw = match base_amount_raw {
Some(base_amount_raw) => base_amount_raw.trim(),
None => return None,
};
let quote_amount_raw = match quote_amount_raw {
Some(quote_amount_raw) => quote_amount_raw.trim(),
None => return None,
};
if base_amount_raw.is_empty() || quote_amount_raw.is_empty() {
return None;
}
let base_amount_result = base_amount_raw.parse::<f64>();
let base_amount = match base_amount_result {
Ok(base_amount) => base_amount,
Err(_) => return None,
};
let quote_amount_result = quote_amount_raw.parse::<f64>();
let quote_amount = match quote_amount_result {
Ok(quote_amount) => quote_amount,
Err(_) => return None,
};
if base_amount <= 0.0 || quote_amount <= 0.0 {
return None;
}
let base_scale = 10_f64.powi(i32::from(base_decimals));
let quote_scale = 10_f64.powi(i32::from(quote_decimals));
if base_scale <= 0.0 || quote_scale <= 0.0 {
return None;
}
let base_ui_amount = base_amount / base_scale;
let quote_ui_amount = quote_amount / quote_scale;
if base_ui_amount <= 0.0 || quote_ui_amount <= 0.0 {
return None;
}
return Some(quote_ui_amount / base_ui_amount);
}
fn kb_compute_price_quote_per_base_from_raw_amounts(
base_amount_raw: std::option::Option<&str>,
quote_amount_raw: std::option::Option<&str>,