v0.0.3-pre.007
This commit is contained in:
@@ -1,5 +1,5 @@
|
||||
<!-- file: docs/architecture/005-DEPENDENCY_GRAPH.md -->
|
||||
<!-- version: 4 -->
|
||||
<!-- version: 5 -->
|
||||
|
||||
# Graphe de dépendances KSP
|
||||
|
||||
@@ -460,6 +460,50 @@ Le mécanisme concret peut être channel, PostgreSQL LISTEN/NOTIFY, IPC ou broke
|
||||
|
||||
---
|
||||
|
||||
# Graphe des pipelines spécialisés
|
||||
|
||||
Les pipelines réutilisent une frontière de processing sans prendre le lifecycle worker/job.
|
||||
|
||||
```text
|
||||
ksp-pipeline-raw-ingestion-lib
|
||||
-> ksp-onchain-transport-lib
|
||||
-> ksp-store-api
|
||||
-> ksp-core-lib
|
||||
-> ksp-logging-lib
|
||||
|
||||
ksp-pipeline-core-processing-lib
|
||||
-> ksp-program-api
|
||||
-> ksp-store-api
|
||||
-> ksp-core-lib
|
||||
-> ksp-logging-lib
|
||||
|
||||
ksp-pipeline-generic-materialization-lib
|
||||
-> ksp-materializer-api
|
||||
-> ksp-store-api
|
||||
-> ksp-core-lib
|
||||
-> ksp-logging-lib
|
||||
|
||||
ksp-pipeline-domain-projection-lib
|
||||
-> ksp-materializer-api
|
||||
-> ksp-store-api
|
||||
-> ksp-core-lib
|
||||
-> ksp-logging-lib
|
||||
```
|
||||
|
||||
Interdictions :
|
||||
|
||||
```text
|
||||
pipelines -X-> ksp-store-lib
|
||||
core-processing pipeline -X-> ksp-program-lib
|
||||
materialization pipelines -X-> ksp-materializer-lib
|
||||
pipelines -X-> ksp-worker-api
|
||||
pipelines -X-> ksp-job-api
|
||||
```
|
||||
|
||||
Les workers/jobs fournissent les implémentations officielles ou externes aux pipelines.
|
||||
|
||||
---
|
||||
|
||||
# Graphe lifecycle Worker
|
||||
|
||||
```text
|
||||
@@ -508,106 +552,135 @@ Un futur orchestrateur peut utiliser workers et jobs séparément sans créer de
|
||||
|
||||
# Graphe des composants concrets d'acquisition/processing
|
||||
|
||||
Les dépendances ci-dessous décrivent la composition attendue. Les détails de processus/IPC seront approfondis plus tard.
|
||||
|
||||
## `ksp-worker-raw-retriever` — transport -> D1
|
||||
## `ksp-worker-raw-retriever`
|
||||
|
||||
```text
|
||||
ksp-worker-raw-retriever
|
||||
-> ksp-worker-api
|
||||
-> ksp-config-lib
|
||||
-> ksp-pipeline-raw-ingestion-lib
|
||||
-> ksp-onchain-transport-lib
|
||||
-> ksp-store-api
|
||||
-> ksp-store-lib
|
||||
-> ksp-config-lib
|
||||
-> ksp-logging-lib
|
||||
```
|
||||
|
||||
Responsabilité de conversion :
|
||||
Il pilote l'acquisition live, puis transmet chaque modèle transport homogène au pipeline raw ingestion.
|
||||
|
||||
```text
|
||||
transport raw model
|
||||
|
|
||||
v
|
||||
store raw persistence model
|
||||
```
|
||||
|
||||
Cette conversion appartient au worker d'acquisition, pas au transport ni au store.
|
||||
Il possède une capacité spécifique de hot reconfiguration et distingue configuration desired/effective.
|
||||
|
||||
## `ksp-job-backfill`
|
||||
|
||||
```text
|
||||
ksp-job-backfill
|
||||
-> ksp-job-api
|
||||
-> ksp-config-lib
|
||||
-> ksp-pipeline-raw-ingestion-lib
|
||||
-> ksp-onchain-transport-lib
|
||||
-> ksp-store-api
|
||||
-> ksp-store-lib
|
||||
-> ksp-config-lib
|
||||
-> ksp-logging-lib
|
||||
```
|
||||
|
||||
Il produit exactement la même famille de DTO D1 persistants et la même notification de données persistées que le worker live.
|
||||
Il pilote pagination/range/checkpoint historique puis transmet les modèles acquis au même pipeline D1 que W1.
|
||||
|
||||
## `ksp-worker-core-processor` — D1 -> D2
|
||||
## `ksp-worker-core-processor`
|
||||
|
||||
```text
|
||||
ksp-worker-core-processor
|
||||
-> ksp-worker-api
|
||||
-> ksp-program-api
|
||||
-> ksp-program-lib
|
||||
-> ksp-store-api
|
||||
-> ksp-pipeline-core-processing-lib
|
||||
-> ksp-program-lib # composition officielle
|
||||
-> ksp-store-lib
|
||||
-> ksp-config-lib
|
||||
-> ksp-logging-lib
|
||||
```
|
||||
|
||||
Il effectue la conversion explicite :
|
||||
Une composition alternative peut fournir une implémentation externe compatible avec `ksp-program-api`.
|
||||
|
||||
## `ksp-job-replay-core`
|
||||
|
||||
```text
|
||||
store raw DTO
|
||||
|
|
||||
v
|
||||
program decode input
|
||||
|
|
||||
v
|
||||
program canonical/decode output
|
||||
|
|
||||
v
|
||||
store Core DTO
|
||||
ksp-job-replay-core
|
||||
-> ksp-job-api
|
||||
-> ksp-pipeline-core-processing-lib
|
||||
-> ksp-program-lib ou implémentation compatible
|
||||
-> ksp-store-lib
|
||||
-> ksp-config-lib
|
||||
-> ksp-logging-lib
|
||||
```
|
||||
|
||||
`ksp-program-lib` ne connaît donc pas le store.
|
||||
Worker live et replay partagent donc exactement la logique D1 -> D2.
|
||||
|
||||
## `ksp-worker-generic-materializer` — D2 -> D3
|
||||
## `ksp-worker-generic-materializer`
|
||||
|
||||
```text
|
||||
ksp-worker-generic-materializer
|
||||
-> ksp-worker-api
|
||||
-> ksp-materializer-api
|
||||
-> ksp-materializer-lib
|
||||
-> ksp-store-api
|
||||
-> ksp-pipeline-generic-materialization-lib
|
||||
-> ksp-materializer-lib # composition officielle
|
||||
-> ksp-store-lib
|
||||
-> ksp-config-lib
|
||||
-> ksp-logging-lib
|
||||
```
|
||||
|
||||
Il consomme le backlog D2, appelle la capacité de matérialisation générique puis persiste le journal D3. La notification éventuelle accélère le traitement mais ne remplace pas la query de backlog.
|
||||
## `ksp-job-replay-generic-materialization`
|
||||
|
||||
## `ksp-worker-domain-projector` — D3 -> D4
|
||||
```text
|
||||
ksp-job-replay-generic-materialization
|
||||
-> ksp-job-api
|
||||
-> ksp-pipeline-generic-materialization-lib
|
||||
-> ksp-materializer-lib ou implémentation compatible
|
||||
-> ksp-store-lib
|
||||
-> ksp-config-lib
|
||||
-> ksp-logging-lib
|
||||
```
|
||||
|
||||
## `ksp-worker-domain-projector`
|
||||
|
||||
```text
|
||||
ksp-worker-domain-projector
|
||||
-> ksp-worker-api
|
||||
-> ksp-materializer-api
|
||||
-> ksp-materializer-lib
|
||||
-> ksp-store-api
|
||||
-> ksp-pipeline-domain-projection-lib
|
||||
-> ksp-materializer-lib # projectors officiels
|
||||
-> ksp-store-lib
|
||||
-> ksp-config-lib
|
||||
-> ksp-logging-lib
|
||||
```
|
||||
|
||||
Son nom reste provisoire.
|
||||
## `ksp-job-replay-domain-projection`
|
||||
|
||||
Il est propriétaire de la composition entre D3, la projection/materialisation spécialisée et les DTO D4 persistants ; `ksp-materializer-lib` reste indépendant du backend. D4 est organisé par faits canoniques plutôt que par familles de tables propres aux protocoles.
|
||||
```text
|
||||
ksp-job-replay-domain-projection
|
||||
-> ksp-job-api
|
||||
-> ksp-pipeline-domain-projection-lib
|
||||
-> ksp-materializer-lib ou implémentation compatible
|
||||
-> ksp-store-lib
|
||||
-> ksp-config-lib
|
||||
-> ksp-logging-lib
|
||||
```
|
||||
|
||||
## Backlog / notification
|
||||
|
||||
Tous les workers de processing :
|
||||
|
||||
```text
|
||||
LISTEN/NOTIFY wake-up
|
||||
+
|
||||
periodic polling
|
||||
|
|
||||
v
|
||||
Store backlog query
|
||||
|
|
||||
v
|
||||
claim/lease bounded batch
|
||||
|
|
||||
v
|
||||
pipeline
|
||||
|
|
||||
v
|
||||
outputs + processing outcome
|
||||
```
|
||||
|
||||
La notification ne remplace jamais le Store.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user