From a902fc37f3a3a8af6a9ac127ab8a358f81d5e216 Mon Sep 17 00:00:00 2001 From: pranitkhamkar Date: Fri, 17 Oct 2025 20:08:45 +0530 Subject: [PATCH] docs: update mkdocs navigation structure for product and system documentation --- .../AcknowledgmentEventPattern.md | 138 +++++ .../dhanman-messaging-architecture.md | 562 ++++++++++++++++++ mkdocs.yml | 232 ++++---- 3 files changed, 817 insertions(+), 115 deletions(-) create mode 100644 docs/system/architecture/AcknowledgmentEventPattern.md create mode 100644 docs/system/architecture/dhanman-messaging-architecture.md diff --git a/docs/system/architecture/AcknowledgmentEventPattern.md b/docs/system/architecture/AcknowledgmentEventPattern.md new file mode 100644 index 0000000..0793dc2 --- /dev/null +++ b/docs/system/architecture/AcknowledgmentEventPattern.md @@ -0,0 +1,138 @@ +# 🧭 Dhanman Messaging Design: Acknowledgment Event Pattern + +## **Objective** + +Enable asynchronous, reliable communication between services using **event-based acknowledgments** instead of blocking request–response. +Each service that initiates a command will later receive a **targeted acknowledgment event** confirming completion and returning necessary identifiers. + +--- + +## 🧩 **1. Design Philosophy** + +- **Fire-and-Forget Commands:** Sales, Purchase, Payroll, and Community send commands to Common or other shared modules. +- **Targeted Acknowledgment Events:** The receiver (e.g., Common) publishes back an acknowledgment event containing IDs or status for the initiating entity. +- **Point-to-Point Routing:** Each acknowledgment event is delivered only to its respective source service, avoiding fanout noise. +- **Guaranteed Delivery:** All acknowledgment events use durable queues bound to a **direct exchange** with explicit routing keys. + +--- + +## 🔁 **2. High-Level Flow Example** + +```mermaid +sequenceDiagram + participant Sales as Dhanman.Sales + participant Common as Dhanman.Common + + Sales->>Common: 🔶 CreateTransactionCommand (InvoiceId, Amount) + Common->>CommonDB: 💾 Create Transaction + Common-->>Sales: 🔵 TransactionPostedForInvoiceEvent (InvoiceId, TransactionId) + Sales->>SalesDB: 🧾 Update Invoice (TransactionId, Posted = true) +``` + +--- + +## 🧱 **3. Exchange and Queue Strategy** + +| **Exchange** | **Type** | **Publisher** | **Routing Key Pattern** | **Consumer Service** | **Queue Example** | +| ------------------- | ---------- | ------------- | ----------------------- | -------------------- | ------------------------------------------------------------------ | +| `common.ack.events` | **direct** | Common | `invoice.posted` | Sales | `sales.invoice_posted.queue` | +| `common.ack.events` | **direct** | Common | `bill.posted` | Purchase | `purchase.bill_posted.queue` | +| `common.ack.events` | **direct** | Common | `salary.posted` | Payroll | `payroll.salary_posted.queue` | +| `common.ack.events` | **direct** | Common | `customer.created` | Sales / Community | `sales.customer_created.queue`, `community.customer_created.queue` | +| `common.ack.events` | **direct** | Common | `vendor.created` | Purchase | `purchase.vendor_created.queue` | +| `common.ack.events` | **direct** | Common | `employee.created` | Payroll | `payroll.employee_created.queue` | + +--- + +## 🧩 **4. Complete List of Acknowledgment Events** + +| **Origin Service (Sender)** | **Target Publisher (Responder)** | **Command Sent** | **Acknowledgment Event** | **Payload Fields (Suggested)** | **Consumer Queue** | +| --------------------------- | -------------------------------- | ---------------------------------------- | ----------------------------------------- | --------------------------------------------------------------------------------- | ------------------------------------- | +| **Sales** | Common | `CreateBasicCustomerCommand` | `CustomerCreatedForSalesEvent` | `CustomerId`, `CompanyId`, `OrganizationId`, `CorrelationId`, `CreatedOnUtc` | `sales.customer_created.queue` | +| | Common | `CreateTransactionCommand` | `TransactionPostedForInvoiceEvent` | `InvoiceId`, `TransactionId`, `CompanyId`, `OrganizationId`, `PostedOnUtc` | `sales.invoice_posted.queue` | +| | Common | `CreateInvoicePaymentCommand` | `TransactionPostedForInvoicePaymentEvent` | `InvoicePaymentId`, `TransactionId`, `CompanyId`, `OrganizationId`, `PostedOnUtc` | `sales.invoice_payment_posted.queue` | +| **Purchase** | Common | `CreateBasicVendorCommand` | `VendorCreatedForPurchaseEvent` | `VendorId`, `CompanyId`, `OrganizationId`, `CreatedOnUtc` | `purchase.vendor_created.queue` | +| | Common | `CreateTransactionCommand` | `TransactionPostedForBillEvent` | `BillId`, `TransactionId`, `CompanyId`, `OrganizationId`, `PostedOnUtc` | `purchase.bill_posted.queue` | +| | Common | `CreateBillPaymentCommand` | `TransactionPostedForBillPaymentEvent` | `BillPaymentId`, `TransactionId`, `CompanyId`, `OrganizationId`, `PostedOnUtc` | `purchase.bill_payment_posted.queue` | +| **Payroll** | Common | `CreateBasicEmployeeCommand` | `EmployeeCreatedForPayrollEvent` | `EmployeeId`, `CompanyId`, `OrganizationId`, `CreatedOnUtc` | `payroll.employee_created.queue` | +| | Common | `CreateTransactionCommand` | `TransactionPostedForSalaryEvent` | `SalaryId`, `TransactionId`, `CompanyId`, `OrganizationId`, `PostedOnUtc` | `payroll.salary_posted.queue` | +| | Common | `CreateSalaryPaymentCommand` | `TransactionPostedForSalaryPaymentEvent` | `SalaryPaymentId`, `TransactionId`, `CompanyId`, `OrganizationId`, `PostedOnUtc` | `payroll.salary_payment_posted.queue` | +| **Community** | Common | `CreateBasicCustomerCommand` | `CustomerCreatedForCommunityEvent` | `CustomerId`, `UnitId`, `CompanyId`, `OrganizationId`, `CreatedOnUtc` | `community.customer_created.queue` | +| **Common (reverse ack)** | Community | (Triggered when Unit creates a Customer) | `CustomerLinkedToUnitEvent` | `UnitId`, `CustomerId`, `CompanyId`, `OrganizationId`, `LinkedOnUtc` | `community.customer_linked.queue` | + +--- + +## ⚙️ **5. MassTransit Configuration Example** + +### **Common.Api Publisher** + +```csharp +await _eventPublisher.PublishAsync(new TransactionPostedForInvoiceEvent +{ + InvoiceId = command.InvoiceId, + TransactionId = transaction.Id, + CompanyId = command.CompanyId, + OrganizationId = command.OrganizationId, + PostedOnUtc = DateTime.UtcNow +}, context => +{ + context.SetRoutingKey("invoice.posted"); +}); +``` + +### **Sales.Api Consumer** + +```csharp +cfg.ReceiveEndpoint("sales.invoice_posted.queue", e => +{ + e.ConfigureConsumeTopology = false; + e.Bind("common.ack.events", x => + { + x.RoutingKey = "invoice.posted"; + }); + e.ConfigureConsumer(context); +}); +``` + +--- + +## 🧠 **6. Why Use Direct Exchange** + +✔ Only specific consumers receive relevant messages +✔ Reduces unnecessary message traffic +✔ Keeps event routing explicit and predictable +✔ Easy to extend — just add a new routing key and queue binding for another event type + +--- + +## 🪶 **7. Schema Example (Event DTO)** + +```csharp +public record TransactionPostedForInvoiceEvent( + Guid InvoiceId, + long TransactionId, + Guid CompanyId, + Guid OrganizationId, + DateTime PostedOnUtc, + Guid CorrelationId); +``` + +All acknowledgment events follow a consistent convention: + +- Include **CorrelationId** from originating command +- Include **EntityId** (InvoiceId, BillId, etc.) +- Include **TransactionId** +- Include **OrganizationId**, **CompanyId**, and **Timestamp** + +--- + +## 🧩 **8. Summary Checklist** + +| ✅ Goal | 🧩 Implementation | +| -------------------------------- | ------------------------------------------------------------------ | +| Maintain async non-blocking flow | Fire commands, listen for acknowledgment events | +| Event naming consistency | Use `TransactionPostedForXEvent`, `CustomerCreatedForXEvent`, etc. | +| Routing isolation | Use direct exchange with service-specific routing keys | +| Observability | Include CorrelationId in every message | +| Reliability | Use Outbox pattern for event publishing | +| Maintainability | Group all acknowledgments under `common.ack.events` exchange | diff --git a/docs/system/architecture/dhanman-messaging-architecture.md b/docs/system/architecture/dhanman-messaging-architecture.md new file mode 100644 index 0000000..47cc6dc --- /dev/null +++ b/docs/system/architecture/dhanman-messaging-architecture.md @@ -0,0 +1,562 @@ +# 🧩 Dhanman Messaging Architecture — RabbitMQ + MassTransit (Detailed) + +**Version:** 2.0 +**Last Updated:** 2025-10-17 (IST) +**Maintainer:** Dhanman Engineering Team (B2A Technologies Pvt. Ltd.) + +--- + +## 0) At-a-glance + +- **Broker:** RabbitMQ +- **Library:** MassTransit (.NET) +- **Message Types:** **Commands** (one-to-one, `direct`) and **Events** (broadcast, `fanout`) +- **Participants:** Common (financial core), Sales, Purchase, Payroll, Inventory, Community _(active)_; Payment (no DB) & Document (own DB) _(currently passive for messaging)_ +- **Error Handling:** MassTransit `_skipped`, `_error`, `_delay` queues; DLX/DLQ policies; retries & scheduled redelivery +- **Environments:** prod / qa per **virtual host** (`/prod`, `/qa`) with separate credentials and permissions + +--- + +## 1) Services & Ownership + +| # | Service | Owns | Messaging Role | +| --- | --------------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | **Common** (Financial Core) | Organizations, Companies, Users, Chart of Accounts, Ledger/Transactions, Reports | **Publishes events** (Org/Company/User created/updated/deleted), **Consumes commands** (CreateTransaction, CreateCustomer, CreateVendor, CreateEmployee) | +| 2 | **Sales** | Customers, Invoices, Receipts, Credit/Debit Notes | **Publishes commands** to Common (CreateTransaction, CreateCustomer); **Consumes events** from Common and Community | +| 3 | **Purchase** | Vendors, Bills, Payments, Credit/Debit Notes | **Publishes commands** to Common (CreateTransaction, CreateVendor); **Consumes events** from Common | +| 4 | **Payroll** | Employees, Salaries (posting & payment) | **Publishes commands** to Common (CreateTransaction, CreateEmployee, CreateUser); **Consumes events** from Common | +| 5 | **Inventory** | Items/Stock (valuation future) | **Consumes events** from Common (and Purchase vendor if needed) | +| 6 | **Community** | Units, Members | **Publishes commands** to Sales/Common (CreateCustomer, CreateUser); **Consumes events** from Common | +| 7 | Payment _(no DB)_ | Gateway orchestration | (future) publish `PaymentVerifiedEvent` | +| 8 | Document _(own DB)_ | Documents/Blobs | (future) publish `DocumentUploadedEvent` | + +--- + +## 2) Commands vs Events (recap) + +| Aspect | **Command** | **Event** | +| --------- | ----------------------------------- | ----------------------------------------- | +| Meaning | “Do X” (imperative) | “X happened” (fact) | +| Direction | One-to-one | One-to-many | +| Exchange | `direct` | `fanout` | +| Coupling | Higher (explicit target) | Lower (broadcast) | +| Example | `CreateTransactionCommand` → Common | `OrganizationCreatedEvent` → all services | + +--- + +## 3) RabbitMQ Primer (with Dhanman conventions) + +### 3.1 Virtual Hosts (vhosts) + +A **virtual host** is a namespaced RabbitMQ instance: its own users/permissions, exchanges, queues, and bindings. +**Why:** strict isolation between **environments** and/or **tenants**. + +- **Recommended:** + - `/prod` for production + - `/qa` for QA +- Create per-vhost users with minimum permissions. +- MassTransit/clients connect to the right vhost via connection string (e.g., `amqp://user:pass@host:5672/qa`). + +### 3.2 Exchanges, Queues, Bindings, Routing Keys + +- **Exchange types used:** + - **`direct`** for **Commands** (point-to-point). + - **`fanout`** for **Events** (broadcast). + - _(Optionally `topic` later for routing-key-based patterns)_ +- **Queues** are where consumers read messages. +- **Bindings** connect exchanges to queues (possibly with routing keys). + +### 3.3 Durability & Persistence + +- Set exchanges/queues to **durable**; publish messages with **delivery mode = persistent** (MassTransit does this by default in production profiles). + +### 3.4 Acknowledgements & Prefetch + +- Consumers **ack** on success; **nack/reject** on failure → triggers retry or DLQ. +- **Prefetch** (QoS) controls parallelism; tune per consumer (MassTransit: `PrefetchCount`). + +### 3.5 Dead-Lettering (DLX/DLQ) + +- Configure a **Dead Letter Exchange (DLX)** at queue level. +- On reject/expired/limit-exceeded, messages are routed to **DLQ** (dead-letter queue). +- MassTransit also creates `_error` and `_skipped` queues per endpoint. + +### 3.6 Policies & HA + +- Use policies for DLX/TTL on a namespace of queues. +- Consider quorum/mirrored queues for HA (depends on cluster topology and throughput). + +--- + +## 4) Naming Conventions (clear & consistent) + +> Keep names lowercase, hyphen-separated, environment-scoped by vhost. Examples below assume vhost `/qa` or `/prod`. + +### 4.1 Exchanges + +- **Events:** `{service}.events` (type: `fanout`) + - e.g., `common.events` +- **Commands:** `{service}.commands` (type: `direct`) + - e.g., `common.commands` (consumed by Common) + +### 4.2 Queues + +- **Per consumer endpoint:** `{service}.{endpoint}` + - e.g., `common.create-transaction`, `common.create-customer`, `sales.org-sync` + +### 4.3 Routing Keys (for `direct`/`topic`) + +- Commands: `{action}` or `{aggregate}.{action}` + - e.g., `transaction.create`, `customer.create`, `vendor.create` +- Events (`fanout`) ignore routing keys; if switching to `topic`, prefer: `{aggregate}.{event}` + - e.g., `organization.created`, `company.updated`, `user.deleted` + +> **MassTransit** will create exchanges/queues automatically based on endpoint configuration when `ConfigureEndpoints()` is used. + +--- + +## 5) Service-wise Contracts (with suggested names) + +### 5.1 Common → Events (fanout on `common.events`) + +| Event | Example Payload (envelope) | Consumers | +| -------------------------- | ----------------------------------------------- | ---------------------------------------------- | +| `OrganizationCreatedEvent` | `{ "id": "...", "name": "...", ... }` | Sales, Purchase, Payroll, Inventory, Community | +| `CompanyCreatedEvent` | `{ "id": "...", "organizationId": "...", ... }` | Same | +| `UserCreatedEvent` | `{ "id": "...", "email": "...", ... }` | Same | + +**Queues (per consumer)** + +- `sales.org-sync`, `purchase.org-sync`, `payroll.org-sync`, `inventory.org-sync`, `community.org-sync` + _(bound to `common.events`)_ + +### 5.2 Commands → Common (direct on `common.commands`) + +| From | Command | Routing Key | Queue (endpoint) | Description | +| --------- | -------------------------- | -------------------- | --------------------------- | ------------------------------------- | +| Sales | `CreateTransactionCommand` | `transaction.create` | `common.create-transaction` | Invoice/Receipt/Reversal/Credit/Debit | +| Purchase | `CreateTransactionCommand` | `transaction.create` | `common.create-transaction` | Bill/Payment/Reversal/Credit/Debit | +| Payroll | `CreateTransactionCommand` | `transaction.create` | `common.create-transaction` | Salary posting/payment | +| Sales | `CreateCustomerCommand` | `customer.create` | `common.create-customer` | Mirror owner entity | +| Purchase | `CreateVendorCommand` | `vendor.create` | `common.create-vendor` | Mirror owner entity | +| Payroll | `CreateEmployeeCommand` | `employee.create` | `common.create-employee` | Mirror owner entity | +| Payroll | `CreateUserCommand` | `user.create` | `common.create-user` | Employee → User mapping | +| Community | `CreateUserCommand` | `user.create` | `common.create-user` | Member → User mapping | + +### 5.3 Community → Sales (direct on `sales.commands`) + +| Command | Routing Key | Queue | Description | +| ----------------------- | ----------------- | ----------------------- | -------------------------------- | +| `CreateCustomerCommand` | `customer.create` | `sales.create-customer` | Unit becomes a Customer in Sales | + +### 5.4 Events Consumed by Sales/Purchase/Payroll/Inventory/Community + +All consume from `common.events` (fanout) to maintain local master copies. + +--- + +## 6) MassTransit Setup (C#) + +### 6.1 Bus Configuration (generic) + +```csharp +using MassTransit; + +builder.Services.AddMassTransit(x => +{ + x.AddConsumers(typeof(Program).Assembly); // or specific consumers + + x.UsingRabbitMq((context, cfg) => + { + cfg.Host("rabbitmq-host", h => + { + h.Username("user"); + h.Password("pass"); + h.VirtualHost("/qa"); // or "/prod" + }); + + // Global topology + cfg.PrefetchCount = 16; // tune + cfg.ConcurrentMessageLimit = 8; // optional + + // Retry policy (immediate + redelivery) + cfg.UseMessageRetry(r => + { + r.Immediate(3); + r.Ignore(); // example + }); + cfg.UseDelayedRedelivery(r => + { + r.Intervals(TimeSpan.FromSeconds(10), TimeSpan.FromMinutes(1)); + }); + + // Error/skip handling + cfg.UseKillSwitch(options => + { + options.SetActivationThreshold(10); + options.SetTripThreshold(0.15); + options.SetRestartTimeout(s: 30); + }); + + // Endpoint auto-bindings per consumer + cfg.ConfigureEndpoints(context); + }); +}); +``` + +### 6.2 Define Contracts + +```csharp +public interface CreateTransactionCommand +{ + Guid CorrelationId { get; } + Guid OrganizationId { get; } + Guid CompanyId { get; } + string SourceService { get; } + DateTime Timestamp { get; } + // domain-specific fields... +} +``` + +### 6.3 Producer (Sales → Common) + +```csharp +public class SalesTransactionPublisher +{ + private readonly IPublishEndpoint _publish; // for events + private readonly ISendEndpointProvider _send; // for commands (to a specific queue) + + public SalesTransactionPublisher(IPublishEndpoint publish, ISendEndpointProvider send) + { + _publish = publish; + _send = send; + } + + public async Task SendCreateTransactionAsync(CreateTransactionCommand cmd) + { + // Resolve the command queue endpoint (MassTransit builds the name as you define endpoints) + var endpoint = await _send.GetSendEndpoint(new Uri("queue:common.create-transaction")); + await endpoint.Send(cmd); + } +} +``` + +### 6.4 Consumer (Common side) + +```csharp +public class CreateTransactionConsumer : IConsumer +{ + public async Task Consume(ConsumeContext context) + { + var msg = context.Message; + + // Process and write ledger/journal entries... + await Task.CompletedTask; + } +} + +// Endpoint mapping +builder.Services.AddMassTransit(x => +{ + x.AddConsumer(); + + x.UsingRabbitMq((context, cfg) => + { + cfg.Host("rabbitmq-host", h => { /* creds + vhost */ }); + cfg.ReceiveEndpoint("common.create-transaction", e => + { + e.ConfigureConsumer(context); + e.PrefetchCount = 32; + e.ConcurrentMessageLimit = 16; + }); + }); +}); +``` + +### 6.5 Events (fanout) + +```csharp +public interface OrganizationCreatedEvent +{ + Guid Id { get; } + string Name { get; } + DateTime Timestamp { get; } +} + +public class CommonEventPublisher +{ + private readonly IPublishEndpoint _publish; + public CommonEventPublisher(IPublishEndpoint publish) => _publish = publish; + + public Task PublishOrganizationAsync(OrganizationCreatedEvent evt) => _publish.Publish(evt); +} +``` + +> MassTransit uses **publish/subscribe** for events. Each subscriber gets its own queue bound to the exchange. + +### 6.6 Outbox & Idempotency (recommended) + +```csharp +// In Startup: +cfg.UseInMemoryOutbox(); // Or EFCore outbox to ensure "once" semantics + +// In consumers, use CorrelationId + natural keys to de-duplicate. +``` + +--- + +## 7) Error Queues, Retries & Reprocessing + +### 7.1 How errors flow + +- If a consumer throws, **MassTransit** applies retry policy (immediate & delayed if configured). +- After retries are exhausted, the message goes to **`{endpoint}_error`** queue. +- Messages explicitly skipped (e.g., validation failure) go to **`{endpoint}_skipped`**. +- If RabbitMQ DLX policy exists, TTL/reject routes to **DLQ** (often same as `_error`). + +### 7.2 Viewing messages (RabbitMQ Management UI) + +1. Navigate to the correct **vhost** (`/qa` or `/prod`). +2. **Queues** → filter for `*_error` or `*_skipped`. +3. Click a queue → **Get messages** → choose count → **Get Message(s)**. +4. Inspect headers/body to diagnose (CorrelationId, exceptions, stack traces). + +### 7.3 Reprocessing a failed message + +- **Option A: Requeue from UI** + - In the `_error` queue, **Get messages** → **Requeue** to the original queue name (`common.create-transaction`). +- **Option B: Move via script** + - Write a small tool/consumer that reads from `_error` and republishes to the main endpoint. +- **Option C: MassTransit Scheduler** + - Use scheduled redelivery to retry after a cool-down (Quartz or RabbitMQ delayed exchange). + +> **Important:** fix the root cause (code/data) before bulk requeue to avoid loops. Consider placing a **poison message parking lot** queue. + +### 7.4 Operational alerts + +- Alert on growth in `_error`/DLQ and on consumer crashes. +- Grafana dashboards for queue depth, publish/consume rates, and connection health. + +--- + +## 8) How to set up DLQ/DLX (example policy) + +```bash +# Example: set DLX for all 'common.*' queues in vhost /qa +rabbitmqctl set_policy -p /qa common-dlx "^common\." \ +'{"dead-letter-exchange":"common.dlx","message-ttl":604800000}' --apply-to queues + +# Create a DLX exchange and bind a DLQ +# (MassTransit can also create these when endpoints are defined) +``` + +> Use **regex** in policy to target a family of queues (`^common\.`). +> Set **message-ttl** (milliseconds) if you want deferred expiration behavior. + +--- + +## 9) Security & Access + +- **Per-vhost users** (e.g., `dhanman-qa`, `dhanman-prod`), least privilege. +- **TLS** if broker is exposed beyond private network. +- Credentials in **Vault**; apps fetch via AppRole. +- Disable guest access; restrict management UI to admin IPs. +- Separate **publisher vs consumer** permissions if needed. + +--- + +## 10) Monitoring & SLOs + +- **Key metrics:** queue depth, consumer count, publish/consume rate, unacked messages, connection churn, error queue size. +- **Alarms:** sustained backlog, dead consumer, error queue growth, DLQ spikes. +- **Tracing:** include `CorrelationId`, `CausationId`, `SourceService` in envelope; feed to logs/APM. + +--- + +## 11) Message Envelope (standard proposal) + +```json +{ + "MessageId": "9f7b9f3c-7dad-4c0b-8d62-6a8c8b7f5f54", + "CorrelationId": "a2b3c4d5-e6f7-4890-9abc-0123456789ab", + "CausationId": "optional-previous-id", + "OrganizationId": "org-uuid", + "CompanyId": "company-uuid", + "SourceService": "sales", + "MessageType": "CreateTransactionCommand", + "OccurredOnUtc": "2025-10-17T10:35:00Z", + "Payload": { + /* domain-specific */ + } +} +``` + +--- + +## 12) End-to-end Flow Tables + +### 12.1 Ledger flows + +| Source | Trigger | Command | Exchange | Routing Key | Target Queue | Result | +| -------- | ----------------------------- | --------------------------- | ----------------- | -------------------- | --------------------------- | ----------------------- | +| Sales | Invoice created | CreateTransactionCommand | `common.commands` | `transaction.create` | `common.create-transaction` | Ledger entry | +| Sales | Receipt posted | CreateTransactionCommand | `common.commands` | `transaction.create` | `common.create-transaction` | Ledger entry | +| Sales | Reversal (paid invoice) | CreateTransactionCommand ×2 | `common.commands` | `transaction.create` | `common.create-transaction` | Reverse money + invoice | +| Purchase | Bill created/payment/reversal | CreateTransactionCommand | `common.commands` | `transaction.create` | `common.create-transaction` | Ledger entry | +| Payroll | Salary posted/paid | CreateTransactionCommand | `common.commands` | `transaction.create` | `common.create-transaction` | Ledger entry | + +### 12.2 Master data replication + +| Publisher | Event | Exchange | Consumers (Queues) | +| --------- | ------------------------ | --------------- | ----------------------------------------------------------------------------------------------------- | +| Common | OrganizationCreatedEvent | `common.events` | `sales.org-sync`, `purchase.org-sync`, `payroll.org-sync`, `inventory.org-sync`, `community.org-sync` | +| Common | CompanyCreatedEvent | `common.events` | Same as above | +| Common | UserCreatedEvent | `common.events` | Same as above | + +### 12.3 Owner entity mirroring + +| Owner | Command | Target | Exchange | Routing Key | Queue | +| ------------------------- | --------------------- | ------ | ----------------- | ----------------- | ------------------------ | +| Sales (Customer) | CreateCustomerCommand | Common | `common.commands` | `customer.create` | `common.create-customer` | +| Purchase (Vendor) | CreateVendorCommand | Common | `common.commands` | `vendor.create` | `common.create-vendor` | +| Payroll (Employee) | CreateEmployeeCommand | Common | `common.commands` | `employee.create` | `common.create-employee` | +| Community (Member→User) | CreateUserCommand | Common | `common.commands` | `user.create` | `common.create-user` | +| Community (Unit→Customer) | CreateCustomerCommand | Sales | `sales.commands` | `customer.create` | `sales.create-customer` | + +--- + +## 13) Diagrams + +### 13.1 High-level system + +```mermaid +flowchart LR + subgraph Common["COMMON (Financial Core)"] + E1["Publishes: Org/Company/User Events (fanout: common.events)"] + C1["Consumes: Commands (direct: common.commands)\nQueues: create-transaction, create-customer, create-vendor, create-employee, create-user"] + end + + subgraph Sales["SALES"] + S1["Publishes: CreateTransaction, CreateCustomer (to Common)"] + S2["Consumes: Org/Company/User Events, UnitCreatedEvent"] + end + + subgraph Purchase["PURCHASE"] + P1["Publishes: CreateTransaction, CreateVendor (to Common)"] + P2["Consumes: Org/Company/User Events"] + end + + subgraph Payroll["PAYROLL"] + Y1["Publishes: CreateTransaction, CreateEmployee, CreateUser (to Common)"] + Y2["Consumes: Org/Company/User Events"] + end + + subgraph Community["COMMUNITY"] + M1["Publishes: CreateCustomer (to Sales), CreateUser (to Common)"] + M2["Consumes: Org/Company/User Events"] + end + + E1 --> S2 & P2 & Y2 & M2 + S1 --> C1 + P1 --> C1 + Y1 --> C1 + M1 --> S1 & C1 +``` + +### 13.2 Command routing (direct) + +```mermaid +sequenceDiagram + participant Sales + participant Exchange as common.commands (direct) + participant Queue as common.create-transaction + participant Common + + Sales->>Exchange: Send(CreateTransactionCommand) routingKey=transaction.create + Exchange->>Queue: bind(routingKey=transaction.create) + Queue->>Common: deliver message + Common-->>Sales: ack (optional reply pattern out-of-band) +``` + +### 13.3 Event fanout (broadcast) + +```mermaid +flowchart TB + E["Exchange: common.events (fanout)"] + Qs["Queues: sales.org-sync | purchase.org-sync | payroll.org-sync | inventory.org-sync | community.org-sync"] + Pub["Publisher: Common"] + Sub["Subscribers: Sales, Purchase, Payroll, Inventory, Community"] + + Pub --> E + E --> Qs + Qs --> Sub +``` + +--- + +## 14) Operational Runbook (daily quick-check) + +1. **Brokers healthy:** all nodes up, disk free > 20%, file descriptors OK. +2. **Connections & channels:** stable counts; no runaway reconnect loops. +3. **Queues depth:** command queues empty/near-empty; event sync queues not growing. +4. **Unacked:** < threshold (e.g., 100 per queue). +5. **Error queues:** no sustained growth; investigate spikes. +6. **Consumers:** expected count matches deployments; restart failed pods/services. +7. **Alerts:** clear or acknowledged; Jira ticket for recurring patterns. + +--- + +## 15) Glossary + +- **Virtual Host (vhost):** Namespaced RabbitMQ instance for isolation (objects & permissions). +- **Exchange:** Routes messages to queues. Types: `direct`, `fanout`, `topic`, `headers`. +- **Queue:** Durable buffer from which consumers read messages. +- **Binding:** Rule tying exchange → queue (plus optional routing key). +- **DLX/DLQ:** Dead Letter Exchange/Queue for failed/expired messages. +- **Prefetch/QoS:** Concurrent delivery control per consumer. +- **MassTransit:** .NET messaging framework providing bus, consumers, retries, outbox, sagas. +- **Outbox Pattern:** Ensures once-only publish relative to DB transactions. + +--- + +## 16) Checklists + +### 16.1 Adding a new command + +- [ ] Define contract interface (`ICommandName`). +- [ ] Choose exchange (`{target}.commands`) & routing key. +- [ ] Add producer (sender) code and unit tests. +- [ ] Add consumer endpoint in target service (`{target}.{action}` queue). +- [ ] Configure retries, idempotency, and outbox. +- [ ] Update docs & diagrams. + +### 16.2 Adding a new event + +- [ ] Define event interface (`I{Aggregate}{PastTense}Event`). +- [ ] Publish via `IPublishEndpoint`. +- [ ] Create subscriber queue per consumer service. +- [ ] Ensure consumer is idempotent (events can be replayed). +- [ ] Update docs & diagrams. + +--- + +## 17) Appendix: Sample Policies & CLI + +```bash +# List vhosts +rabbitmqctl list_vhosts + +# Create vhost +rabbitmqctl add_vhost /qa + +# Add user & set permissions +rabbitmqctl add_user dhanman-qa 'REDACTED_PASSWORD' +rabbitmqctl set_permissions -p /qa dhanman-qa ".*" ".*" ".*" # tune for least-privilege + +# Enable management plugin (if not already) +rabbitmq-plugins enable rabbitmq_management +``` + +--- + +**End of document.** diff --git a/mkdocs.yml b/mkdocs.yml index 3097161..fb58c9d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -5,123 +5,125 @@ nav: # === PRODUCT DOCUMENTATION === - Product: - - Overview: product/index.md - - Getting Started: - - App UI Structure: product/getting-started/app-ui-structure.md - - Email Templates: product/getting-started/email-templates.md - - Community & Engagement: - - Complaints: product/community/complaints/ - - Notice Board: product/community/notice-board/ - - Polls & Voting: product/community/polls-voting/ - - Resident Portal: product/community/resident-portal/ - - Gate & Security: - - Visitor Management: product/gate-management/visitor-management/ - - Access Control: product/gate-management/access-control/ - - Security Staff: product/gate-management/security-staff/ - - Vehicle Tracking: product/gate-management/vehicle-tracking/ - - Events & Calendar: - - Event Management: product/events-calendar/event-management/ - - Calendar Views: product/events-calendar/calendar-views/ - - Facility Booking: product/events-calendar/facility-booking/ - - Financial Management: - - Overview: product/financial-management/introduction.md - - Invoices: product/financial-management/invoices/ - - Maintenance Bills: product/financial-management/maintenance-bills/ - - Receipts: product/financial-management/receipts/ - - Purchase & Vendor Management: - - Vendors: product/purchase-management/vendors/ - - Purchase Orders: product/purchase-management/purchase-orders/ - - Inventory & Assets: - - Asset Management: product/inventory-assets/asset-management/ - - Stock Management: product/inventory-assets/stock-management/ - - Payroll & HR: - - Employee Management: product/payroll/employee-management/ - - Attendance: product/payroll/attendance/ - - Salary Processing: product/payroll/salary-processing/ - - Statutory Compliance: product/payroll/statutory-compliance/ - - Water Management: - - Meter Reading: product/water-management/meter-reading/ - - Water Billing: product/water-management/water-billing/ - - Consumption Reports: product/water-management/consumption-reports/ - - Reports & Analytics: product/reports-analytics/ - - Integrations: - - Payment Gateways: product/integrations/payment-gateways/ - - Email Providers: product/integrations/email-providers/ - - SMS Gateways: product/integrations/sms-gateways/ - - Accounting Export: product/integrations/accounting-export/ - - Mobile Applications: - - Resident App: product/mobile-app/resident-app/ - - Guard App: product/mobile-app/guard-app/ - - Audit & Compliance: - - Audit Trails: product/audit-compliance/audit-trails/ - - Compliance Reports: product/audit-compliance/compliance-reports/ - - Document Management: product/audit-compliance/document-management/ - - Settings & Configuration: - - User & Roles: product/users-roles/ - - Global Configuration: product/settings-configuration/ + - Overview: product/index.md + - Getting Started: + - App UI Structure: product/getting-started/app-ui-structure.md + - Email Templates: product/getting-started/email-templates.md + - Community & Engagement: + - Complaints: product/community/complaints/ + - Notice Board: product/community/notice-board/ + - Polls & Voting: product/community/polls-voting/ + - Resident Portal: product/community/resident-portal/ + - Gate & Security: + - Visitor Management: product/gate-management/visitor-management/ + - Access Control: product/gate-management/access-control/ + - Security Staff: product/gate-management/security-staff/ + - Vehicle Tracking: product/gate-management/vehicle-tracking/ + - Events & Calendar: + - Event Management: product/events-calendar/event-management/ + - Calendar Views: product/events-calendar/calendar-views/ + - Facility Booking: product/events-calendar/facility-booking/ + - Financial Management: + - Overview: product/financial-management/introduction.md + - Invoices: product/financial-management/invoices/ + - Maintenance Bills: product/financial-management/maintenance-bills/ + - Receipts: product/financial-management/receipts/ + - Purchase & Vendor Management: + - Vendors: product/purchase-management/vendors/ + - Purchase Orders: product/purchase-management/purchase-orders/ + - Inventory & Assets: + - Asset Management: product/inventory-assets/asset-management/ + - Stock Management: product/inventory-assets/stock-management/ + - Payroll & HR: + - Employee Management: product/payroll/employee-management/ + - Attendance: product/payroll/attendance/ + - Salary Processing: product/payroll/salary-processing/ + - Statutory Compliance: product/payroll/statutory-compliance/ + - Water Management: + - Meter Reading: product/water-management/meter-reading/ + - Water Billing: product/water-management/water-billing/ + - Consumption Reports: product/water-management/consumption-reports/ + - Reports & Analytics: product/reports-analytics/ + - Integrations: + - Payment Gateways: product/integrations/payment-gateways/ + - Email Providers: product/integrations/email-providers/ + - SMS Gateways: product/integrations/sms-gateways/ + - Accounting Export: product/integrations/accounting-export/ + - Mobile Applications: + - Resident App: product/mobile-app/resident-app/ + - Guard App: product/mobile-app/guard-app/ + - Audit & Compliance: + - Audit Trails: product/audit-compliance/audit-trails/ + - Compliance Reports: product/audit-compliance/compliance-reports/ + - Document Management: product/audit-compliance/document-management/ + - Settings & Configuration: + - User & Roles: product/users-roles/ + - Global Configuration: product/settings-configuration/ # === SYSTEM DOCUMENTATION === - System: - - Overview: system/index.md - - Architecture: - - Overview: system/architecture/overview.md - - Design Decisions: system/architecture/design-decisions.md - - Deployment & Scalability: system/architecture/deployment-scalability.md - - Security Architecture: system/architecture/security-architecture.md - - Scheduling Architecture: system/architecture/adr/Sequential_Scheduling_Execution.md - - Principles: system/architecture/principles.md - - ADR: system/architecture/adr/ - - Diagrams: system/architecture/diagrams/ - - Modules: system/architecture/modules/ - - Patterns: - - CQRS: system/architecture/patterns/cqrs.md - - Domain-Driven Design: system/architecture/patterns/domain-driven-design.md - - Event Sourcing & Messaging: system/architecture/patterns/event-sourcing.md - - Communication Patterns: system/architecture/patterns/communication-patterns.md - - Scheduled Jobs (Hangfire): system/architecture/patterns/scheduled-jobs.md - - Resilience & Fault Tolerance: system/architecture/patterns/resilience.md - - Infrastructure: - - Overview: system/infrastructure/overview.md - - Database: system/infrastructure/database/postgresql-setup.md - - Messaging: system/infrastructure/messaging/rabbitmq-setup.md - - Monitoring: system/infrastructure/monitoring/grafana-loki.md - - Storage: system/infrastructure/storage/minio-setup.md - - Development: - - Overview: system/development/index.md - - Getting Started: system/development/getting-started.md - - Standards: - - Backend Principles: system/development/standards/cc_principle_backend.md - - Frontend Principles: "system/development/standards/clean code principle - front end.md" - - Code Review Checklist: system/development/standards/code-review-checklist.md - - Git Workflow: system/development/standards/git-workflow.md - - Project Structure: - - Create New Project: system/development/project-structure/create-new-project.md - - Repository Conventions: system/development/project-structure/repository-conventions.md - - API Internal: - - Event Schemas: system/development/api-internal/event-schemas.md - - Service Contracts: system/development/api-internal/service-contracts.md - - Testing: - - Troubleshooting: system/development/testing/troubleshooting.md - - Test ID Strategy: system/development/testing/testid-strategy.md - - Operations: - - Commands & Service Management: system/operations/commands.md - - Deployment: - - QA Deployment Guide: system/operations/deployment/qa_deployment_guide.md - - QA-PROD Deployment: system/operations/deployment/qa_prod_deployment_guide.md - - Data Backup & Restore: system/operations/deployment/dump_restore_clean.md - - Monitoring: - - Dashboards: system/operations/monitoring/dashboards.md - - Setup: system/operations/monitoring/smonitoring-setup-guide.md - - Runbooks: system/operations/runbooks/qa-refresh-procedure.md - - Onboarding: - - Developer Onboarding: system/onboarding/developer-onboarding.md - - First Contribution: system/onboarding/first-contribution.md - - Security: - - Overview: system/security/index.md - - Authentication Flow: system/security/authentication-flow.md - - Permissions Guidelines: system/security/permissions-naming-guidelines.md - - Secrets Management: system/security/secrets-management.md - - Office Etiquette Policy: system/security/office_etiquette_policy.md + - Overview: system/index.md + - Architecture: + - Overview: system/architecture/overview.md + - Design Decisions: system/architecture/design-decisions.md + - Deployment & Scalability: system/architecture/deployment-scalability.md + - Security Architecture: system/architecture/security-architecture.md + - Acknowledgment Event Pattern: system/architecture/AcknowledgmentEventPattern.md + - Messaging Architecture: system/architecture/dhanman-messaging-architecture.md + - Scheduling Architecture: system/architecture/adr/Sequential_Scheduling_Execution.md + - Principles: system/architecture/principles.md + - ADR: system/architecture/adr/ + - Diagrams: system/architecture/diagrams/ + - Modules: system/architecture/modules/ + - Patterns: + - CQRS: system/architecture/patterns/cqrs.md + - Domain-Driven Design: system/architecture/patterns/domain-driven-design.md + - Event Sourcing & Messaging: system/architecture/patterns/event-sourcing.md + - Communication Patterns: system/architecture/patterns/communication-patterns.md + - Scheduled Jobs (Hangfire): system/architecture/patterns/scheduled-jobs.md + - Resilience & Fault Tolerance: system/architecture/patterns/resilience.md + - Infrastructure: + - Overview: system/infrastructure/overview.md + - Database: system/infrastructure/database/postgresql-setup.md + - Messaging: system/infrastructure/messaging/rabbitmq-setup.md + - Monitoring: system/infrastructure/monitoring/grafana-loki.md + - Storage: system/infrastructure/storage/minio-setup.md + - Development: + - Overview: system/development/index.md + - Getting Started: system/development/getting-started.md + - Standards: + - Backend Principles: system/development/standards/cc_principle_backend.md + - Frontend Principles: "system/development/standards/clean code principle - front end.md" + - Code Review Checklist: system/development/standards/code-review-checklist.md + - Git Workflow: system/development/standards/git-workflow.md + - Project Structure: + - Create New Project: system/development/project-structure/create-new-project.md + - Repository Conventions: system/development/project-structure/repository-conventions.md + - API Internal: + - Event Schemas: system/development/api-internal/event-schemas.md + - Service Contracts: system/development/api-internal/service-contracts.md + - Testing: + - Troubleshooting: system/development/testing/troubleshooting.md + - Test ID Strategy: system/development/testing/testid-strategy.md + - Operations: + - Commands & Service Management: system/operations/commands.md + - Deployment: + - QA Deployment Guide: system/operations/deployment/qa_deployment_guide.md + - QA-PROD Deployment: system/operations/deployment/qa_prod_deployment_guide.md + - Data Backup & Restore: system/operations/deployment/dump_restore_clean.md + - Monitoring: + - Dashboards: system/operations/monitoring/dashboards.md + - Setup: system/operations/monitoring/smonitoring-setup-guide.md + - Runbooks: system/operations/runbooks/qa-refresh-procedure.md + - Onboarding: + - Developer Onboarding: system/onboarding/developer-onboarding.md + - First Contribution: system/onboarding/first-contribution.md + - Security: + - Overview: system/security/index.md + - Authentication Flow: system/security/authentication-flow.md + - Permissions Guidelines: system/security/permissions-naming-guidelines.md + - Secrets Management: system/security/secrets-management.md + - Office Etiquette Policy: system/security/office_etiquette_policy.md theme: name: material @@ -154,4 +156,4 @@ extra_javascript: extra_css: - https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.3.1/styles/default.min.css - extra.css - - https://cdn.jsdelivr.net/npm/glightbox/dist/css/glightbox.min.css \ No newline at end of file + - https://cdn.jsdelivr.net/npm/glightbox/dist/css/glightbox.min.css