feat: Implement Phase 2 cross-cutting services with email and notification enhancements
- Refactor email verification and password reset utilities to use new email service. - Introduce email delivery queue and notification delivery model for better tracking. - Enhance file validation and storage services for improved security and ownership management. - Add cron job for cleaning inactive notifications with retention policy. - Update document worker to handle document generation and storage more efficiently. - Implement logging improvements in activity and log workers. - Create comprehensive documentation for new API endpoints and services. - Add unit tests for file validation and notification policies to ensure robustness.
This commit is contained in:
Sathira Sri Sathara
@@ -0,0 +1,31 @@
|
||||
# ZUMRI Cross-Cutting Services API
|
||||
|
||||
All paths also exist below `/api`; clients should use `/api/v1`. Protected routes accept the Phase 1 access cookie or bearer token. Examples use placeholders and never expose storage keys.
|
||||
|
||||
## Media
|
||||
|
||||
- `POST /api/v1/upload` — multipart field `file`, optional text field `use_for`; creates a private owner-bound upload.
|
||||
- `GET /api/v1/upload/signed-url/:id` — returns `{ id, url, expiresIn }` after ownership/permission checks.
|
||||
- `DELETE /api/v1/upload/:id` — marks an owned/authorized upload deleted and removes its object best-effort.
|
||||
- `GET /api/v1/profile/me/avatar` and `/background` — signed self profile media access. Legacy owner-checked ID routes remain.
|
||||
|
||||
## Notifications
|
||||
|
||||
- `POST /api/v1/notification` — `notifications.manage`; body includes headline, description, `USER|ANNOUNCEMENT`, and `userIds` for USER messages.
|
||||
- `GET /api/v1/notification/announcements`
|
||||
- `GET /api/v1/notification/me`
|
||||
- `PATCH /api/v1/notification/:notificationId/read`
|
||||
- `PATCH /api/v1/notification/read-all`
|
||||
|
||||
## Documents
|
||||
|
||||
- `GET /api/v1/document/types`
|
||||
- `GET /api/v1/document/saved`
|
||||
- `POST /api/v1/document/draft`
|
||||
- `POST /api/v1/document/generate` with `{ "document":"<registered-type>", "documentType":"pdf", "documentData":{} }`; returns HTTP 202 and persisted status.
|
||||
- `GET /api/v1/document/jobs/:jobId`
|
||||
- `GET /api/v1/document/:docId`
|
||||
- `GET /api/v1/document/:docId/download` — returns a short-lived URL; it does not delete the artifact.
|
||||
- `DELETE /api/v1/document/job/:jobId`
|
||||
|
||||
The legacy reference-number GET returns 410 because reads must not consume sequences. References are assigned as part of resource creation.
|
||||
@@ -1,5 +1,13 @@
|
||||
# ZUMRI Current Backend Status
|
||||
|
||||
## Phase 2 Completion Update
|
||||
|
||||
Completion date: 2026-09-03. Phase 2 hardens the existing shared-service foundation without adding commerce domains. Module 14 (notifications) is now approximately 72%; Module 19 (file/media) 82%; Module 20 (audit/config/logging) 68%; and Module 21 (background jobs) 78%. The document subsystem is approximately 82%.
|
||||
|
||||
Storage now has a reusable S3/S3-compatible boundary, actual-content validation, controlled keys, checksums, owner/status metadata, compensation, and authorization-safe signed URLs. Email is routed through BullMQ with delivery status and final-failure persistence. Notifications have fixed aliases, unique assignment migration, self-only inbox/read operations, announcement support, and preference policy. Queue defaults, idempotent job IDs, Bull Board registration, safe failure handling, stronger append-only activity records, and structured redacted logs are in place. Documents now have ownership, controlled registry validation, persisted generation lifecycle, safe status, and non-destructive signed download.
|
||||
|
||||
The full mocked suite contains 10 suites/40 tests and passes; syntax checks cover 153 JavaScript files. The new migration was not executed. Remaining work is staging migration/data pre-checks plus real MySQL, Redis, S3-compatible, SMTP, PDF/browser, retention-volume, and concurrency validation. After those operational checks and permission seeding, it is safe to begin Phase 3. See `Documentation/PHASE_2_CROSS_CUTTING_SERVICES.md` and `Documentation/API_CROSS_CUTTING_SERVICES.md`.
|
||||
|
||||
Audit date: 2026-09-03
|
||||
Scope: repository source, configuration, lockfile, existing documentation, safe syntax/test/dependency checks. No database, Redis, S3, email, or other external service was mutated.
|
||||
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
# ZUMRI Phase 2 Cross-Cutting Services
|
||||
|
||||
## Objective
|
||||
|
||||
Harden the shared storage, messaging, queue, audit, logging, and document infrastructure without starting commerce modules.
|
||||
|
||||
## Existing Components Reused
|
||||
|
||||
The existing AWS SDK client, Upload/Notification/UserNotification/Document models, Redis/BullMQ topology, generators, templates, reference utility, workers, cron lifecycle, Phase 0 operations, and Phase 1 identity/RBAC remain the foundation.
|
||||
|
||||
## Storage Architecture
|
||||
|
||||
`storage.service.js` is the sole AWS SDK boundary. It supports AWS S3 and endpoint/path-style compatible providers, buffer upload, delete, HEAD existence checks, and signed downloads. Objects remain private.
|
||||
|
||||
## Upload Security
|
||||
|
||||
Multer performs an early allowlist/size check. The service then rejects empty content and validates JPEG, PNG, WebP, PDF, and XLSX magic bytes against the claimed MIME. It derives the extension, sanitizes display filenames, and stores SHA-256 checksums. Object keys contain a controlled owner identifier, UTC year/month, UUID, and detected extension; original filenames and personal data are excluded.
|
||||
|
||||
## File Ownership
|
||||
|
||||
Uploads record uploader, owner type/id, purpose, visibility, and lifecycle status. Signed URL and deletion endpoints load metadata and enforce owner or explicit permission access. DB persistence failure after upload triggers best-effort object deletion. Deletion marks metadata DELETED before object removal.
|
||||
|
||||
## Signed URL Policy
|
||||
|
||||
Downloads use `S3_SIGNED_URL_TTL_SECONDS` (default 900 seconds). URLs are generated after each authorization decision and are not globally cached or exposed with bucket/key details.
|
||||
|
||||
## Email Architecture
|
||||
|
||||
Auth email enters `email.service.js`, creates a minimal delivery record, and queues a template-keyed job. The worker alone calls Nodemailer. OTPs/links may exist transiently in job data, so jobs have aggressive completion retention and payloads must never be logged.
|
||||
|
||||
## Email Retry Strategy
|
||||
|
||||
Email uses five exponential attempts. Envelope/message and SMTP 5xx failures are treated as permanent; transient provider/network errors retry. Final state is persisted without storing message bodies or variables.
|
||||
|
||||
## Notification Architecture
|
||||
|
||||
Notification aliases are explicit. Admin publication can assign one or many users atomically; announcements need no join rows. Self-service listing/read/read-all always derives the user from the access token.
|
||||
|
||||
## Notification Preferences
|
||||
|
||||
The central policy permits mandatory login OTP, password reset/change, and email verification even when marketing notifications are disabled. Optional external-channel messages honor profile preferences. In-app messages remain available.
|
||||
|
||||
## Queue Architecture
|
||||
|
||||
Activity, log, document, and email queues define retry, backoff, success retention, and failure retention appropriate to each workload. Bull Board includes all four and retains Phase 0 admin protection.
|
||||
|
||||
## Idempotency
|
||||
|
||||
Activity uses an event ID, email uses event/delivery ID, and documents use the generation record ID as BullMQ job ID. Workers check persisted state where duplicate execution could create a second artifact.
|
||||
|
||||
## Failed Job Handling
|
||||
|
||||
Email and document final failures update their associated database record with a bounded error code and timestamp. Stack traces and job payloads are not returned by APIs.
|
||||
|
||||
## Audit Logging
|
||||
|
||||
Activity events now support event ID, nullable actor, target, action/type, request/IP/user-agent context, sanitized JSON metadata, and occurrence time. APIs expose read operations only; inserts are idempotent by event ID.
|
||||
|
||||
## Logging Security
|
||||
|
||||
Queued file logs are JSON lines. Error stacks, request bodies, Authorization/Cookie values, and fields named like passwords, OTPs, tokens, or secrets are excluded/redacted. Log jobs have bounded retention; `LOG_RETENTION_DAYS` documents the intended operational file-retention window.
|
||||
|
||||
## Document Generation Lifecycle
|
||||
|
||||
Generation validates a controlled registry and PDF/XLSX format before queueing, creates an owner-bound record, and moves through QUEUED, PROCESSING, COMPLETED, or FAILED. Successful output becomes an owned Upload. Downloads create a signed URL and never delete the object.
|
||||
|
||||
## Document Ownership
|
||||
|
||||
Creator and owner may view status/data/download. SUPER_ADMIN bypasses; other administrative access requires the appropriate document permission. Cancellation follows the same ownership boundary.
|
||||
|
||||
## Migrations
|
||||
|
||||
`20260903020000-phase-2-cross-cutting-services.js` is new and forward-only. Before execution, back up and test a restored database. Pre-check duplicate `(user_id, notification_id)`, duplicate activity event IDs, duplicate document job IDs, duplicate document type names, and legacy uploads/documents without resolvable owners. Resolve duplicates explicitly; the migration intentionally does not delete data.
|
||||
|
||||
## Permissions
|
||||
|
||||
Shared names are `media.read/upload/delete`, `documents.read/create/delete`, `notifications.manage/read`, `audit.read`, and `queues.read`. SUPER_ADMIN retains the Phase 1 bypass. Production permission rows/grants must be seeded through the environment's controlled authorization process.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Added: `S3_ENDPOINT`, `S3_FORCE_PATH_STYLE`, `S3_SIGNED_URL_TTL_SECONDS`, `S3_MAX_UPLOAD_BYTES`, `EMAIL_QUEUE_CONCURRENCY`, `DOCUMENT_QUEUE_CONCURRENCY`, `NOTIFICATION_RETENTION_DAYS`, and `LOG_RETENTION_DAYS`. Optional integrations remain optional unless enabled.
|
||||
|
||||
## Tests
|
||||
|
||||
Unit coverage verifies magic bytes, mismatch/empty rejection, checksums, safe keys, HTML escaping, security-notification preference policy, and email queue retry/retention. Existing Phase 0/1 tests remain in the full suite. AWS, SMTP, Redis, and MySQL are not contacted.
|
||||
|
||||
## Remaining Known Issues
|
||||
|
||||
The migration has not been run against staging data. Real S3-compatible provider, SMTP, MySQL migration, Redis concurrency, large notification retention, and actual PDF/browser generation need staging validation. File-log deletion/rotation still belongs to deployment logrotate or a future controlled maintenance worker. Push delivery is intentionally an architecture placeholder only.
|
||||
|
||||
## Phase 3 Prerequisites
|
||||
|
||||
Complete the documented data pre-checks, apply all migrations to a restored database, seed shared permissions, run API and worker processes against staging Redis/MySQL, and exercise one upload/email/document lifecycle with non-production provider credentials.
|
||||
Reference in New Issue
Block a user