Sathira Sri Sathara
b6b345f245
- Refactor email verification and password reset utilities to use new email service. - Introduce email delivery queue and notification delivery model for better tracking. - Enhance file validation and storage services for improved security and ownership management. - Add cron job for cleaning inactive notifications with retention policy. - Update document worker to handle document generation and storage more efficiently. - Implement logging improvements in activity and log workers. - Create comprehensive documentation for new API endpoints and services. - Add unit tests for file validation and notification policies to ensure robustness.
6.3 KiB
ZUMRI Phase 2 Cross-Cutting Services
Objective
Harden the shared storage, messaging, queue, audit, logging, and document infrastructure without starting commerce modules.
Existing Components Reused
The existing AWS SDK client, Upload/Notification/UserNotification/Document models, Redis/BullMQ topology, generators, templates, reference utility, workers, cron lifecycle, Phase 0 operations, and Phase 1 identity/RBAC remain the foundation.
Storage Architecture
storage.service.js is the sole AWS SDK boundary. It supports AWS S3 and endpoint/path-style compatible providers, buffer upload, delete, HEAD existence checks, and signed downloads. Objects remain private.
Upload Security
Multer performs an early allowlist/size check. The service then rejects empty content and validates JPEG, PNG, WebP, PDF, and XLSX magic bytes against the claimed MIME. It derives the extension, sanitizes display filenames, and stores SHA-256 checksums. Object keys contain a controlled owner identifier, UTC year/month, UUID, and detected extension; original filenames and personal data are excluded.
File Ownership
Uploads record uploader, owner type/id, purpose, visibility, and lifecycle status. Signed URL and deletion endpoints load metadata and enforce owner or explicit permission access. DB persistence failure after upload triggers best-effort object deletion. Deletion marks metadata DELETED before object removal.
Signed URL Policy
Downloads use S3_SIGNED_URL_TTL_SECONDS (default 900 seconds). URLs are generated after each authorization decision and are not globally cached or exposed with bucket/key details.
Email Architecture
Auth email enters email.service.js, creates a minimal delivery record, and queues a template-keyed job. The worker alone calls Nodemailer. OTPs/links may exist transiently in job data, so jobs have aggressive completion retention and payloads must never be logged.
Email Retry Strategy
Email uses five exponential attempts. Envelope/message and SMTP 5xx failures are treated as permanent; transient provider/network errors retry. Final state is persisted without storing message bodies or variables.
Notification Architecture
Notification aliases are explicit. Admin publication can assign one or many users atomically; announcements need no join rows. Self-service listing/read/read-all always derives the user from the access token.
Notification Preferences
The central policy permits mandatory login OTP, password reset/change, and email verification even when marketing notifications are disabled. Optional external-channel messages honor profile preferences. In-app messages remain available.
Queue Architecture
Activity, log, document, and email queues define retry, backoff, success retention, and failure retention appropriate to each workload. Bull Board includes all four and retains Phase 0 admin protection.
Idempotency
Activity uses an event ID, email uses event/delivery ID, and documents use the generation record ID as BullMQ job ID. Workers check persisted state where duplicate execution could create a second artifact.
Failed Job Handling
Email and document final failures update their associated database record with a bounded error code and timestamp. Stack traces and job payloads are not returned by APIs.
Audit Logging
Activity events now support event ID, nullable actor, target, action/type, request/IP/user-agent context, sanitized JSON metadata, and occurrence time. APIs expose read operations only; inserts are idempotent by event ID.
Logging Security
Queued file logs are JSON lines. Error stacks, request bodies, Authorization/Cookie values, and fields named like passwords, OTPs, tokens, or secrets are excluded/redacted. Log jobs have bounded retention; LOG_RETENTION_DAYS documents the intended operational file-retention window.
Document Generation Lifecycle
Generation validates a controlled registry and PDF/XLSX format before queueing, creates an owner-bound record, and moves through QUEUED, PROCESSING, COMPLETED, or FAILED. Successful output becomes an owned Upload. Downloads create a signed URL and never delete the object.
Document Ownership
Creator and owner may view status/data/download. SUPER_ADMIN bypasses; other administrative access requires the appropriate document permission. Cancellation follows the same ownership boundary.
Migrations
20260903020000-phase-2-cross-cutting-services.js is new and forward-only. Before execution, back up and test a restored database. Pre-check duplicate (user_id, notification_id), duplicate activity event IDs, duplicate document job IDs, duplicate document type names, and legacy uploads/documents without resolvable owners. Resolve duplicates explicitly; the migration intentionally does not delete data.
Permissions
Shared names are media.read/upload/delete, documents.read/create/delete, notifications.manage/read, audit.read, and queues.read. SUPER_ADMIN retains the Phase 1 bypass. Production permission rows/grants must be seeded through the environment's controlled authorization process.
Environment Variables
Added: S3_ENDPOINT, S3_FORCE_PATH_STYLE, S3_SIGNED_URL_TTL_SECONDS, S3_MAX_UPLOAD_BYTES, EMAIL_QUEUE_CONCURRENCY, DOCUMENT_QUEUE_CONCURRENCY, NOTIFICATION_RETENTION_DAYS, and LOG_RETENTION_DAYS. Optional integrations remain optional unless enabled.
Tests
Unit coverage verifies magic bytes, mismatch/empty rejection, checksums, safe keys, HTML escaping, security-notification preference policy, and email queue retry/retention. Existing Phase 0/1 tests remain in the full suite. AWS, SMTP, Redis, and MySQL are not contacted.
Remaining Known Issues
The migration has not been run against staging data. Real S3-compatible provider, SMTP, MySQL migration, Redis concurrency, large notification retention, and actual PDF/browser generation need staging validation. File-log deletion/rotation still belongs to deployment logrotate or a future controlled maintenance worker. Push delivery is intentionally an architecture placeholder only.
Phase 3 Prerequisites
Complete the documented data pre-checks, apply all migrations to a restored database, seed shared permissions, run API and worker processes against staging Redis/MySQL, and exercise one upload/email/document lifecycle with non-production provider credentials.