Inbound Email (SMTP) to Webhook

Author: Martin Krivosija - LinkedIn

A simple, efficient script that provides an SMTP server to receive emails, parse content (including headers), store attachments in Amazon S3, and forward email content to a webhook. Graceful handling of multiple concurrent SMTP sessions and webhook requests.

Features

• SMTP server to receive emails concurrently
• Parses incoming emails using mailparser
• Enhanced Error Recovery: Automatic fallback to local storage when S3 is unavailable, with background retry mechanism
• Multiple Webhook Support: Route emails to different webhooks based on sender, recipient, subject, or custom rules
• Uploads attachments to Amazon S3 with intelligent fallback
• Forwards parsed email content to webhook endpoints
• Configurable via environment variables
• Handles large attachments gracefully
• Robust queue system for processing multiple emails and webhook requests simultaneously
• Durable disk-backed queue for crash-safe webhook delivery
• Webhook request signing with HMAC (X-Inbound-Email-Signature)
• SMTP ingress hardening (IP allowlists, sender restrictions, rate limits, message size limits)
• Optional encrypted local attachment storage at rest (AES-256-GCM)
• Comprehensive Test Coverage: >90% code coverage with Jest testing framework
• Daily rotating logs with 90-day retention

Prerequisites

• Node.js (v18 or later recommended)
• If saving attachments, an Amazon Web Services (AWS) account with S3 access or a compatible system
• A HTTP(s) webhook endpoint to receive the processed emails

Installation

1. Clone this repository:

   git clone https://github.com/kriiv/inbound-email.git
   cd inbound-email
   

2. Install dependencies:

   npm install
   

3. Copy the .env.example file to .env and set the required configuration: (eg. mv .env.example .env)

   Core Configuration

   Variable	Description	Required	Default
   WEBHOOK_URL	The URL where parsed emails will be sent (if not using rules)	Yes*	null
   WEBHOOK_RULES	JSON configuration for multiple webhook routing (see below)	No	null
   PORT	The port for the SMTP server to listen on	No	25
   SMTP_SECURE	Set to 'true' for TLS support (requires key/cert setup)	No	false
   WEBHOOK_CONCURRENCY	Number of concurrent webhook requests	No	5
   WEBHOOK_RETRY_DELAY_MS	Delay before retrying failed durable queue tasks	No	60000
   SMTP_MAX_CLIENTS	Maximum simultaneous SMTP client connections	No	200
   SMTP_SOCKET_TIMEOUT	Socket idle timeout in milliseconds	No	60000
   SMTP_CLOSE_TIMEOUT	Graceful close wait timeout in milliseconds	No	30000
   SMTP_MAX_MESSAGE_SIZE	Maximum accepted SMTP message size in bytes	No	10485760
   SMTP_RATE_LIMIT_WINDOW_MS	Window length for per-IP connection limiting in milliseconds	No	60000
   SMTP_RATE_LIMIT_MAX_CONNECTIONS	Max connections per IP per window	No	120
   MAX_QUEUE_SIZE	Maximum in-memory active queue items before temporary reject	No	1000
   DURABLE_QUEUE_PATH	Path for persisted webhook queue items	No	./queue-data

   Attachment Storage

   Variable	Description	Required	Default
   MAX_FILE_SIZE	Maximum attachment size in bytes (0 to disable S3 uploads)	No	5242880 (5MB)
   AWS_REGION	Your AWS region	If saving	null
   AWS_ACCESS_KEY_ID	Your AWS access key ID	If saving	null
   AWS_SECRET_ACCESS_KEY	Your AWS secret access key	If saving	null
   S3_BUCKET_NAME	The name of your S3 bucket for storing attachments	If saving	null

   Enhanced Error Recovery

   Variable	Description	Required	Default
   LOCAL_STORAGE_PATH	Directory for temporary storage when S3 fails	No	./temp-attachments
   LOCAL_STORAGE_RETENTION	Hours to keep local files before cleanup	No	24
   LOCAL_STORAGE_ENCRYPTION_KEY	32-byte key (hex/base64) for local AES-256-GCM encryption	No	null
   EXPOSE_LOCAL_ATTACHMENT_PATHS	Include local disk paths in webhook payloads	No	false
   S3_RETRY_INTERVAL	Minutes between S3 retry attempts	No	5

   Ingress Trust Policy

   Variable	Description	Required	Default
   REQUIRE_TRUSTED_RELAY	Reject SMTP clients unless they are in TRUSTED_RELAY_IPS	No	false
   TRUSTED_RELAY_IPS	Comma-separated trusted relay/source IPs	If enabled	[]
   ALLOWED_SMTP_CLIENTS	Comma-separated allowed SMTP client/source IPs	No	[]
   ALLOWED_SENDER_DOMAINS	Comma-separated envelope sender domain allowlist	No	[]
   REQUIRED_AUTH_RESULTS	Required Authentication-Results tokens (e.g. spf=pass)	No	[]

   Webhook Signing

   Variable	Description	Required	Default
   WEBHOOK_SECRET	HMAC secret used to sign outbound webhook payloads	No	null
   ALLOW_INSECURE_WEBHOOK_HTTP	Allow non-TLS (http://) webhook URLs	No	false

   Security (TLS)

   Variable	Description	Required	Default
   TLS_KEY_PATH	Path to the TLS private key file (if SMTP_SECURE=true)	If secure	null
   TLS_CERT_PATH	Path to the TLS certificate file (if SMTP_SECURE=true)	If secure	null

   Note: Either WEBHOOK_URL or WEBHOOK_RULES must be configured. S3 credentials are required if MAX_FILE_SIZE > 0.

Usage

Start the server:

npm start

The SMTP server will start and listen on the specified port (default: 25) on all network interfaces.

You can use pm2 or supervisor to keep the server running after restart. Example: pm2 start server.js

When NODE_ENV=production, startup validation enforces hardened defaults:

• REQUIRE_TRUSTED_RELAY=true
• TRUSTED_RELAY_IPS must be set
• ALLOWED_RECIPIENT_DOMAINS must be set
• WEBHOOK_SECRET must be set
• ALLOW_INSECURE_WEBHOOK_HTTP must be false

Multiple Webhook Configuration

The service supports routing emails to different webhooks based on email content. Configure using the WEBHOOK_RULES environment variable:

Simple Example

{
  "rules": [
    {
      "name": "support-emails",
      "conditions": { "to": "support@*" },
      "webhook": "https://support.webhook.com",
      "priority": 1
    },
    {
      "name": "sales-inquiries", 
      "conditions": { "subject": "*quote*" },
      "webhook": "https://sales.webhook.com",
      "priority": 2
    },
    {
      "name": "default",
      "conditions": {},
      "webhook": "https://default.webhook.com",
      "priority": 999
    }
  ]
}

Advanced Example

{
  "rules": [
    {
      "name": "urgent-admin",
      "conditions": {
        "from": "admin@company.com",
        "subject": "/^URGENT:/i"
      },
      "webhook": "https://urgent.webhook.com",
      "priority": 1,
      "stopProcessing": true
    },
    {
      "name": "with-attachments",
      "conditions": { "hasAttachments": "true" },
      "webhook": "https://attachments.webhook.com",
      "priority": 5
    },
    {
      "name": "team-notifications",
      "conditions": { "to": "team-*@company.com" },
      "webhook": "https://team.webhook.com",
      "priority": 10
    }
  ]
}

Condition Types

• Exact match: "from": "admin@company.com"
• Wildcard: "to": "support@*" or "subject": "*report*"
• Regex: "subject": "/^URGENT:/i"
• Built-in fields:
  • from, to, cc: Email addresses
  • subject: Email subject line
  • hasAttachments: "true" or "false"
• Custom headers: "header": {"name": "X-Priority", "value": "high"}

Rule Processing

1. Rules are sorted by priority (lower = higher priority)
2. All matching rules receive the email (unless stopProcessing: true)
3. If no rules match, falls back to WEBHOOK_URL if configured
4. Each webhook receives the email data plus _webhookMeta with rule information

Enhanced Error Recovery

The service provides automatic resilience for attachment storage:

Local Storage Fallback

• When S3 is unavailable, attachments are stored locally in LOCAL_STORAGE_PATH
• Background retry process attempts S3 upload every S3_RETRY_INTERVAL minutes
• Files older than LOCAL_STORAGE_RETENTION hours are automatically cleaned up
• Webhooks receive storage location and type information

Email Response Structure

{
  "from": "sender@example.com",
  "subject": "Email with attachments",
  "attachmentInfo": [
    {
      "filename": "document.pdf",
      "size": 1024,
      "location": "https://s3.amazonaws.com/bucket/document.pdf",
      "storageType": "s3"
    },
    {
      "filename": "backup.zip", 
      "size": 2048,
      "location": "/temp-attachments/123456-backup.zip",
      "storageType": "local",
      "note": "Temporarily stored locally, will be uploaded to S3 when available"
    }
  ],
  "storageSummary": {
    "total": 2,
    "uploadedToS3": 1,
    "storedLocally": 1,
    "skipped": 0
  }
}

Testing

Run the comprehensive test suite:

# Install dev dependencies
npm install

# Run tests
npm test

# Run tests with coverage
npm run test:coverage

# Run tests in watch mode (during development)
npm run test:watch

The test suite provides >90% code coverage across all services.

Sample Use Cases

1. Email to Ticket System: Use this bridge to receive support emails and automatically create tickets in your helpdesk system via the webhook.

2. Document Processing: Receive emails with document attachments, store them in S3, and trigger a document processing pipeline through the webhook.

3. Email Marketing Analysis: Collect incoming emails from a campaign, store any images or attachments, and send the content to an analytics system for processing.

4. Automated Reporting: Set up an email address that receives automated reports, stores them in S3, and notifies your team via the webhook.

5. DMARC Reporting: Receive DMARC reports via email and store them in S3.

Using inbound parse for something interesting? Please let me know, I'd love to hear about it.

Todo

• Rate limiting
• Log Storage (completed)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request or get in touch.

License

This project is licensed under the MIT License. This means you are free to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the software, including for commercial purposes.

Disclaimer

Please ensure you have the necessary permissions and security measures in place when deploying an SMTP server. Depending on your firewall configuration, you may be exposing this service to the internet.

Security Considerations

When deploying this SMTP server, please keep the following security considerations in mind:

• Ensure that your server is properly secured and that only authorized IPs can access the SMTP port.
• Use strong, unique passwords for your AWS credentials and keep them secure.
• Regularly update the Node.js runtime and all dependencies to their latest versions.
• Consider implementing additional authentication mechanisms for the SMTP server if needed.

Logging and Monitoring

The server logs information about received emails, webhook responses, and any errors that occur. The current logging setup includes:

• Console output for immediate visibility
• Daily rotating log files for persistent storage
• JSON formatting of log entries for easy parsing
• Timestamp inclusion for each log entry

Logging settings:

• Log files are stored in the logs/ directory
• Files are named application-YYYY-MM-DD.log
• Log files are rotated daily and compressed
• Maximum log file size is set to 20MB
• Log files are kept for 90 days

I recommend:

• Review log files regularly for errors or unusual patterns.
• Consider setting up log aggregation and analysis tools (e.g., ELK stack, Splunk).
• Implement alerts for critical errors or unusual activity patterns.
• Monitor system resources (CPU, memory, disk space) to ensure smooth operation.
• Set up uptime monitoring for the SMTP server and webhook endpoint.

System Requirements

• Node.js v18 or later
• Sufficient disk space for temporary storage of attachments before S3 upload and 90 days of logging.
• Outbound internet access for S3 uploads and webhook calls
• Inbound access on the configured SMTP port. (Default: 25, or 587 if SMTP_SECURE is set to 'true')

Troubleshooting

If you encounter issues:

1. Check the server logs for any error messages.
2. Ensure all environment variables are correctly set.
3. Verify that your AWS credentials have the necessary permissions for S3 operations.
4. Check that the webhook endpoint is accessible and responding correctly.
5. For attachment issues, verify that the MAX_FILE_SIZE setting is appropriate for your use case.

If problems persist, please open an issue on the GitHub repository with detailed information about the error and your setup.