Introduction

In this document, you'll find useful and full steps that you need to take in order to properly set up and use your local development environment.

Installations

• Node.js (version >= 18.10.0)
• Docker Engine (and Docker Compose)
• Visual Studio Code (and "Prettier - Code formatter" extension)

Setup

1. Prerequisites:

• must have installed tools from the "Installations" section
• must have checked out / cloned source code from this repository

2. Open up the cloned repository/project/folder and run following commands in the given order, one by one:

   npm install
   npm run build
   

3. Both of the above commands should have completed successfully, and should have not caused any "package.json" or "package-lock.json" changes.

4. Now, navigate to root directory of this project, and run the following command:

   docker-compose up -d
   

5. This will spin up the whole "Docker stack" with other tools/containers used in the whole project, so the command must finish successfully. Running the following command should give a list of all relevant running containers:

   docker ps
   

6. Confirm that all of the containers have the status of something like "Up X minutes/hours", and that you have a total of 4 of them running and showing in the mentioned list.

7. Then make a copy of "example.env" and name it just ".env" in the root directory. For the environment values that are empty (16 of them), reach out to the person leading the backend development - currently, these values are the most secret ones and therefore not present or shared in the source. Also, you can start the service without these present or provided, just make sure environment variable validation and service/module usage is properly addressed, so it does not cause any start-up problems.

   SENDGRID_API_KEY
   TWILIO_KEY
   TWILIO_NUMBER
   TWILIO_SID
   FACEBOOK_APP_ID
   FACEBOOK_SECRET
   INSTAGRAM_APP_ID
   INSTAGRAM_SECRET
   GOOGLE_CLIENT_ID
   GOOGLE_SECRET
   GOOGLE_MAPS_ACCESS_KEY
   GOOGLE_RECAPTCHA_SECRET_KEY
   AWS_ACCESS_KEY_ID
   AWS_SECRET_ACCESS_KEY
   DISCORD_WEBHOOK_URL
   SENTRY_DSN
   

8. After the environment file setup, run following commands in the given order, one by one:

   npm run lint
   npm run test
   npm run start:app
   

9. All of the last commands should have completed successfully, and you should have all of the components now up and running properly. You can then use the links listed below this, in order to try and access all of these tools and apps.

10. There are two additional start commands, one for Administration UI, one for Command-line Interface:

    npm run start:admin
    npm run start:cli
    

11. If you're getting "JavaScript heap out of memory" error for any of the commands, prepend the following line to the problematic command execution, and it should work normally:

    cross-env NODE_OPTIONS=--max-old-space-size=8192 <command+params>
    

Localhost

• API
• Statistics
• AdminJS
• Adminer
• RabbitMQ

Features

• Basic Continuous Integration pipeline and Pull Request Template
• Basic Dockerfile and docker-compose.yml, for containerization purposes
• Useful decorators, filters, strategies, interceptors and guards
• Various smaller custom modules that hold their appropriate logic and files
• Register new, login to existing and delete existing account
• Activate account via link or PIN code
• Sign up with Google, Facebook or Instagram OAuth
• Base of 20 database entities/tables and appropriate repositories
• Base of 20 unit tests to demonstrate how to do them in future
• Auditing that keeps track of everything that happened on platform
• Turn various platform features on/off with ease, such as OAuth login, registering, mail sending, etc.
• Scheduler - periodic Cron jobs of caching data and invalidating tokens
• Swagger - API documentation, development usage
• Swagger-Stats - API and endpoint monitoring
• Terminus - various healthcheck endpoints
• Throttler - request rate limiting
• PassportJWT - token based authorization
• SendGrid - mail sending
• EventEmmitter - asynchronous event handling, such as SMS or mail sending
• TypeORM - managing and querying the business database
• Postgres - business database of choice
• Cloudwatch - AWS log sinking
• S3 - AWS file hosting and serving
• Handlebars - templating purposes, specifically custom mail templates
• Axios - HTTP request invocation
• Helmet - various HTTP security headers and features in a middleware
• i18n - multi-language support, or just a singular place for messages
• Pino - logging purposes, with "pretty" format available
• Redis - caching data and invalid tokens
• JOI - environment variable validation
• Commander - command-line interface
• ESLint - codestyle, not much defined
• Prettier - code formatting, basic on save
• Husky - pre-commit hooks, "lint" and "test" stages must pass before commit is created
• Google APIs - basic geolocation (Google Maps) and OAuth usages
• RabbitMQ - asynchronous, intra-service communication, or just general asynchronous request processing
• Sentry - error collecting and auditing
• Stripe - online payments
• Discord - message notifying
• Twilio - SMS sending
• AdminJS - plug-n-play administration UI
• Jest - unit testing

Unused

This section contains a small list of all previously live tested, but currently unused services and/or methods, that should be actually used in the projects that are made out of this template. This list also contains suggestions on where and when to potentially use these. On the other hand, you could also simply remove them completely from your project, if they are deemed unnecessary. As for everything else, it is used and put in place with purpose, however if any issues are encountered, please report them.

Services

• maps.service.ts - Google Maps implementation, use if in need of location or similar geopositional services
• discord.service.ts - Discord message sending implementation, simply send a message to a channel to notify whoever is needed and in there
• rabbit-mq.service.ts - RabbitMQ structures and exchanging logic, for intra-service communication(s)
• sms.service.ts - Twilio SMS sending service, for 2FA/MFA or other mobile phone related services
• sentry.service - error auditing with Sentry

Pipes / Decorators / Guards

• parse-int.pipe.ts - number parsing pipe, can be used if numbers are in query or path parameters, for actual conversion
• validation.pipe.ts - validation pipe that checks if the properties are of a proper type
• api-search-pagination.decorator.ts - an example combination of Swagger decorators to support pagination, filtering and sorting of an endpoint
• admin-jwt-auth.guard.ts - a JWT authorization guard that should be used for Administration related API endpoints
• cluster.ts - an example implementation for starting a "multi-threaded" backend API service

user.service.ts

• setStatus - could be used in Administration application, to set a user's status, or in some kind of Cron job that monitors users activities and updates accordingly
• updateUserType - could be used in Administration application, to change a user's type, or in
• changePassword - can and should be used in both Administration application and backend API, to expose the password changing functionality to all users

ban.service.ts

• addBan - add a ban that prevents a user/phone/IP/email from entering/using/do whatever on the backend, should be a part of Administration application, or some kind of spam detection and automatic banning
• removeBan - remove previously added ban, should be a part of Administration application or some batch processes that unban users based on some terms

stripe.service.ts

• getPaymentIntentFee - this can be invoked to get an exact fee amount that Stripe has taken for a given transaction

CI/CD

In order to use the following plug-n-play CI/CD pipeline for Node.js/Nest.js applications, you have to first set up your AWS account, EC2 instance and ECR registry+repository, as well as Actions Variables and Actions Secrets here in GitHub (found under Repository/Settings/Secrets and Variables/Actions). Make sure to add them 1 by 1 and that they have proper values.

Variables:

AWS_REGION=us-east-1
ECR_REPOSITORY=<repository>
BUILD_CMD=build
TEST_CMD=test
NODE_VERSION=18.x
RUN_LINT=true
RUN_TEST=true

Secrets:

AWS_ECR_REGISTRY=<registry>
AWS_ACCESS_KEY_ID=<key>
AWS_SECRET_ACCESS_KEY=<secret>
AWS_EC2_HOST=<hostname>
AWS_EC2_USER=<username>
AWS_EC2_KEY=<private-key>

Pipeline:

name: Continuous Integration / Continuous Deployment
on:
  push:
    branches: [main, master, develop]
  pull_request:
    branches: [develop]
jobs:
  integrate:
    name: Install, Build, Lint & Test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Setup
        uses: actions/setup-node@v3
        with:
          node-version: ${{ vars.NODE_VERSION }}
          cache: "npm"
      - name: Install
        run: npm ci
      - name: Build
        run: npm run $CMD
        env:
          CMD: ${{ vars.BUILD_CMD }}
          NODE_OPTIONS: "--max_old_space_size=8192"
      - name: Lint
        if: ${{ vars.RUN_LINT == 'true' }}
        run: npm run lint
      - name: Test
        if: "!contains(github.event.head_commit.message, '--skip-tests') && vars.RUN_TEST == 'true'"
        run: npm run $CMD
        env:
          CMD: ${{ vars.TEST_CMD }}
          NODE_OPTIONS: "--max_old_space_size=8192"
  build:
    needs: integrate
    name: Build Docker Image
    runs-on: ubuntu-latest
    if: github.event_name != 'pull_request'
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ vars.AWS_REGION }}
      - name: Login to AWS ECR
        uses: aws-actions/amazon-ecr-login@v1
      - name: Build & Upload Image to AWS ECR
        run: |
          docker build --build-arg ENTRY=app --build-arg PORT=3000 --build-arg BUILD_CMD=$CMD  -f ./Dockerfile -t $ECR_REGISTRY/$ECR_REPOSITORY:latest-$BRANCH .
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:latest-$BRANCH
        env:
          CMD: ${{ vars.BUILD_CMD }}
          ECR_REGISTRY: ${{ secrets.AWS_ECR_REGISTRY }}
          ECR_REPOSITORY: ${{ vars.ECR_REPOSITORY }}
          BRANCH: ${{ github.ref_name }}
  deploy:
    needs: build
    name: Deploy Docker Image
    runs-on: ubuntu-latest
    if: github.event_name != 'pull_request'
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Trigger Deployment
        uses: appleboy/ssh-action@v0.1.4
        env:
          ECR_REGISTRY: ${{ secrets.AWS_ECR_REGISTRY }}
          ECR_REPOSITORY: ${{ secrets.AWS_ECR_REPOSITORY }}
          ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          REGION: ${{ secrets.AWS_REGION }}
        with:
          host: ${{ secrets.AWS_EC2_HOST }}
          port: 22
          username: ${{ secrets.AWS_EC2_USER }}
          key: ${{ secrets.AWS_EC2_KEY }}
          envs: ECR_REGISTRY,ECR_REPOSITORY,ACCESS_KEY_ID,SECRET_ACCESS_KEY,REGION
          script: |
            aws --profile default configure set aws_access_key_id "$ACCESS_KEY_ID"
            aws --profile default configure set aws_secret_access_key "$SECRET_ACCESS_KEY"
            aws --profile default configure set aws_default_region "$REGION"
            aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $ECR_REGISTRY
            docker pull $ECR_REGISTRY/$ECR_REPOSITORY:latest-$BRANCH
          #  "docker-compose up -d" after this line in order to restart container with the latest image
      - name: Cleanup history
        run: history -c && history -w

AWS Setup

Oman Reference and Instructions

EC2 Setup

1. Configure AMI instance through the AWS Web UI and download MobaXterm for SSH-ing
2. Generate and download new key pair
3. Create new security group with newly generated key pair
4. Enable access from all IPs for that security group
5. Add security group to EC2 instance
6. Install Docker engine on Ubuntu as "ubuntu" user (root, but not the "root"):

   sudo apt update
   sudo apt upgrade -y
   sudo apt install unzip docker.io
   docker --version
   sudo systemctl start docker
   sudo systemctl enable docker
   sudo systemctl status docker

7. Install AWS CLI:

   curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
   unzip awscliv2.zip
   sudo ./aws/install

8. Install Docker Compose on Ubuntu:

   sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
   sudo chmod +x /usr/local/bin/docker-compose
   sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
   docker-compose --version

9. Create Docker group and user:

   sudo adduser --disabled-password --home /docker-home docker-user
   sudo su - docker-user
   mkdir .ssh
   chmod 700 .ssh
   touch .ssh/authorized_keys
   ssh-keygen -y -f ./docker-user.pem -> copy public key
   vi .ssh/authorized_keys # then paste public key -> CTRL + O -> SHIFT + . -> wq! -> Enter
   logout
   sudo groupadd docker
   sudo usermod -aG docker docker-user
   sudo newgrp docker