Server Setup

This document outlines how we set up our remote servers for running projects.

1. Provision a Server

We use DigitalOcean as our provider. Regardless of your choice, provision a new VPS using the latest Ubunutu LTS version. Add your ssh key AND Naomi’s ssh key in the setup process.

2. Set Up User

You should never run applications on root. SSH into the new VPS to prepare your user.

2.1. Creating the User

You’ll need to set a password for the root account first.

passwd

Once you have set a password, ensure that you have provided it to Naomi to store in the vault.

Create an nhcarrigan user for our organisation.

adduser nhcarrigan

Set a different password, and provide that to Naomi as well. For all of the user information, use the default blank values.

Add the new user to the sudoers file.

usermod -aG sudo nhcarrigan

Then sync the SSH keys so we can authenticate as that user.

rsync --archive --chown=nhcarrigan:nhcarrigan ~/.ssh /home/nhcarrigan

While you are there, set the timezone for the server to our business’ local timezone.

sudo timedatectl set-timezone America/Los_Angeles

3. Preparing For Web Requests

To prepare the server to receive web requests, you’ll need to follow a few steps.

3.1. SSL Certificate

Note

If the Firewall has been set up, you’ll need to temporarily allow port 80 for the certificate to generate.

We use LetsEncrypt to provision our SSL certificates. If it is not installed, install it with:

sudo snap install --classic certbot

Then link the snap to our usr directory.

sudo ln -s /snap/bin/certbot /usr/bin/certbot

Generate a certificate with:

sudo certbot certonly --standalone

And allow applications to read it:

sudo chmod -R a+rwx /etc/letsencrypt

When you need to renew the certificate:

sudo certbot renew

3.2. NGINX

All requests should be routed through NGINX. At no point should an application run directly on ports 80 or 443.

Install NGINX:

sudo apt-get install nginx

Edit the configuration file:

sudo emacs /etc/nginx/conf.d/server.conf

Use this template to set up a reverse proxy on the standard HTTPS port 443:

server {
    listen 443 ssl;
    server_name subdomain.domain.tld;
    ssl_certificate /etc/letsencrypt/live/subdomain.domain.tld/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/subdomain.domain.tld/privkey.pem;

    location / {
        proxy_set_header Host $host;
        proxy_pass https://127.0.0.1:port;
        proxy_redirect off;
    }
}

Validate that the config is correct with:

sudo nginx -t

If so, restart NGINX to apply the changes:

sudo systemctl restart nginx

4. Securing the Server

We have a minimum level of security that is required on ALL of our servers. This section should not be treated as the best effort, but as the minimal requirements to comply with our policies.

4.1. Firewall

We use ufw as our firewall. First, enable the SSH port.

sudo ufw allow "OpenSSH"

Then, allow the standard HTTPS port and deny the standard HTTP port.

sudo ufw deny http
sudo ufw allow https

Enable the firewall. You may get dropped from the SSH connection.

sudo ufw enable

4.2. Fail2Ban

We also use Fail2Ban to block IP addresses which fail to make requests too often.

Install the tool:

sudo apt-get install fail2ban

Configure the NGINX jail in /etc/fail2ban/jail.d/nginx-auth.conf:

[nginx-auth]
enabled = true
filter = nginx-auth
logpath = /var/log/nginx/access.log
maxretry = 3
findtime = 86400
bantime = 86400

Configure the NGINX filter in /etc/fail2ban/filter.d/nginx-auth.conf:

[Definition]
failregex = ^<HOST> - .* \[.*\] ".*" (4\d{2}) .*$

Because we use Cloudflare, you’ll need to grab the original IP for all requests. Start by creating a file to store Cloudflare’s IPs.

sudo touch /etc/nginx/cloudflare_ips.conf

Then create your script:

nano ~/update_cf_ips.sh

#!/bin/bash

# Create a temporary file
temp_file=$(mktemp)

# Download IPv4 ranges and format each line
curl -s https://www.cloudflare.com/ips-v4 | while read ip; do
    echo "set_real_ip_from $ip;" >> "$temp_file"
done

# Download IPv6 ranges and format each line
curl -s https://www.cloudflare.com/ips-v6 | while read ip; do
    echo "set_real_ip_from $ip;" >> "$temp_file"
done

# Add the real_ip_header directive
echo "real_ip_header CF-Connecting-IP;" >> "$temp_file"

# Replace the old file with the new one
sudo mv "$temp_file" /etc/nginx/cloudflare_ips.conf

# Test Nginx configuration
sudo nginx -t

# If the test is successful, reload Nginx
if [ $? -eq 0 ]; then
    sudo systemctl reload nginx
    echo "Nginx configuration updated and reloaded successfully."
else
    echo "Nginx configuration test failed. Please check your configuration."
fi

Make it executable and run it:

sudo chmod +x update_cf_ips.sh
sudo ./update_cf_ips.sh

If it runs as expected, set it up to run on a CRON.

sudo crontab -e

0 3 * * 1 ~/update_cf_ips.sh

Then, update the /etc/nginx/nginx.conf to use all of this new logic. This goes at the end of your http directive block.

# Look at the real IP, not the cloudflare IP.
include /etc/nginx/cloudflare_ips.conf;

log_format custom_format '$remote_addr - $remote_user [$time_local] '
                        '"$request" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent" '
                        '"$http_x_forwarded_for"';

access_log /var/log/nginx/access.log custom_format;

Confirm the NGINX configuration is correct:

sudo nginx -t

Then restart everything.

sudo systemctl restart nginx
sudo systemctl restart fail2ban

To view banned IPs:

sudo fail2ban-client status nginx-auth

And to unban them:

sudo fail2ban-client set nginx-auth unbanip <ip>

5. Running a Project

Now you are ready to start running the project.

5.1. Node.js

Most of our projects will run on Node. For a new machine, you’ll need to set that up.

We use nvm to manage Node versions. Fetch and run the install script:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash

The script will automatically update the .bashrc file to load nvm into the PATH. Reload that:

source ~/.bashrc

Install the long-term support Node version.

nvm install --lts

This should automatically set it as the default. When updating, be sure to remove any older versions!

Finally, install pnpm as the package manager.

npm i -g pnpm

5.2. PM2

All of our processes run with PM2 to allow for monitoring and auto-restarts. You’ll need to install it.

pnpm i -g pm2

To start a project, use this template:

pm2 start '<script>' --name '<name>'

Then run pm2 save to save the application list.