May 27, 2023

[CI/CD] Flow from Instance to DooD Deployment (Feat. Nginx, Jenkins)

Solidifying your deployment foundation with Docker and Nginx

23-06-16 Modified path for volume mounting clarity 23-07-05 Adjust settings for development convenience

Introduction

[CI/CD] Organizing automatic deployment of FastAPI server using Docker + Jenkins + Git Webhook

The reason for writing this post is that the last post above contains what I initially tried to learn, but I wanted to fix what needs to be fixed, polish what needs to be polished, and complete it in one neat post.

I've gained a certain amount of know-how and structured my deployment tasks since the last time and helped people around me deploy, so I hope I can keep a record of it and use it as a consistent process for the next time I deploy! If you follow this post in order, I don't think you'll find it difficult to deploy.

Additional things I've learned since part 1

Why implement out of Docker rather than in Docker?

By default, Docker discourages allowing commands to be executed on top of Docker containers

Docker in Docker grants powerful privileges to the Docker-installed container, which can be a security risk

Docker out of Docker allows you to run Docker commands through a socket share with Docker outside of the container.

What about using the Docker API?

Actually, that's what I used to solve the problem in part 1, and it worked. But if you have a successful build out of Docker, I don't think you'll need it much in your CI/CD process...

Something about the API that I thought was too vague to write about separately, so I just wrote it down.

Docker uses a REST API to communicate between the engine and the client, which is mostly unused, but is available.

  docker --tlsverify --tlscacert=/.docker/ca.pem \
  --tlscert=/.docker/cert.pem --tlskey=/.docker/key.pem \
  -H={server_ip}:{port_number} exec nginx \
  sed -i '4s/.*$/ server {container}:8000;/' /etc/nginx/sites-available/default.conf

This code modifies the contents of an Nginx configuration file located on an arbitrary server via a Docker Client with a TLS certificate. It communicates with the Daemon (Docker Engine) through the port on that server via the -H command, and the rest of the code is a procedure to secure access via the certificate.

If you leave a port open without using a certificate, someone can crash your container through that port!

Real world experience with container jamming. I left a common port used by Docker open without authentication security...

Securing via certificates is a must, and if for some unavoidable reason you can't, it's imperative to limit the commands that can be run through that port.

See [Docker API Documents] (https://docs.docker.com/engine/api/v1.43/) for more details!

Procedure for building an All in One deployment base

1. Access and update instances

ssh -i {secure shell key 경로} {사용자 이름}@{IP 주소}

sudo apt-get update
sudo apt-get upgrade

If you are setting up ubuntu, the username is often ubuntu.

2. Firewall settings

sudo ufw allow ssh    # ssh를 22로 대체할 수 있다.
sudo ufw allow http   # http를 80으로 대체할 수 있다.
sudo ufw allow https  # https를 443으로 대체할 수 있다.

sudo ufw enable 	  # 방화벽 가동
sudo ufw status 	  # 방화벽 상태 확인

If you are using the cloud, you will probably have to open the port in the cloud's own port security settings by default. UFW is Ubuntu's firewall, and if it is turned off when you initially set it up, you need to authorize the port and turn it on. With the above settings, you can turn on the firewall with the default HTTP(80), HTTPS(443), and SSH(22) ports open.

It is recommended to turn on the port allow setting first. If you turn on the firewall with SSH port 22 closed, you will not be able to connect to it.

3. Install Docker and Docker Compose

# Docker 설치

sudo apt-get update
sudo apt-get install ca-certificates curl gnupg

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Docker Compose 설치

sudo apt install docker-compose

Ubuntu Docker Installation Documents

As of May 2023, I've culled only what I needed from the Documents.

sudo usermod -aG docker $USER

This command allows you to use docker commands without sudo. If you don't include it, you will get /var/run/docker.sock: connect: permission denied.

4. Write docker-compose.yml

version: '3.8'

networks:
  {원하는 네트워크 이름}:
    driver: bridge

services:

  # Jenkins Container 설정
  jenkins:
    container_name: jenkins
    image: jenkins/jenkins:lts
    restart: unless-stopped
    ports:
      - "{원하는 포트 번호}:8080"
    volumes:
      - ./mount/jenkins:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock
    user: root
    environment:
      - TZ=Asia/Seoul
    networks:
      - {원하는 네트워크 이름}

We wrote the deployment pipeline tool as if it were utilizing Jenkins.

If you don't have a network to add, you can leave out the network part altogether and proceed with the volume bindings for the sockets to use out of Docker. You can set the time zone or additional volume bindings as needed.

5. Set up DNS for the domain you want to use

Different domain providers use different methods. You'll need to look at your domain provider's registration instructions and go through the process of registering the IP of the server you'll be using and associating it with your domain.

What's My DNS

It's not instantaneous, it takes time for DNS servers to reflect it. I've attached a site above where you can easily check how much it's spread.

6. Issuing a certificate using Let's Encrypt

1) docker-compose.yml

# Nginx 설정
  nginx:
    container_name: nginx
    image: nginx:latest
    restart: unless-stopped
    volumes:
      - ./mount/nginx/nginx.conf:/etc/nginx/nginx.conf 			# Nginx 설정을 위한 바인딩
      - ./mount/nginx/sites-available:/etc/nginx/sites-available
      - ./data/certbot/conf:/etc/letsencrypt				# 인증서 공유를 위한 바인딩
      - ./data/certbot/www:/var/www/certbot
    ports:
      - 80:80
      - 443:443
    environment:
      - TZ=Asia/Seoul
    networks:
      - {원하는 네트워크 이름}

  # Certbot 컨테이너 설정
  certbot:
    container_name: certbot
    image: certbot/certbot
    restart: unless-stopped
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    environment:
      - TZ=Asia/Seoul

Add this to docker-compose.yml and don't process it.

2) mount/sites-available/default.conf

server {
     listen 80;
     listen [::]:80;

     server_name my_domain.org; 

     location /.well-known/acme-challenge/ {
             allow all;
             root /var/www/certbot;
     }
}

file and include server_name followed by your domain.

3) mount/nginx.conf

user nginx;
worker_processes auto;
worker_priority 0;

pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 1024;
        multi_accept off;
}

http {

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        access_log /var/log/nginx/access.log main;
        error_log /var/log/nginx/error.log debug;

        gzip on;

        include /etc/nginx/sites-available/*;

}

This is a pretty basic setup based on what you'll find when you first install nginx.

4) Issue a validated certificate through Let's Encrypt

curl -L https://raw.githubusercontent.com/wmnnd/nginx-certbot/master/init-letsencrypt.sh -o init-letsencrypt.sh
chmod +x init-letsencrypt.sh
vi init-letsencrypt.sh			# 여기서 도메인과 메일 주소를 입력해야 함
sudo ./init-letsencrypt.sh

Get the file that will help you with the certificate issuance process externally and set the permissions. You can then enter your domain and email address into the file and run the file to issue the certificate. If the issuance fails, you can view the logs to find out why it failed, fix it, and try again.

7. Setting up Nginx for certificate renewal

# Nginx 설정
  nginx:
    container_name: nginx
    image: nginx:latest
    restart: unless-stopped
    volumes:
      - ./mount/nginx.conf:/etc/nginx/nginx.conf
      - ./mount/sites-available:/etc/nginx/sites-available
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    ports:
      - 80:80
      - 443:443
    environment:
      - TZ=Asia/Seoul
    networks:
      - {연결을 원하는 네트워크 이름}
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"

  # Certbot 컨테이너 설정
  certbot:
    container_name: certbot
    image: certbot/certbot
    restart: unless-stopped
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    environment:
      - TZ=Asia/Seoul
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"

Certificates issued by Let's Encrypt have a limited lifespan, but you can reissue them if they are nearing expiration. You can add reissue-related commands to the certbot and nginx containers to handle certificate renewals automatically. This will prevent nginx from failing to build the container with an error because the certificate has been issued.

8. Install Docker inside the Jenkins container

Added) I originally listed the commands directly, but realized it wasn't intuitive, so I fixed it!

# /jenkins-docker-install.sh

apt-get update
apt-get install ca-certificates curl gnupg

install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null

apt-get update
apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

I created a sh file with the code to remove the sudo command from the existing docker installation method. You can put this file inside via mount and run it for a clean, one-room installation!

9. Processing Docker Compose

version: '3.8'

networks:
  {네트워크 이름}:
    driver: bridge

services:

  # Jenkins Container 설정
  jenkins:
    container_name: jenkins
    image: jenkins/jenkins:lts
    restart: unless-stopped
    ports:
      - "{포트 번호}:8080"
    volumes:
      - ./mount/jenkins:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock
      - ./mount/jenkins-html:/var/lib/jenkins
      - ./jenkins-docker-install.sh:/jenkins-docker-install.sh
    user: root
    environment:
      - TZ=Asia/Seoul
    networks:
      - {네트워크 이름}

# Nginx 설정
  nginx:
    container_name: nginx
    image: nginx:latest
    restart: unless-stopped
    volumes:
      - ./mount/nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./mount/nginx/sites-available:/etc/nginx/sites-available
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    ports:
      - 80:80
      - 443:443
    environment:
      - TZ=Asia/Seoul
    networks:
      - {네트워크 이름}
    depends_on:
      - {선행 컨테이너}
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"

  # Certbot 컨테이너 설정
  certbot:
    container_name: certbot
    image: certbot/certbot
    restart: unless-stopped
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    environment:
      - TZ=Asia/Seoul
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"

After adding these Jenkins configurations, run

docker-compose -f {yml 파일 위치 및 파일명} up -d

command to bring up all the containers.

docker exec -it jenkins bin/bash
sh /jenkins-docker-install.sh

By installing Docker inside the Jenkins container, we can control the Docker Engine through the Jenkins internal Shell commands. We'll follow the existing Docker documentation, but the difference is that we won't be using sudo and we'll be installing Docker for Debian.

Optional

1. Set up Swap Memory

# swap memory 설정

sudo fallocate -l {설정 용량} /swapfile   # 설정 용량은 대체로 G 단위로 설정한다.
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# 재부팅 시에도 지속 사용을 원할 경우

sudo vi /etc/fstab 			# '/swapfile swap swap defaults 0 0' 내용을 해당 파일에 삽입

# swap 비활성화

sudo vi /etc/fstab 			# '/swapfile swap swap defaults 0 0' 내용을 제거
sudo swapoff -v /swapfile
sudo rm /swapfile

Depending on your instance performance and needs, you can set up swap memory to prevent instances from popping.

2. What to try if you get a Jenkins plugin installation error

<Solution 1 Access Jenkins Administration > Plugin Manager > Advanced settings > Update sites part Change https://updates.jenkins.io/update-center.json to http://updates.jenkins.io/update-center.json

<Solution 2> Install the plugin named skip-certificate-check

This is not a perfect solution, but we have seen some success with all of these solutions.

Closing remarks

I'm going to stop writing about Docker here, except for some framework deployments later on. I'll write more about deployments in the future if I get the chance, as it's a great feeling to see your work on a real network.