Aashiq Jacob 28 Apr 2022
Ways to host a MongoDB cluster on Kubernetes
28 Apr 2022

Ways to host a MongoDB cluster on Kubernetes

MongoDB is one of the most used database programs among developers. It is an open-source, general purpose, document-based, distributed NoSQL database server that is especially popular with JavaScript projects. It is instrumental in managing vast databases. In this blog, I will explain two ways of hosting a MongoDB cluster on Kubernetes.

Containerization provides developers flexibility, versatility, and support for many deployment environments. MongoDB helps Kubernetes in the automation of various critical aspects within containerized applications. Now, there are multiple ways to host a MongoDB cluster on K8s. However, I will discuss two of the easiest options:

  • Using Community Kubernetes Operator.
  • Using a custom Docker Image and Deployments.

Let's see how you can use these options in detail.

1. Using Community Kubernetes Operator

Kubernetes Operator provides an interface to manage third-party applications just like Kubernetes-native objects. MongoDB Kubernetes Operator helps in creating, configuring, and managing MongoDB StatefulSet. The MongoDB Operators are of two types viz., MongoDB Community Operator and MongoDB Enterprise Kubernetes Operator. Both possess different sets of features and requirements. Let’s look at the steps to set up a cluster using the Community Kubernetes Operator.

Community Kubernetes Operator

    • Install the Community Kubernetes Operator

      • Clone this repository:

        git clone https://github.com/mongodb/mongodb-kubernetes-operator.git

      • Run the following command to create cluster-wide roles and role-bindings in the default namespace:

        kubectl apply -f deploy/clusterwide

      • For each namespace that you want the Operator to watch, run the following commands to deploy a Role, RoleBinding, and ServiceAccount in that namespace:

        kubectl apply -k config/rbac --namespace

      • Install the Custom Resource Definitions.

        • - Invoke the following command:

          kubectl apply -f config/crd/bases/mongodbcommunity.mongodb.com_mongodbcommunity.yaml

        • - Verify that the Custom Resource Definitions installed successfully:

          kubectl get crd/mongodbcommunity.mongodbcommunity.mongodb.com

      • Install the Operator.

        • - Invoke the following kubectl command to install the Operator in the specified namespace:

          kubectl create -f config/manager/manager.yaml --namespace <my-namespace>

        • - Verify that the Operator installed successfully:

          kubectl get pods --namespace <my-namespace>

    • Generate Self-signed CA Certificate

      • Generate the CA key

        openssl genrsa -out rootca.key 4096

      • Configure rootca.cnf

        # For the CA policy 
[ policy_match ] 
countryName = match 
stateOrProvinceName = match 
organizationName = match 
organizationalUnitName = optional 
commonName = supplied 
emailAddress = optional 

[ req ] 
default_bits = 4096 
default_keyfile = rootca.pem    ## The default private key file name. 
default_md = sha256                           ## Use SHA-256 for Signatures 
distinguished_name = req_dn 
req_extensions = v3_req 
x509_extensions = v3_ca # The extentions to add to the self-signed cert 

[ v3_req ] 
subjectKeyIdentifier  = hash 
basicConstraints = CA:FALSE 
keyUsage = critical, digitalSignature, keyEncipherment 
nsComment = "OpenSSL Generated Certificate for TESTING only.  NOT FOR PRODUCTION        USE." 
extendedKeyUsage  = serverAuth, clientAuth 

[ req_dn ] 
countryName = Country Name (2 letter code) 
countryName_default = IN 

countryName_min = 2 
countryName_max = 2 

stateOrProvinceName = State or Province Name (full name) 
stateOrProvinceName_default = Pune 
stateOrProvinceName_max = 64 

localityName = Locality Name (eg, city) 
localityName_default = Pune 
localityName_max = 64 

organizationName = Organization Name (eg, company) 
organizationName_default = TestComp 
organizationName_max = 64 

organizationalUnitName = Organizational Unit Name (eg, section) 
organizationalUnitName_default = TestComp 
organizationalUnitName_max = 64 

commonName = Common Name (eg, YOUR name) 
commonName_max = 64 

[ v3_ca ] 
# Extensions for a typical CA 

subjectKeyIdentifier=hash 
basicConstraints = critical,CA:true 
authorityKeyIdentifier=keyid:always,issuer:always

      • Generate CA Certificate

        openssl req -new -x509 -days 36500 -key rootca.key -out rootca.crt -config rootca.cnf

      • Upload CA Certificate to Kubernetes Cluster

        kubectl create configmap ca-config-map --from-file=”~/ca.crt” --namespace <your-namespace> 
kubectl create secret tls ca-key-pair  --cert=”~/ca.crt”  --key=”~/ca.key” --namespace <your-namespace>

    • Create a MongoDB replica set in Kubernetes

      • Install Cert Manager.

      • Create the Cert Manager issuer, this will create the certificates required by the MongoDB replica set.

        cat <<EOF | kubectl apply -f - 
apiVersion: cert-manager.io/v1 
kind: Issuer 
metadata: 
name: ca-issuer-mongo 
  namespace: mongodb 
spec: 
  ca: 
    secretName: ca-key-pair 
--- 
apiVersion: cert-manager.io/v1 
kind: Certificate 
metadata: 
  name: cert-manager-certificate 
  namespace: mongodb 
spec: 
  secretName: mongodb-tls 
  issuerRef: 
    name: ca-issuer-mongo 
    kind: Issuer 
  commonName: "*.mongo-replicaset-svc.mongodb.svc.cluster.local" 
  dnsNames: 
  - "*.mongo-replicaset-svc.mongodb.svc.cluster.local" 
  - mongo-replicaset-0.com 
  - mongo-replicaset-1.com 
  - mongo-replicaset-2.com 
EOF

      • Use the Operator to create the MongoDB Replica set.

        cat <<EOF | kubectl apply -f - 
apiVersion: mongodbcommunity.mongodb.com/v1 
kind: MongoDBCommunity 
metadata: 
  name: mongo-replicaset 
  namespace: mongodb 
spec: 
  members: 3 
  type: ReplicaSet 
  version: "5.0.2" 
  replicaSetHorizons: 
  - horizon: mongo-replicaset-0.com:27017 
  - horizon: mongo-replicaset-1.com:27017 
  - horizon: mongo-replicaset-2.com:27017 
  security: 
    tls: 
      enabled: true 
      certificateKeySecretRef: 
        name: mongodb-tls 
      caConfigMapRef: 
        name: ca-config-map 

    authentication: 
      modes: ["SCRAM"] 
  users: 
    - name: admin 
      db: admin 
      passwordSecretRef: # a reference to the secret that will be used to generate the user's password 
        name: admin-password 
      roles: 
        - name: clusterAdmin 
          db: admin 
        - name: userAdminAnyDatabase 
          db: admin 
        - name: root 
          db: admin 
      scramCredentialsSecretName: admin-scram 
    - name: dumpUser 
      db: admin 
      passwordSecretRef: # a reference to the secret that will be used to generate the user's password 
        name: dumpuser-password 
      roles: 
        - name: readWriteAnyDatabase 
          db: admin 
      scramCredentialsSecretName: dumpuser-scram 
  additionalMongodConfig: 
    storage.wiredTiger.engineConfig.journalCompressor: zlib 
  statefulSet: 
    spec: 
      volumeClaimTemplates: 
        - metadata: 
            name: data-volume 
          spec: 
            storageClassName: mongodb-ssd-storage 
            accessModes: [ "ReadWriteOnce" ] 
            resources: 
              requests: 
                storage: 512Gi 
--- 
apiVersion: v1 
kind: Secret 
metadata: 
  name: admin-password 
  namespace: mongodb 
type: Opaque 
stringData: 
  password: password 
--- 
apiVersion: v1 
kind: Secret 
metadata: 
  name: dumpuser-password 
  namespace: mongodb 
type: Opaque 
stringData: 
  password: password 
EOF

      • Expose the MongoDB replica set to the internet by creating Load Balancer service.

        cat <<EOF | kubectl apply -f - 
apiVersion: v1 
kind: Service 
metadata: 
  name: mongo-replicaset-0 
  namespace: mongodb 
spec: 
  ports: 
  - port: 27017 
    protocol: TCP 
    targetPort: 27017 
  selector: 
    statefulset.kubernetes.io/pod-name: mongo-replicaset-0 
  type: LoadBalancer 
--- 
apiVersion: v1 
kind: Service 
metadata: 
  name: mongo-replicaset-1 
  namespace: mongodb 
spec: 
  ports: 
  - port: 27017 
    protocol: TCP 
    targetPort: 27017 
  selector: 
    statefulset.kubernetes.io/pod-name: mongo-replicaset-1 
  type: LoadBalancer 
--- 
apiVersion: v1 
kind: Service 
metadata: 
  name: mongo-replicaset-2 
  namespace: mongodb 
spec: 
  ports: 
  - port: 27017 
    protocol: TCP 
    targetPort: 27017 
  selector: 
    statefulset.kubernetes.io/pod-name: mongo-replicaset-2 
  type: LoadBalancer 
EOF

The IP address generated from the Load balancer SVC should be bound to the domain names mentioned in the replicaSetHorizons, for example: mongo-replicaset-0.com, mongo-replicaset-1.com, mongo-replicaset-2.com.

2. Using a custom Docker Image and Deployments

While using an operator to deploy a MongoDB cluster sure makes the cluster creation simpler, you lose a lot of control over the MongoDB cluster setup. For instance, to expand the Persistent Volume (PV) size, you won't be able to do it without downtime. You need to use self-signed certificates since no CA provides certificates for the local domain.

To tackle the challenges of controlling the MongoDB cluster in Kubernetes, you can use the official MongoDB Docker image with some modifications to make it production-ready. The official MongoDB Docker image out of the box doesn't have authentication enabled. It is pretty trickly to enable it after creating the container. You can use the official MongoDB image to create a custom image with authentication, log rotation, and Replica set initiated.

custom Docker Image and Deployments

    • Create Custom MongoDB image.

      Using the Dockerfile and MongoDB scripts makes it easy to create an Admin, a Database, and a Database User when the container is first launched. Create the following files in the exact location as mentioned in the header of the code snippet.

      • ~/MongoDB/Dockerfile

        FROM mongo:5.0.6 

ENV DEBIAN_FRONTEND noninteractive 
ENV DEBCONF_NONINTERACTIVE_SEEN true 

RUN apt-get -qqy update \ 
&& apt-get -qqy upgrade \ 
&& apt-get -qqy install -y netcat \ 
&& apt-get -qqy install rsyslog 

# Add scripts 
ADD scripts /scripts 
RUN chmod +x /scripts/*.sh 
RUN touch /.firstrun 

RUN (crontab -l ; echo "0 0 * * * bash /scripts/logs.sh") | crontab 

# Command to run 
ENTRYPOINT ["/scripts/run.sh"] 

# Expose listen port 
EXPOSE 27017 
EXPOSE 28017 

# Expose our data volumes 
VOLUME ["/data"]

      • ~/MongoDB/scripts/first_run.sh

        #!/bin/bash 
USER=$MONGODB_USERNAME 
PASS=$MONGODB_PASSWORD 
DB=$MONGODB_DBNAME 
ROLE=$MONGODB_ROLE 
PRIMARY_HOST=$HOST 

# Start MongoDB service 
/usr/bin/mongod --dbpath /data --nojournal & 
while ! nc -vz localhost 27017; do sleep 1; done 

# Create User only on the Primary Pod. 
echo "Creating user: \"$USER\"..." 

a="$HOSTNAME" 
b=${a%-*-*} 

if [ ! -f /data/admin-user.lock ]; then 
  sleep 60; 
  touch /data/admin-user.lock 
  if [ "$b" = "$PRIMARY_HOST" ]; then 
    mongo $DB --eval "db.createUser({ user: '$USER', pwd: '$PASS', roles: [ { role: '$ROLE', db: '$DB' } ] });" 
  fi; 

  /usr/bin/mongod --dbpath /data --shutdown 
fi; 

/usr/bin/mongod --dbpath /data --shutdown 
echo "========================================================================" 
echo "MongoDB User: \"$USER\"" 
echo "MongoDB Database: \"$DB\"" 
echo "MongoDB Role: \"$ROLE\"" 
echo "========================================================================" 

rm -f /.firstrun

      • ~/MongoDB/scripts/run.sh

        #!/bin/bash 
set -e 

# Initialize first run 
if [[ -e /.firstrun ]]; then 
    /scripts/first_run.sh 
fi 

# Startup cron for log rotation. 
cron 

echo "Starting MongoDB..." 

/usr/bin/mongod --replSet $REPLICA_ID --dbpath /data --bind_ip 0.0.0.0 --clusterAuthMode keyFile --keyFile /etc/secrets-volume/mongodb-keyfile --setParameter authenticationMechanisms=SCRAM-SHA-256 --auth --logpath /data/mongodb.log; 

exec "$@"

      • ~/MongoDB/scripts/logs.sh

        #!/bin/sh 

# Log directory 
LOGDIR=/log-volume 

# Maximum number of archive logs to keep 
MAXNUM=5 

#Log files to be handled in that log directory  
files=(mongodb.log) 

for LOGFILE in "${files[@]}" 
do  

## Check if the last log archive exists and delete it. 
if [ -f $LOGDIR/$LOGFILE.$MAXNUM.gz ]; then 
rm $LOGDIR/$LOGFILE.$MAXNUM.gz 
fi 

NUM=$(($MAXNUM - 1)) 

## Check the previous log file. 
while [ $NUM -ge 0 ] 
do 
NUM1=$(($NUM + 1)) 
if [ -f $LOGDIR/$LOGFILE.$NUM.gz ]; then 
mv $LOGDIR/$LOGFILE.$NUM.gz $LOGDIR/$LOGFILE.$NUM1.gz 
fi 

NUM=$(($NUM - 1)) 
done 

# Compress and clear the log file 
if [ -f $LOGDIR/$LOGFILE ]; then 
cat $LOGDIR/$LOGFILE | gzip > $LOGDIR/$LOGFILE.0.gz 
cat /dev/null > $LOGDIR/$LOGFILE 
fi 

done

Run Docker "build" in the MongoDB folder. Once built, you can tag and push the image to the container registry of your choice.

  • Deploy a MongoDB Cluster using Deployments.

    Let's create a three-node MongoDB cluster with one primary and two secondary nodes, which will be three different deployments in Kubernetes.

        • Create a Keyfile secret for the MongoDB cluster to communicate among the nodes. Make sure to base64 encode the key and replace "CHANGEME".

          cat <<EOF | kubectl apply -f - 
apiVersion: v1 
data: 
  mongodb-keyfile: CHANGEME 
kind: Secret 
metadata: 
  name: mongo-key 
  namespace: mongo-db 
type: Opaque 
EOF

        • Create PVC which would create a PV for each deployment.

          cat <<EOF | kubectl apply -f - 
--- 
apiVersion: v1 
kind: PersistentVolumeClaim 
metadata: 
  name: mongo-0-disk 
  namespace: mongo-db 
spec: 
  accessModes: 
  - ReadWriteOnce 
  resources: 
    requests: 
      storage: 512Gi 
--- 
apiVersion: v1 
kind: PersistentVolumeClaim 
metadata: 
  name: mongo-1-disk 
  namespace: mongo-db 
spec: 
  accessModes: 
  - ReadWriteOnce 
  resources: 
    requests: 
      storage: 512Gi 
--- 
apiVersion: v1 
kind: PersistentVolumeClaim 
metadata: 
  name: mongo-2-disk 
  namespace: mongo-db 
spec: 
  accessModes: 
  - ReadWriteOnce 
  resources: 
    requests: 
      storage: 512Gi 
--- 
EOF

        • Create three different deployments which would act as three different Mongo nodes.

          cat <<EOF | kubectl apply -f - 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  namespace: mongo-db 
  name: mongodb0 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: mongo-0 
  template: 
    metadata: 
      labels: 
        app: mongo-0 
    spec: 
      terminationGracePeriodSeconds: 10 
      containers: 
        - name: mongo 
          image: containerregistry.azurecr.io/mongodb:5.0.6 
          env: 
            - name: MONGODB_USERNAME 
              value: admin 
            - name: MONGODB_PASSWORD 
              value: password 
            - name: MONGODB_DBNAME 
              value: admin 
            - name: MONGODB_ROLE 
              value: root 
            - name: REPLICA_ID 
              value: mongoRS 
            - name: HOST 
              value: mongodb0 
          ports: 
            - containerPort: 27017 
          volumeMounts: 
            - name: mongo-key 
              mountPath: "/etc/secrets-volume" 
              readOnly: true 
            - name: mongo-persistent-storage 
              mountPath: /data 
      volumes: 
      - name: mongo-key 
        secret: 
          defaultMode: 0400 
          secretName: mongo-key 
      - name: mongo-persistent-storage 
        persistentVolumeClaim: 
          claimName: mongo-0-disk 
--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  namespace: mongo-db 
  name: mongodb1 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: mongo-1 
  template: 
    metadata: 
      labels: 
        app: mongo-1 
    spec: 
      terminationGracePeriodSeconds: 10 
      containers: 
        - name: mongo 
          image: containerregistry.azurecr.io/mongodb:5.0.6 
          env: 
            - name: MONGODB_USERNAME 
              value: admin 
            - name: MONGODB_PASSWORD 
              value: password 
            - name: MONGODB_DBNAME 
              value: admin 
            - name: MONGODB_ROLE 
              value: root 
            - name: REPLICA_ID 
              value: mongoRS 
            - name: HOST 
              value: mongodb0 
          ports: 
            - containerPort: 27017 
          volumeMounts: 
            - name: mongo-key 
              mountPath: "/etc/secrets-volume" 
              readOnly: true 
            - name: mongo-persistent-storage 
              mountPath: /data 
      volumes: 
      - name: mongo-key 
        secret: 
          defaultMode: 0400 
          secretName: mongo-key 
      - name: mongo-persistent-storage 
        persistentVolumeClaim: 
          claimName: mongo-1-disk 
--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  namespace: mongo-db 
  name: mongodb2 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: mongo-2 
  template: 
    metadata: 
      labels: 
        app: mongo-2 
    spec: 
      terminationGracePeriodSeconds: 10 
      containers: 
        - name: mongo 
          image: containerregistry.azurecr.io/mongodb:5.0.6 
          env: 
            - name: MONGODB_USERNAME 
              value: admin 
            - name: MONGODB_PASSWORD 
              value: password 
            - name: MONGODB_DBNAME 
              value: admin 
            - name: MONGODB_ROLE 
              value: root 
            - name: REPLICA_ID 
              value: mongoRS 
            - name: HOST 
              value: mongodb0 
          ports: 
            - containerPort: 27017 
          volumeMounts: 
            - name: mongo-key 
              mountPath: "/etc/secrets-volume" 
              readOnly: true 
            - name: mongo-persistent-storage 
              mountPath: /data 
      volumes: 
      - name: mongo-key 
        secret: 
          defaultMode: 0400 
          secretName: mongo-key 
      - name: mongo-persistent-storage 
        persistentVolumeClaim: 
          claimName: mongo-2-disk 
--- 
EOF

        • Create a service of type Load balancer to expose the MongoDB pods.

          cat <<EOF | kubectl apply -f - 
apiVersion: v1 
kind: Service 
metadata: 
  namespace: mongo-db 
  name: mongo-lb-0 
spec: 
  type: LoadBalancer 
  ports: 
  - protocol: TCP 
    port: 27017 
    targetPort: 27017 
  selector: 
    app: mongo-0 
--- 
apiVersion: v1 
kind: Service 
metadata: 
  namespace: mongo-db 
  name: mongo-lb-1 
spec: 
  type: LoadBalancer 
  ports: 
  - protocol: TCP 
    port: 27017 
    targetPort: 27017 
  selector: 
    app: mongo-1 
--- 
apiVersion: v1 
kind: Service 
metadata: 
  namespace: mongo-db 
  name: mongo-lb-2 
spec: 
  type: LoadBalancer 
  ports: 
  - protocol: TCP 
    port: 27017 
    targetPort: 27017 
  selector: 
    app: mongo-2 
--- 
EOF

    After a while, the service mongo-lb-0, mongo-lb-1, and mongo-lb-2 will be assigned a public IP. Then you can bind a domain name to the IP (optional). Suppose the domain names are mongo-DB-0.com, mongo-DB-1.com, and mongo-DB-2.com.

      • Once all the resources are created, access the primary pod through a bash shell.

        kubectl -n mongo-db exec -it mongodb0-68fb678849-tb558 -- bash

      • Create a Replica set after executing the Mongo shell within the primary pod. Use the Domain name or the Load balancer IP created while creating the SVC.

        $ mongo 
> db.getSiblingDB("admin").auth("admin", "password") 
1 
>rs.initiate({ _id: "mongoRS", version: 1,  
  members: [  
  { _id: 0, host: "mongo-db-0.com:27017" },  
  { _id: 1, host: "mongo-db-1.com:27017" },  
  { _id: 2, host: "mongo-db-2.com:27017" } ]}); 
mongoRS:PRIMARY>

        Now, you can connect to the Replica set using the database's connection string.

        mongosh
"mongodb://admin:password@10.0.0.1:27017,10.0.0.2:27017,10.0.0.3:27017/admin?authSource=admin&replicaSet=mongoRS"

    Even though using separate deployments for each node of the MongoDB Replica set is a time-consuming task, you can have complete control over the Replica set. Also, in the case of PV expansion, you can increase the PV size of each Secondary Node individually. All this with zero time while performing any operation on MongoDB.

    You can efficiently run a highly available MongoDB cluster on Kubernetes using any of the above methods. While each technique has its pros, it is up to the business use case on which cluster type is required. To sum it all up, you can quickly create and deploy the cluster using the Kubernetes Operator to deploy MongoDB. But you will lose many of the customization features. On the other hand, using custom Docker images and individual deployments is more complex than using the Operator. But you will get complete control over the database configurations and customization options for the configurations as per the use case. The choice is yours!

Related Blogs

Subscribe to our feed

select webform