How to set up High Availability Vault with Consul backend?
Usable security, trusted delivery, and infrastructure independence are the three pillars of Docker container security that result in safer apps. Another important aspect of this security is that apps must communicate with other apps and systems in safe and secure ways. Docker secrets is a container-based solution that ensures secret distribution within container platforms. A secret literally means anything that is not meant to be seen or known by unauthorized personnel and demands tokens, API keys, passwords, and certificates, among others, to unlock. Docker secrets is an integral part of container security that strengthens its “trusted delivery” component. In this blog, I am going to talk about how you can strengthen the security of your Docker containers using Docker secrets and a HashiCorp duo of Vault and Consul. I will focus mostly on achieving Vault High Availability with Consul Cluster. But before moving toward that, let us first have a look at Vault and Consul as individual components of the system that we are going to build.
Vault
Vault provides high-level policy management, secret leasing, audit logging, and automatic revocation to protect sensitive information using UI, CLI, or HTTP API. All the secrets stored in it have an associated lease to enable key usage auditing, perform key rolling, and ensure automatic revocation. It provides multiple revocation mechanisms to give operators a clear “break glass” procedure in case of a potential compromise. It consists of a unified interface to store secrets while providing tight access control and recording a detailed audit log. It can be deployed to practically any environment and does not require any special hardware such as physical HSMs (Hardware Security Modules). Vault is an open-source tool, and you can get it here.
Consul Storage Backend
The Consul storage backend is used to retain Vault's data in Consul's key-value store. In addition to providing durable storage, the inclusion of this backend will also register Vault as a service in Consul with a default health check. The Consul storage backend supports high availability and is backed by HashiCorp.
Consul is a tool that can aid in service discovery or, basically, the registration of apps to your system. It also can be used for a key/value configuration storage system and can store app authentication, configuration flags, etc. When Vault is coupled with a Consul back-end, it makes your key/value configuration more secure. The tool itself requires authentication to be able to access all the secret goodies that are stored inside.
The token management system of Vault allows you to create temporary access tokens to Vault as per time and usage. Hence, applications can have only temporary access to those secrets. For example, a lot of applications need a username and password for some DB authorization parameters upon startup. These credentials are then stored in the memory. By creating limited authentication tokens, you can prevent the application from requesting credentials while it is running.
Consul is a distributed, highly available, and data-center-aware service discovery & configuration system. It has a flexible and powerful interface that allows clients to view the current state of the infrastructure, including service & node discovery mechanisms, a tagging system, health checks, consensus-based election routines, system-wide key/value storage, and more. With Consul, you can easily build a new level of awareness into your applications and services. Now, let us see how to achieve high availability using Consul and Vault.
High Availability
A Vault cluster is a highly-available unit of deployment within a data center. It is recommended to have three Vault servers with a Consul storage backend. With this configuration, during a Vault server outage, failover is addressed immediately without human intervention. To learn more about setting up your Vault servers in HA mode, you can refer the Vault HA with Consul guide. High availability is achieved using Performance Standby Nodes and data locality across datacenters that are exclusively available in Vault Enterprise.
Now, let's build new images and spin up containers:
###Docker Network
$ docker network create -d overlay --attachable
###Deploy Stack
$ docker stack deploy -c docker-stack.yml
###Remove Stack
$ docker stack rm
###List Docker Services
$ docker service ls
###Access Vault UI
open http://:8200 URL in the browser
Network Connectivity Details
The following is a sample configuration block for Vault HA with Consul cluster:
docker-stack.yml
version: '3.6'
services:
server-bootstrap:
image: consul:1.2.1
ports:
- 8500:8500
command: "agent -server -bootstrap -ui -client 0.0.0.0 -bind '{{ GetInterfaceIP \"eth0\" }}'"
server:
image: consul:1.2.1
command: "agent -server -retry-join server-bootstrap -client 0.0.0.0 -bind '{{ GetInterfaceIP \"eth0\" }}'"
deploy:
replicas: 2
depends_on:
- server-bootstrap
client:
image: consul:1.2.1
command: "agent -retry-join server-bootstrap -client 0.0.0.0 -bind '{{ GetInterfaceIP \"eth0\" }}'"
deploy:
replicas: 2
depends_on:
- server-bootstrap
vault:
depends_on:
- consul
image: vault
container_name: "vault"
hostname: "vault"
restart: always
cap_add:
- IPC_LOCK
ports:
- 8200:8200
environment:
VAULT_LOCAL_CONFIG: '{"backend": {"consul": {"address": "http://server-bootstrap:8500", "path": "vault"}}, "listener": {"tcp": {"address": "0.0.0.0:8200", "tls_disable": "1"}}, "default_lease_ttl": "24h", "ui":true, "max_lease_ttl": "720h", "disable_mlock":true}'
VAULT_ADDR: "http://0.0.0.0:8200"
command:
- server
ulimits:
nproc: 65535
cap_add:
- ALL
privileged: true
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
networks:
default:
external:
name: network_name
Setup Consul Client Agents on Vault Nodes
Let’s set up the initial set of master keys that you may need in case of emergency -
The above screenshot shows the first screen that appears after the initial deployment. Key Shares is basically the number of keys that you want and Threshold is the number of keys needed to be entered to unseal your vault. At least 5 Key Sharez and 2 Key Thresholds are recommended for good security. Also, make sure to download this content securely somewhere. It also provides the root authentication token.
Vault has “seal” and “unseal” features. The auto unseal feature is available only in the Enterprise version. In case of a compromise, this can be your “kill switch”. Seal your vault, and everything gets restricted.
###Access Consul UI
Open http://:8500 URL in the browser
Since we used Consul as the Vault backend, all Vault data is stored in Consul KV, and everything is encrypted.
Basic Key-Value Example
Writing Static credential to the vault using CLI:
vault write secret/staticcreds/username my-value=N7OVkOH6yQGfkGciySsek9knZfSzKj
Read Static credential from vault using CLI by key name:
vault read secret/staticcreds/username
Once you’ve completed this, Vault will be up and running in HA!
[root@host1 ~]# vault status Sealed: false Key Shares: 5 Key Threshold: 3 Unseal Progress: 0 Unseal Nonce: Version: 0.7.3 Cluster Name: vault-cluster-975049e2 Cluster ID: 6efde828-3c74-a994-ed47-32e5d8451f3d High-Availability Enabled: true Mode: active Leader: http://host1:8200 [root@host2 ~]# vault status Sealed: false Key Shares: 5 Key Threshold: 3 Unseal Progress: 0 Unseal Nonce: Version: 0.7.3 Cluster Name: vault-cluster-975049e2 Cluster ID: 6efde828-3c74-a994-ed47-32e5d8451f3d High-Availability Enabled: true Mode: standby Leader: http://host2:8200
Well, that was a quick look at how you can achieve Vault High Availability with Consul Cluster. To gain a further in-depth understanding of the topic, I am giving some additional references in the links mentioned below:
I hope this piece added value to your understanding of the topic. If you have any queries or suggestions, feel free to write them down in the comments section.
