Terraform empowers you to automate infrastructure provisioning across various cloud platforms, streamlining your workflow. But what if your needs extend beyond these built-in functionalities, and your infrastructure relies on a unique API? Here's where custom Terraform providers come to the rescue!
Imagine managing any service with an API directly through Terraform. Custom providers unlock this potential, granting you complete control and seamless integration within your existing workflows. In this blog, we'll explore why you might need a custom provider and then guide you through the creation process step-by-step.
What is a Terraform provider?
A Terraform provider is a Go binary plugin responsible for interacting with cloud APIs or self-hosted APIs. The provider implements a set of resources and data sources that Terraform can manage. In a nutshell, Providers bridge the gap between Terraform and external services by translating Terraform's configuration language (HCL) into API calls, and the target service understands and interprets the responses back to Terraform. There are built-in providers for many services, but custom providers unlock the potential to manage resources from any unique API.
Terraform can be broken down into two key components: Terraform Core and Terraform Plugins
- Terraform Core: This is the Terraform binary written in the GO programming language that communicates with plugins through RPC (remote procedure calls). Terraform core is responsible for reading the configuration, building the resource dependency graph, managing the resources' states, executing Terraform plans (e.g., Terraform plan, apply, destroy, refresh), and communicating with the custom Terraform providers/plugins.
- Terraform Plugins: These are separately executable binaries written in Go that communicate with Terraform Core over an RPC interface. Each plugin exposes an implementation for a specific application. Plugin is responsible for communicating with the actual application/service eg. initialization of REST libraries used to make API calls, authentication for application.
Writing a Terraform plugin
Let's write a simple custom Terraform plugin that will help configure the simple Python-based Flask application.
Prerequisites:
- Let's create a simple Flask web application that can manage the information for a single resource: a Person. This application exposes the API to perform the CRUD operations for the person. Please refer here to install person app.
- Install Go 1.13+. Please refer here.
- Install Terraform 0.13+. Refer here for installing Terraform.
Let's write a simple Terraform plugin to help configure the Person app.
1. Create a source code directory for Terraform plugin inside $GOPATH/src.
Example:
export GOPATH=$HOME mkdir –p src/person_terraform
The code structure will look like this:
person_terraform |____go.mod |____person | |____provider.go | |____resource_person.go | |____rest.go |____main.go
2. Write a rest client library to help interact with APIs to perform the CRUD for our Person application. Please refer here for rest.go.
3. Then write the provider.go, which will read the provider block from the Terraform plan and initialize the session for our application. Here, define the schema for the Terraform provider block. Along with it, map the available data sources and resources for the app you will implement in the following steps. Implement a single available resource from the application, i.e., a person.
package person
import (
"log"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
)
func Provider() *schema.Provider {
return &schema.Provider{
Schema: map[string]*schema.Schema{
"person_service_url": {
Type: schema.TypeString,
Optional: true,
Description: "Person API service url.",
},
"person_service_port": {
Type: schema.TypeString,
Optional: true,
Description: "Person API service port.",
},
},
ResourcesMap: map[string]*schema.Resource{
"person_person": resourcePerson(),
},
ConfigureFunc: providerConfigure,
}
}
func providerConfigure(d *schema.ResourceData) (interface{}, error) {
person_service_url := d.Get("person_service_url").(string)
person_service_port := d.Get("person_service_port").(string)
personsess, err := NewPersonSession(person_service_url + ":" + person_service_port)
log.Printf("Person session created for service url %s and port %s\n", person_service_url, person_service_port)
return personsess, err
}
4. Implement the resources we have just mapped to the provider.go. Map the methods to perform the CRUD operations for the resource. As there is only one resource (person) in the current application, create resource_person.go and implement CRUD to manage person.
package person
import (
"fmt"
"log"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
)
func ResourcePersonSchema() map[string]*schema.Schema {
return map[string]*schema.Schema{
"person_id": {
Type: schema.TypeString,
Computed: true,
},
"name": {
Type: schema.TypeString,
Optional: true,
},
"address": {
Type: schema.TypeString,
Optional: true,
},
"email": {
Type: schema.TypeString,
Optional: true,
},
"mobile_number": {
Type: schema.TypeString,
Optional: true,
},
}
}
func resourcePerson() *schema.Resource {
return &schema.Resource{
Create: resourcePersonCreate,
Read: ResourcePersonRead,
Update: resourcePersonUpdate,
Delete: resourcePersonDelete,
Schema: ResourcePersonSchema(),
}
}
func ResourcePersonRead(d *schema.ResourceData, meta interface{}) error {
log.Println("ResourcePersonRead")
client := meta.(*PersonSession)
var robj interface{}
id := d.Get("person_id").(string)
err := client.Get("api/person/"+id, &robj)
respMap := robj.(map[string]interface{})
person_id := respMap["person_id"].(float64)
id = fmt.Sprintf("%.0f", person_id)
d.SetId(id)
d.Set("person_id", id)
return err
}
func resourcePersonCreate(d *schema.ResourceData, meta interface{}) error {
log.Println("resourcePersonCreate")
client := meta.(*PersonSession)
person := make(map[string]string)
person["name"] = d.Get("name").(string)
person["address"] = d.Get("address").(string)
person["email"] = d.Get("email").(string)
person["mobile_number"] = d.Get("mobile_number").(string)
var pres interface{}
err := client.Post("api/person", &person, &pres)
respMap := pres.(map[string]interface{})
person_id := respMap["person_id"].(float64)
id := fmt.Sprintf("%.0f", person_id)
d.SetId(id)
d.Set("person_id", id)
return err
}
func resourcePersonUpdate(d *schema.ResourceData, meta interface{}) error {
log.Println("resourcePersonUpdate")
client := meta.(*PersonSession)
var robj interface{}
id := d.Get("person_id").(string)
err := client.Get("api/person/"+id, &robj)
respMap := robj.(map[string]interface{})
respMap["name"] = d.Get("name").(string)
respMap["address"] = d.Get("address").(string)
respMap["email"] = d.Get("email").(string)
respMap["mobile_number"] = d.Get("mobile_number").(string)
var pres interface{}
err = client.Put("api/person/"+id, &respMap, &pres)
putRespMap := pres.(map[string]interface{})
person_id := putRespMap["person_id"].(float64)
id = fmt.Sprintf("%.0f", person_id)
d.SetId(id)
d.Set("person_id", id)
return err
}
func resourcePersonDelete(d *schema.ResourceData, meta interface{}) error {
log.Println("resourcePersonDelete")
id := d.Get("person_id").(string)
client := meta.(*PersonSession)
err := client.Delete("api/person/" + id)
return err
}
5. Now, configure our newly created terraform plugin in main.go.
package main
import (
"github.com/hashicorp/terraform-plugin-sdk/v2/plugin"
"github.com/persontest/person_terraform/person"
)
func main() {
plugin.Serve(&plugin.ServeOpts{
ProviderFunc: person.Provider})
}
6. Build the Terraform plugin
go build -o terraform-provider-person_v99.9.1
- This command will create binary terraform-provider-person_v99.9.1
- Run following commands to check your os and architecture
go env GOOS go env GOARCH
- Check if the ${HOME}/.terraform.d/plugins/personterraform/person/person/99.9.1/<GOOS>_<GOARCH> path exists on your machine. If not then create it. Replace <GOOS> and <GOARCH> with the respective command output.
- Copy terraform-provider-person_v99.9.1 binary into directory ${HOME}/.terraform.d/plugins/personterraform/person/person/99.9.1/<GOOS>_<GOARCH>/
- Here we have considered 99.9.1 as a version of our plugin. Will use this version while writing our terraform plan.
Now the plugin is ready to use.
To test the provider:
- Create a Terraform plan for the provider
- Run Terraform commands
Create Terraform plan
Create declarative Terraform plans for our application using HCL (Hashicorp Configuration Language).
terraform {
required_providers {
person = {
# Terraform provider source path where we kept our plugin
source = "personterraform/person/person"
# Terraform provider plugin version (the one which we have used for binary directory location)
version = "99.9.1"
}
}
}
# Terraform provider name
provider "person" {
# Person service backend url
person_service_url = "0.0.0.0"
# Person service backend port
person_service_port = "5001"
}
# person_person is a resource name the one we have added in provider.go map
resource "person_person" "test_person" {
name = "John"
address = "California"
email = "john.test@mail.com"
mobile_number = "+12344554447"
}
Run Terraform commands
terraform init
This command will locate the binary based on the configuration we have provided in the required provider's block.
Now, you can run Terraform commands, Terraform plan, Terraform apply, and Terraform destroy commands to create, update, and delete the configuration as needed.
Conclusion
In conclusion, custom Terraform providers empower you to bridge the gap between Terraform and any API-driven service. This unlocks the potential to manage unique in-house systems, niche cloud offerings, and more – all within your existing Terraform workflows. You can publish Terraform provider by following the terraform documentation. We can write the custom Terraform Provider for any API based application. Please refer to this for the source code. Explore the possibilities and unleash the full potential of your infrastructure automation!
