SitecoreAI infrastructure with Terraform

When using a SaaS product like SitecoreAI there is a lot less infrastructure for us to take care of. Applications in different operating systems, databases, docker instances etc. all of this is managed with Sitecore and it is way simpler to start a new project.

However there is still some configuration and now we need to have multiple systems taking together instead. For SitecoreAI you will need to configure environments, you need ContextIDs and EditingSecrets for your head application. If you are using something else than the exact fit for eg. SitecoreAI and Vercel on repositories eg. using selfhosted GitHub Enterprise, GitLab, Gitea or if you just want your own pipeline, then you need to manage secrets to upload code to SitecoreAI.

You can click around in Sitecore Deploy to create the environments, copy values and it is “just” a one-time task so who cares? I am always risk-aware of those manual tasks…

Part of my professional Software Engineering practice is really that I need to create reproducible results and clearly describe any requirements for the next developer and the rest of my team.

Infrastructure as code

Infrastructure as Code (IaC) is a way to manage servers, networks, services, and other infrastructure using code instead of manual setup. Instead of clicking through menus or running commands by hand, you write files to describe the desired state. A tool reads these files and automatically sets everything up for you.

Like everything managed in code, there is a learning curve where it in the beginning seems more complicated to use a specialized tool instead of just clicking the button(s). However, in the long run, when creating multiple projects, supporting them for years, switching projects and people - it can really be a life saver.

With IaC we can get consistency, repeatability, and scalability across environments, eliminating the risks of human error and configuration drift. Teams can version-control infrastructure, track changes, and collaborate efficiently using familiar tools like Git. It can enable faster recovery from failures, cost savings by avoiding over-provisioning, and scalability to adapt to changing needs. Overall, IaC makes infrastructure more reliable, secure, and easier to manage.

Terraform and providers

I really like Terraform, it is widely known and supported and there is a lot of supported providers so we can within the same tool with shared configuration manage multiple systems and easily carry the relevant information from one system to another. In Terraform Registry you can currently find 6.099 providers, raging from large hyperscalers to specific systems. Whatever you are using currently, there is a high chance that there already is a provider to help you. There is a standardized way to create documentation for a provider and you can browse the documentation within the registry.

Terraform was launched by Hashicorp back in 2014. In 2023 the license changed to a “Business Source License” which made some worries in the community that it is not really open source and hardly managed by HashiCorp, which is now an IBM Company. However, from my point of view the strength is not really in the tool itself but the variety of providers. There is a maintained fork called OpenTofu with its own registry of providers which will ensure business continuity. Not everyone submits their provider to both registries, but there are currently 4.360 providers here.

But SitecoreAI was missing

However, there were no SitecoreAI provider!

Sitecore do care about automation scenarios and there are documented options. Apart from the UI for Sitecore Deploy there is also a CLI and there is Sitecore Deploy REST api documented with Swagger/OpenAPI.

As you probably have guessed, I do like to have everything together in Terraform, but there weren’t any provider for SitecoreAI.

I have now created a Terraform provider for Sitecore that lets you automate SitecoreAI with Terraform - at least with what the DeployAPI allows.

A small note on creating the provider

I attempted to create a provider almost two years ago. While I have been using Terraform for quite some time, but go is not the programming language I use everyday and there are quite some specific details on how to integrate with the Terraform ecosystem which is not the first step in a learning guide.

With the rise of AI I could now get significant help, I still read all code and have an opinion on how to structure things, but it was way more achievable. I have still not created unit tests for each and every method, but I guess that is not different if it was something I have written completely myself.

Using the SitecoreAI Terraform Provider

The terraform provider can of course create environments. Sitecore recently released decoupled deployments (which is implemented in the Deploy APi v2) and both models are supported in the provider.

You can create the legacy couple environment or the new cm_environment and eh_environment for the editing host.

If you already have created your environment with the SitecoreAI Deploy app, you might be more interested using the environment data source and combine with your landscape. Hereby you can eg. easily get the context id and configure Vercel environment variables with the Vercel provider:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

data "sitecoreai_project" "existing" {
  name = "XMC"
}

data "sitecoreai_environment" "prod" {
  project_id = data.sitecoreai_project.existing.id
  name       = "production"
}

resource "vercel_project_environment_variable" "contextid" {
  project_id = vercel_project.example.id
  key        = "SITECORE_EDGE_CONTEXT_ID"
  value      = data.sitecoreai_environment.prod.live_context_id
  target     = ["production"]
  comment    = "Sitecore Live context ID, managed by Terraform"
}

Maybe you have a project that needs to make requests to the Authoring API and hereby need an automation client. The provider can also help you with that, here it is using the Azure provider to set app settings:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

resource "sitecoreai_cm_client" "authoring" {
  name           = "Authoring for ${data.sitecoreai_environment.name}"
  description    = "CM automation client created via Terraform"
  project_id     = data.sitecoreai_environment.prod.project_id
  environment_id = data.sitecoreai_environment.prod.id
}

resource "azurerm_linux_web_app" "example" {
  name                = "example"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_service_plan.example.location
  service_plan_id     = azurerm_service_plan.example.id

  app_settings = {
    Sitecore__AuthoringUrl          = "https://${data.sitecoreai_environment.prod.host}/sitecore/api/authoring/graphql/v1/"
    Sitecore__AuthoringClientId     = sitecoreai_cm_client.authoring.client_id
    Sitecore__AuthoringClientSecret = sitecoreai_cm_client.authoring.client_secret
  }
}

Or maybe you need to run your own pipeline in eg. AzureDevOps and need a deploy client for that, you can configure the variable group in Azure DevOps from Terraform with the AzureDevOps provider:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

resource "sitecoreai_deploy_client" "deploy" {
  name        = "AzureDevOps deploy"
  description = "Deploy client managed by via Terraform"
}

resource "azuredevops_variable_group" "sitecoreai" {
  project_id   = azuredevops_project.example.id
  name         = "SitecoreAI"
  description  = "SitecoreAI variables managed via Terraform"
  allow_access = true

  variable {
    name  = "deploy_client_id"
    value = sitecoreai_deploy_client.deploy.client_id
  }

  variable {
    name         = "deploy_client_secret"
    secret_value = sitecoreai_deploy_client.deploy.client_secret
    is_secret    = true
  }

  variable {
    name  = "environment_id"
    value = data.sitecoreai_environment.prod.id
  }
}

Authentication

The SitecoreAI Terraform provider needs to authenticate to te Deploy API. You can create an organization level automation client and configure the provider with this (and run it in a pipeline) or the provider can use the authentication from the Sitecore CLI, so you run the usual dotnet sitecore cloud login and let the provider use your credentials.

Give it a try!

There are huge benefits of managing your infrastructure as code! With the SitecoreAI provider you can clearly express and manage the resources and the relations between them.

We are already using this on some of our projects and will include it on others in the upcoming future.

If you are using the SitecoreAI provider, I would be happy if you ping me, and of course it is completely open for you to inspect the code create issues and pull requests. Find the SitecoreAI Provider repository on GitHub

Open APIs are gold, I want more

I am really happy that the DeployAPI is clearly described and documented so that I could built this provider and I really believe it can give value to a lot of teams and projects. It is also well maintained without breaking changes and new features and changes are introduced in new versions of the API, so we are now at v4.

There are still plenty of configuration in Sitecore Portal that I would love to automate as well and Sitecore is really getting a good way with the Sitecore API Catalog (SitecoreAI DeployAPI is not listed here though)

Custom hostname for Edge is really infrastructure and involves DNS - and it is documented how to setup custom hostnames with API requests. However nether a DeployAPI automation client or an environment specific edge client have access to make those configurations. You have to log in to the portal and copy-paste your access_token. So we can support using CLI authentication, but it cannot be automated in a pipeline.

Users and app access involves a lot of manual management. Single-Sign-On connections involves creating both Azure Entra app registration and DNS records which could all be automated with Terraform. However, currently there are no public API to setup those.

With the introduction of Scoped Context IDs we can create specific context ids and have ones that can be exposed to the clients. Really nice. It also significantly increases the management overhead which really calls for automation, but there is no supported API.

Even though we can be smart with Terraform providers, we can only automate what is supported by the platform. Hopefully it will be there one day.