Terraform Version

1.12.1

Terraform Configuration Files

# Configure TF remote-state
terraform {
  backend "s3" {
    bucket = "tf-state-terraform-myorg"
    key    = "org/prod/project/.terraform.tfstate"
    region = "somewhere" # Required, but in our case ignored
    #access_key = ENV_AWS_ACCESS_KEY_ID # set in env, not via tf
    #secret_key = ENV_AWS_SECRET_ACCESS_KEY # set in env, not via tf
    skip_credentials_validation = true
    skip_region_validation      = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_s3_checksum            = true
    endpoints = {
      s3 = "https://internal-s3.prod.m/" # our custom internal S3
    }
    use_path_style = true
  }
}

Today we migrated a project from using a consul backend to an s3 backend.

After running terraform-init -migrate-state, I immediately see issues with checksums showing up in the state even though the option to skip checksums is enabled.

$ terraform init
Initializing the backend...
╷
│ Error: Error refreshing state: 2 problems:
│
│ - Unsupported state file format: The state file could not be parsed as JSON: syntax error at byte offset 9.
│ - Unsupported state file format: The state file does not have a "version" attribute, which is required to identify the format version.

Sure enough, if I manually pull the state file, I can see things like this in the middle of my nice JSON state:

5000
{
    "version": 4,
...
          },
     
0
x-amz-checksum-crc32:jcuQew==

1d37ef
     "sensitive_attributes": [],
   ...
} 

I know this was an issue in earlier TF versions, but we successfully set up a project in terraform 1.8 using the S3 provider. This is the first project on TF 1.12 to get moved to S3 from consul.

I considered moving the 1.12 project to 1.8, but that seems like a lot of work, and I'm more posting this out of curiosity to see if anyone else has had this issue come up.

Also, I even tried fixing the state manually after migration (cleaning out checksum stuff) and uploading a cleaned state copy. This gets the project in a state to plan and apply successfully, but after apply, the state will be broken again.

Debug Output

...debug output, or link to a gist...

Expected Behavior

State file should have been created in S3 without any checksums.

Actual Behavior

Checksums are present in the statefile uploaded to S3 bucket.

Steps to Reproduce

1. Add config for S3 backend
2. Run terraform init -migrate-state
3. Run terraform init

Additional Context

This is not utilizing AWS' S3 implementation, rather a S3 variant solution, but the same bucket works with terraform 1.8.

References

• https://developer.hashicorp.com/terraform/language/backend/s3#skip_s3_checksum
• Maybe? S3 backend on nonAWS implementation (OCI) not working in 1.6.3 #34053

Generative AI / LLM assisted development?

N/A