Amazon DynamoDB migration to STACKIT

A DynamoDB table with a global secondary index

From the published AWS Retail Store run. Left: the AWS source exactly as AWS publishes it. Right: the resource blocks StackToIT wrote; the review comments above each block in the file are left out here.

module "dynamodb_carts" {
  source  = "terraform-aws-modules/dynamodb-table/aws"
  version = "3.3.0"

  name     = "${var.environment_name}-carts"
  hash_key = "id"

  attributes = [
    {
      name = "id"
      type = "S"
    },
    {
      name = "customerId"
      type = "S"
    }
  ]

  global_secondary_indexes = [
    {
      name            = "idx_global_customerId"
      hash_key        = "customerId"
      projection_type = "ALL"
    }
  ]

  tags = var.tags
}

resource "stackit_mongodbflex_instance" "dynamodb_tables" {
  project_id = var.project_id
  name       = "dynamodb-tables"
  acl        = var.dynamodb_tables_allowed_cidrs
  version    = var.dynamodb_tables_version
  flavor = {
    cpu = var.dynamodb_tables_cpu
    ram = var.dynamodb_tables_ram
  }
  replicas = 3
  storage = {
    class = var.mongodbflex_storage_class
    size  = var.dynamodb_tables_storage_gb
  }
  backup_schedule = "0 2 * * *"
  options = {
    type                       = "Replica"
    point_in_time_window_hours = var.dynamodb_tables_pitr_hours
  }
}

• Each table becomes a collection

  The carts table becomes the collection retail-store-ecs-carts on one MongoDB Flex instance.

• Keys and indexes mapped

  The hash key id becomes _id or a unique index, and the global secondary index on customerId becomes a secondary index.

• Redundancy kept in the region

  DynamoDB replicates across zones; a replica set of three members keeps redundancy within the STACKIT region.

• Sizing and backup as inputs

  CPU, memory, storage and the point-in-time window are variables, and a daily backup is set as a starting point.

What your developers plan for

• Data access layer

  The application’s DynamoDB calls are rewritten against the MongoDB API.

• Data movement

  Export from DynamoDB to S3, then import with mongoimport. The plan says so, because nothing replicates the data live.

• Features to replace

  Capacity modes, DynamoDB Streams and conditional writes are named as not carried, with what to use instead, such as change streams or an outbox.

Data movement in five steps

From the data-move document the run wrote for the carts table. StackToIT moves no data itself: every step is for a person.

• 1 · Write freeze

  Owner: Migration Owner. List who writes, record the connection settings of every client, then stop the writers. The source stays untouched, so the way back needs nothing on the source.

• 2 · Export

  Owner: Data Architect. Export the table to S3 inside the freeze. The export needs point-in-time recovery, and the table in the run does not state it, so recovery is switched on first. Keep the manifest-summary.json: its itemCount feeds the final comparison.

• 3 · Conversion

  Owner: Data Architect. A script of your own unwraps the DynamoDB JSON type descriptors into BSON values and makes the key id the _id. Numbers of more than 34 significant digits do not fit a decimal128, so the script keeps them as strings or reports them.

• 4 · Final sync

  Owner: Data Architect. Load the documents with mongoimport or a driver script, then build the indexes. The draft creates the instance empty, and the instance user and access list are yours to set.

• 5 · Reconcile

  Owner: Data Architect. The document count must equal the itemCount less the items the conversion reported, a sample of keys must read the same on both sides, and the application’s most important read must return the same answer. Then the owner records GO or NO-GO.

What differs between the two stores

• Sets become arrays

  String, number and binary sets hold unique values in no order. An array does not, so code that relies on set operations changes.

• Index consistency

  A global secondary index is eventually consistent. A MongoDB index is updated with the write.

• Item size

  DynamoDB items hold up to 400 KB and a BSON document up to 16 MiB, so no size check is needed in this direction.

• Capacity and billing

  The pay-per-request billing mode and index projections are not carried. Instance size is chosen as CPU, memory and storage.

• Recovery window

  DynamoDB point-in-time recovery covers 1 to 35 days. MongoDB Flex documents point-in-time restore for the last 30 hours, which is the draft’s default of 30.

• Backup rhythm

  MongoDB Flex runs backups every 6, 8, 12 or 24 hours, with daily snapshot retention of 3 to 360 days.

Questions developers ask

Sources

• AWS Docs: What is Amazon DynamoDB?
• STACKIT Docs: MongoDB Flex
• STACKIT Docs: PostgreSQL Flex
• AWS Docs: Enable point-in-time recovery in DynamoDB
• STACKIT Docs: MongoDB Flex, backup, restore and clone