Bigtable migration: designing the wide-column layer

A Bigtable instance and its policies, assessed

An estate written for this page and assessed by StackToIT on 4 November 2026: one instance with two clusters, a table with two column families, two garbage collection policies and an application profile. The right pane is a verbatim excerpt of engineering/architect-report.md for the instance: the reason, the table it includes and the three warnings about its uncatalogued neighbours. The run wrote no Terraform for any of the four units.

resource "google_bigtable_instance" "telemetry" {
  name                = "telemetry"
  deletion_protection = true

  cluster {
    cluster_id   = "telemetry-ew3-a"
    zone         = "europe-west3-a"
    num_nodes    = 3
    storage_type = "SSD"
  }

  cluster {
    cluster_id   = "telemetry-ew3-b"
    zone         = "europe-west3-b"
    num_nodes    = 3
    storage_type = "SSD"
  }
}

resource "google_bigtable_table" "readings" {
  name          = "device-readings"
  instance_name = google_bigtable_instance.telemetry.name
  split_keys    = ["a", "m", "t"]

  column_family {
    family = "metrics"
  }

  column_family {
    family = "meta"
  }
}

resource "google_bigtable_gc_policy" "metrics_ttl" {
  instance_name = google_bigtable_instance.telemetry.name
  table         = google_bigtable_table.readings.name
  column_family = "metrics"

  max_age {
    duration = "720h"
  }
}

resource "google_bigtable_gc_policy" "meta_versions" {
  instance_name = google_bigtable_instance.telemetry.name
  table         = google_bigtable_table.readings.name
  column_family = "meta"

  max_version {
    number = 1
  }
}

resource "google_bigtable_app_profile" "ingest" {
  instance       = google_bigtable_instance.telemetry.name
  app_profile_id = "ingest"

  single_cluster_routing {
    cluster_id                 = "telemetry-ew3-a"
    allow_transactional_writes = false
  }

  ignore_warnings = true
}

- **Why:** STACKIT has no wide-column database service; row-key design and the HBase/Bigtable client API move to a self-operated store or a redesigned data model.
- **Includes:** `google_bigtable_table.readings`
- **WARNING** (uncatalogued-related): google_bigtable_app_profile.ingest (google_bigtable_app_profile) is linked to this unit but not in the knowledge base; its effect on the migration is not assessed.
- **WARNING** (uncatalogued-related): google_bigtable_gc_policy.meta_versions (google_bigtable_gc_policy) is linked to this unit but not in the knowledge base; its effect on the migration is not assessed.
- **WARNING** (uncatalogued-related): google_bigtable_gc_policy.metrics_ttl (google_bigtable_gc_policy) is linked to this unit but not in the knowledge base; its effect on the migration is not assessed.

• Four units, none drafted

  The table travels with the instance as a facet. The two garbage collection policies and the application profile are separate units classed MANUAL_REVIEW, because they are not in the engine's catalogue: nothing about them is assessed or drafted, and the run says so.

• Why the confidence is LOW here

  The same instance alone, in the twelve-resource estate of our service mapping page, came out as MEDIUM. Here the architect report lists the instance under the linked units of the uncatalogued garbage collection policies with a confidence cap of LOW. That tells you the policies and the profile need an architect's eyes, not that the instance is more or less portable.

What the run classifies

Data movement reads DECISION REQUIRED with the mechanism UNKNOWN: the source states neither the size of the data nor its change rate, and the run does not guess them.

The options, with their trade-offs

The first two are the targets the engine names. The last two are general practice the engine does not propose; they fit only when the workload is smaller or colder than a Bigtable deployment suggests.

Where each Bigtable construct goes

How to choose

1. 01

   1. Read the access pattern out of the code

   List the queries: point reads by key, prefix or range scans, time-ordered reads per device. Range scans on a composite key are what a wide-column store does best and what every other option must rebuild.

2. 02

   2. Get the numbers the run cannot give you

   Data size, change rate and write throughput are not in the Terraform. The data-movement file states that size and change rate are not in the source and that tolerated downtime and recovery point were not provided; collect all of them from the running system before you shortlist anything.

3. 03

   3. State the consistency your readers accept

   A multi-cluster Bigtable instance reads eventually consistent by default. If that was a deliberate choice, a single primary with a replica may do; if nobody noticed, say so now.

4. 04

   4. Write down retention as a requirement

   The 30-day and one-version rules in the example are requirements, not settings to copy. Name the retention in business terms, then express it in the target.

5. 05

   5. Count the operating cost

   A self-managed store buys the closest model and costs a team that can run it. A managed service lowers that cost and raises the rewrite. The run prices neither.

6. 06

   6. Plan the data movement last, not first

   Mechanism, window and validation depend on the target. Choose them after the store, with rollback in mind: keep Bigtable readable until the new store has served real traffic.

Questions architects ask

Sources

• Google Cloud documentation: Bigtable overview
• Google Cloud documentation: Bigtable garbage collection
• Google Cloud documentation: Bigtable schema design
• STACKIT documentation: MongoDB Flex