on August 02, 2018 Data

Migrating Large Databases to Public Cloud - A Phased Approach

Subscribe to Email Updates

In this blog post, we discuss a phased approach that might be beneficial for large volume heterogeneous database migrations and highlight some important considerations. 

Potential use cases are businesses which might have requirements to vacate from on-prem data centre(s) in constrained timeframes; or allowing migration projects to make progress while developers work on converting application code to ANSI standard SQL that work on both source and target database engines.

Large volume (either in terms of size or rate of change) database migrations can be difficult.  With public cloud database migrations, we add network bandwidth limitations to the challenge.  

Consider a staged migration approach breaking down heterogeneous database migrations into two main phases.  This allows efforts to be concentrated on a smaller set of issues/constraints at each phase.

cloud_migration (1)Phase 1: Homogeneous migration to public cloud platforms.

This phase mainly focuses on moving the source data to the target public cloud platforms and leaving the database engine conversion component to phase 2.  This allows block-based change data capture (CDC) replication tools [1] such as Actifio, DBvisit Standby or NetApp to be utilised to combat network bandwidth limitation efficiently.  Block-based CDC tools are often faster in most cases and can provide a rollback strategy.  Actifio and NetApp particularly provide space optimised cloning capabilities on public cloud platforms which can help reduce cloud storage cost.

Many other migration tasks can be incorporated into this phase, e.g. removal of obsolete objects such as users/database links, DDL changes for character set conversion, cold data archival etc.

[1] Supportability: Ensure you verify supportability of your choice of data replication tools for your RDBMS.  For example, will you get support (timely responses) to issues such as replication failures?

Phase 2: Heterogeneous database migration once in the cloud. 

The second phase of the migration tackles on the database engine conversion task by using SQL based CDC replication tools such as HVR, Attunity, AWS Database Migration Service (DMS) and Oracle GoldenGate.  Notably, AWS DMS is a free tool with comparable features available to be used when migrating to AWS.

It is important to emphasis the potential drawbacks of this approach – e.g. integration testing effort overhead.  It is therefore paramount to review the pros and cons of this approach carefully with reference to your specific migration project. 

Let’s also take the opportunity to highlight the importance of performing the following tasks in database migration projects:

Source database discovery

An in-depth database discovery exercise will provide good coverage of required migration tasks and issues.  Some important but often overlooked issues are data validity, character set conversion (server and client, superset considerations), operational DDLs, database links, proprietary drivers etc.

Scope, requirements and constraints

Like many IT projects, the first step is to understand the required scope, requirements and constraints.  Ensure you involve all relevant teams (i.e. product owners, application support, application developers and DBAs) in this process. We typically ask the following questions to initiate the conversation:

  • What is the outage window the business can tolerate? How long does it take to stop & restart the consuming applications if applicable?
  • What are the consuming applications and the touchpoints?
  • Will there be active application development that require DDL changes during the migration?
  • Is there any static data that can be archived off or moved separately or purged?
  • Are there any independent groups of tables/data?
  • What are the client drivers used?
  • What is the available network bandwidth for the migration? Is this dedicated network bandwidth for the migration or is it shared with other workloads?  If shared, is it with production or dev/test workloads?
  • How much effort and time is required to implement the required changes on the target database engine?

Target database platform readiness

Ensuring the target database platform is designed, understood and setup properly to meet requirements (i.e. RTO & RPO, integration, performance, monitoring, alternative data stores etc.) is essential and this holds true to all database platforms including managed database (DaaS) solutions.

Database migrations can involve large amount of intense work and team co-ordination but proper planning will ensure desired outcomes are achieved.  

Photo by Markus Spiske on Unsplash

Aimee Lin

Aimee is a Data Management Practice Lead at OSS and has worked in the IT industry for 15 years, specialised in database consulting and architecture.

Need database migration assistance?