# Migrating data from Azure Blob Storage to AWS S3 with AWS DataSync

Migrating data between different cloud storage services can be a complex and time-consuming task. In this post, we'll explore how to efficiently transfer data from Azure Blob Storage to AWS S3 using [AWS DataSync](https://docs.aws.amazon.com/datasync/latest/userguide/what-is-datasync.html), a powerful and secure solution for simplifying the migration process.

> AWS DataSync is an online data movement and discovery service that simplifies data migration and helps you quickly, easily, and securely transfer your file or object data to, from, and between AWS storage services.

## Concepts

* **Agent**: An agent is a self-hosted virtual machine (VM) that is utilized for reading or writing data from the storage system.
    
* **Location**: A location specifies where you want to copy data from or to (source and destination).
    
* **Task**: It specifies a source and destination location, along with the configuration that determines how data is transferred.
    
* **Task Execution**: This is an individual run of a task.
    

## Pre-requisites

* Ensure you have an [IAM User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console) with programmatic access.
    
* Ensure you have an [Azure Account](https://azure.microsoft.com/en-us/free/)
    
* Install [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli)
    
* Install [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
    
* Install [AzCopy](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10#download-and-install-azcopy)
    
* The following Windows features will need to be enabled on your local machine:
    
    1. Hyper-V Module for Windows PowerShell
        
    2. Hyper-V Services
        

## Preparing the source: Azure Blob Storage

Log into Azure by running `az login` and create and storage account using the following commands:

```bash
az group create -l eastus -n MyResourceGroup
az storage account create --name 24f12674a14143d --resource-group MyResourceGroup --location eastus --sku Standard_ZRS --encryption-services blob
```

Next, create a container within the storage account:

```bash
$assignee = az ad signed-in-user show --query id -o tsv
az role assignment create --role "Storage Blob Data Contributor" --assignee $assignee --scope "/subscriptions/e759b3f9-6ac3-4f9d-b479-1ba4471235cd/resourceGroups/MyResourceGroup/providers/Microsoft.Storage/storageAccounts/24f12674a14143d"
az storage container create --account-name 24f12674a14143d --name mycontainer --auth-mode login
```

Upload a file within:

```bash
New-Item myFile.txt
az storage blob upload --account-name 24f12674a14143d --container-name mycontainer --name myFile.txt --file myFile.txt --auth-mode login
```

## Preparing the destination: AWS S3

Run the following command to create a bucket:

```bash
aws s3api create-bucket --bucket my-destination-989282 --region us-east-2 --create-bucket-configuration LocationConstraint=us-east-2
```

## Creating the Agent

To create a DataSync Agent on Azure, first, download its VM image from AWS. Navigate to AWS DataSync, create a new agent, and select `Microsoft Hyper-V` as Hypervisor:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1687716104748/70562ed3-8aea-430c-9f73-9f24f288bb4e.png align="center")

Extract the VHDX image file(VHDX is a file format that describes a virtual hard disk) and convert it to [VHD](https://learn.microsoft.com/en-us/previous-versions/windows/desktop/legacy/dd323654(v=vs.85)) format:

```bash
Convert-VHD -Path aws-datasync-2.0.1678813931.1-x86_64.xfs.gpt.vhdx -DestinationPath aws-datasync-2016788139311-x86_64.vhd -VHDType Fixed
```

Let's [upload the VHD file to managed disk](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/disks-upload-vhd-to-managed-disk-cli) on Azure (run `ls` to obtain the VHD file size in bytes):

```bash
az disk create -n MyDisk -g MyResourceGroup -l eastus --upload-type Upload --upload-size-bytes 85899346432 --sku standard_lrs
```

Generate a writable shared access signature (SAS):

```bash
az disk grant-access -n MyDisk -g MyResourceGroup --access-level Write --duration-in-seconds 86400
```

Upload the VHD to the [managed disk](https://learn.microsoft.com/en-us/azure/virtual-machines/managed-disks-overview):

```bash
./AzCopy.exe copy "aws-datasync-2.0.1686143940.1-x86_64.vhd" "https://md-impexp-r3jbg4mth1f3.z38.blob.storage.azure.net/c5qllmmcgfh5/abcd?sv=2018-03-28&sr=b&si=2bb3a8bf-68d5-444b-8fee-a704d4223128&sig=MMaPZAc6aibl6KV3tq%2FVeiXQ94nZj%2BPMZXA1WaxSG7A%3D" --blob-type PageBlob
```

Revoke the SAS (don't forget this step):

```bash
az disk revoke-access -n MyDisk -g MyResourceGroup
```

Create the VM using the managed disk (the VM must meet the requirements specified [here](https://docs.aws.amazon.com/datasync/latest/userguide/agent-requirements.html)):

```bash
az vm create --resource-group MyResourceGroup --location eastus --name myVM --size Standard_E4as_v4 --os-type linux --attach-os-disk myDisk
```

Navigate to our VM in the Azure Portal and Enable the [boot diagnostics](https://learn.microsoft.com/en-us/azure/virtual-machines/boot-diagnostics):

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1687718261644/f90119e5-fa4b-4c52-ac65-3c76cf4b6f4d.png align="center")

Navigate to the [Serial Console](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-overview), and enter the [credentials](https://docs.aws.amazon.com/datasync/latest/userguide/deploy-agents.html#create-hyper-v-agent): login `admin` and password `password`:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1687718420168/a6615a25-d1a4-43c4-b89a-083289cecdaa.png align="center")

Enter `0` to get the activation key, and then follow the instructions:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1687718708970/decd66a5-5efe-46e1-979e-2dd1da19fd97.png align="center")

Return to the AWS Console and create the agent using the activation key (and the same service endpoint type):

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1687719059588/fb3398dc-543b-4df8-8d04-1e5cc1d897ac.png align="center")

## Setting up the source Location

Navigate to the Azure Portal to get a SAS token for the storage account:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1687719469193/8e417121-a533-4adb-a835-73fa9e64091c.png align="center")

Navigate to the AWS Console and create a `Microsoft Azure Blob Storage` location:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1687719689190/51f70b7f-264c-4fe6-93e3-3b437658f5b8.png align="center")

## Setting up the destination Location

Create an `Amazon S3` location (autogenerate the IAM role):

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1687719990562/78a89ec4-7058-4269-8ae0-be761664dcda.png align="center")

## Creating a Task

Select the source:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1687720236862/c7d08d52-5403-42e6-91a5-b04d13aa9f8b.png align="center")

Select the destination:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1687720263290/ccb86bdf-9a14-4e73-aadd-ab6e4313bf99.png align="center")

Configure the remaining settings:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1687720573331/151abf7d-05b0-4281-95a3-261b4427a116.png align="center")

## Starting the Task

Navigate to the task and start it. You can view the task's execution progress in the Task History option:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1687720810659/43bbfa17-959e-41df-b44e-07458df23e87.png align="center")

By following the detailed steps outlined in this guide, you can successfully set up and execute tasks to transfer data between these cloud storage services, simplifying the migration process and reducing the time and effort required. Thanks, and happy coding.