AzureFeeds

In certain scenarios, a storage account may contain a significant number of page blobs classified under the hot access tier that are infrequently accessed or retained solely for backup purposes. To optimise costs, it is desirable to transition these page blobs to the archive tier. However, as indicated in the following documentation – https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview the ability to set the access tier is only available for block blobs; this functionality is not supported for append or page blobs.

The Azure blob storage connector in Azure data factory is capable of copying blobs from block, append, or page blobs and copying data to only block blobs. https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-blob-storage?tabs=data-factory#supported-capabilities

Note: No extra configuration is required to set the blob type on the destination. By default, the ADF copy activity creates blobs as Block Blobs.

In this blog, we will understand how to make use of Azure Data Factory to copy the page blobs to block blobs. Please note that this is applicable to append blobs as well.

Let’s take a look at the steps ahead

Step 1: Creating ADF instance

• Create an Azure data factory resource in the Azure portal referring to the following document – https://learn.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory


• After creation, click on “Launch Studio” as shown below

Step 2: Creating datasets

• Create two datasets by navigating to Author -> Datasets -> New dataset. These datasets are used in source and sink for the ADF copy activity

• Select “Azure blob storage” -> click on continue -> select “binary” and continue

Step 3: Creating Linked service

• Create a new linked service and provide the storage account name which contains page blobs

• Provide the file path where the page blobs are located.


• You would also need to create another dataset for destination. Repeat the steps from 3 to 6 to create another destination dataset to copy the blobs to the storage account as block blobs.

Note: You can use same or different storage account for the destination dataset. Set it as per your requirements.

Step 4: Configuring a Copy data pipeline

• Once the two datasets are created, now create a new pipeline and under “Move and Transform” section, drag and drop the “Copy data” activity as shown below.

• Under the Source and Sink sections from the drop down, select the source and destination datasets respectively which were created in the previous steps. Select the “Recursively” option and publish the changes.
  
  
  
  • Source:
  
  
  
  

• 
  
  
  
  • Sink:
  
  
  
  

Note: You can configure the filters and copy behaviour as per your requirements.

Step 5: Debugging and validating

• Now as the configuration is completed, click on “Debug”.


• If the pipeline activity ran successfully, you should be able to see “succeeded” status in the output section as below.

• Verify the blob type of the blobs in the destination storage account and it should show as block blob and access tier as Hot.

After converting the blobs to block blobs, several methods are available to change their access tier to archive. These include implementing a blob lifecycle management policy, utilizing storage actions, or by using Az CLI or PowerShell scripts.

Conclusion

Utilising ADF enables the conversion of page or append blobs to block blobs, after which any standard method such as LCM policy or storage actions may be used to change the access tier to archive. This strategy offers a more streamlined and efficient solution compared to developing custom code or scripts.

Reference links:

• https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview


• https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-blob-storage?tabs=data-factory#supported-capabilities


• https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview


• https://learn.microsoft.com/en-us/azure/storage-actions/storage-tasks/storage-task-quickstart-portal


• https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview


• https://learn.microsoft.com/en-us/azure/storage/blobs/archive-blob?tabs=azure-powershell#bulk-archive