AzureFeeds

We are excited to announce the General Availability of client-side data integrity in Azure Blob Storage. Blob Storage has long supported integrity validation, starting with MD5 as part of the original HTTP standards and adding CRC64-NVME support in 2019. Today, CRC64-NVME is now integrated into the latest Azure Blob SDKs for .NET, C++, and JavaScript, extending Azure’s data integrity protections all the way into client applications.

Azure has had a longstanding commitment to provide the world’s most secure and reliable cloud storage platform. By extending data integrity protections into client applications, customers can end-to-end verify the data they read and write to Blob Storage is fully intact byte-for-byte from their application to Azure physical storage media using CRC64-NVME checksums – without the need to deploy and manage additional tools.

Azure’s Commitment to Data Integrity

Azure Blob Storage considers the durability and integrity of the data stored by customers as a fundamental promise of a cloud object storage provider. The Blob Storage platform was designed to prevent, detect, and repair any data integrity issues, along with making data durable and available at up to sixteen 9s with GRS. From the beginning, Blob Storage has stored checksums alongside every data fragment and continuously verifies them with background jobs, ensuring the bytes we durably store remain exactly the bytes we received —for every data replica (LRS/ZRS/GRS/GZRS).

Extending integrity to your client applications

Today, Azure Blob Storage is extending that protection into your client applications for end-to-end integrity. When enabled by customers, with CRC64-NVME transactional verification built into the latest Blob SDKs, every byte uploaded or downloaded has a CRC64-NVME checksum generated and that checksum is validated end-to-end. This protects against silent corruption that can occur within client-side code, on faulty client hardware, or in proxies and other services that sit between your application and Blob Storage. With CRC64-NVME validation enabled, the SDK computes and sends a checksum over the data and the service independently verifies it before acknowledging the write; the same happens in reverse on read. For multi-part reads/writes, individual checksums are reconciled against a whole-object checksum to catch any corruption within the client library itself. Every byte is now integrity-checked from the moment it enters the SDK, through the Blob Storage service and your storage account, and back again.

Benefits of CRC64-NVME

Microsoft has always considered 64 bit-checksums a requirement at cloud scale, with the CRC64-NVME polynomial protecting exabytes of customer data in Blob Storage from accidental data corruption. Microsoft has invested and shared high-performance implementations of our CRC64-NVME implementation that other cloud providers and industry participants have adopted as the default for data integrity use cases.

The Cyclic Redundancy Check (CRC) algorithm was originally designed for detecting bit corruption in data transit over network protocols like Ethernet, so it has relatively minimal CPU usage. The CRC64-NVME polynomial variant used for Blob Storage data integrity takes advantage of hardware acceleration present in modern x86 and ARM CPUs to further reduce CPU usage. Further performance gains are achieved by composing CRC checksums – meaning if you have the CRC of two chunks of data, you can combine them to get the CRC of the whole data, without needing to re-read the data. CRC64-NVME also provides strong reliability: a collision probability of a random bit error is 1 in 2^64, or 1 in 18 quintillion.

Performance

Performance is paramount in Azure Blob and great care was taken to implement the CRC64-NVME end to end integrity performantly. The latest version of the Blob SDK implements data integrity through the Structured Message Body REST API, which boosts our efficiency. Structured message bodies effectively allow the Blob client libraries to process the data, compute the CRC64-NVME checksum, and compose/unpack the HTTP request all in one flow, rather than processing the same data two or three times.

There are no restrictions on the size of your data for end-to-end data integrity to work. The latest version of the Blob SDKs using the structured message body implementation can handle any size of data to perform the end-to-end Data Integrity, unlike our previous MD5 version.

Throughput and CPU utilization can vary between using CRC64-NVME, MD5, and no validation. Customers can weigh the benefits of end-to-end integrity with the performance implications that might be right for their application.

Usage

Requires upgrading to the latest SDK versions.

Upload data to Blob with end-to-end data integrity (.NET):

BlobUploadOptions options = new BlobUploadOptions
{
TransferValidation = new UploadTransferValidationOptions
{
ChecksumAlgorithm = StorageChecksumAlgorithm.StorageCrc64
}
};
Response response = blobClient.Upload(upStream, options);

Download data to Blob with end-to-end data integrity (.NET):

BlobDownloadToOptions options = new BlobDownloadToOptions
{
TransferValidation = new DownloadTransferValidationOptions
{
ChecksumAlgorithm = StorageChecksumAlgorithm.StorageCrc64
}
};
Response response = blobClient.DownloadTo(downStream, options);

Upload and Download on all client operations to Blob with end-to-end data integrity (.NET):

BlobClientOptions options = new BlobClientOptions
{
TransferValidation =
{
Upload = { ChecksumAlgorithm = StorageChecksumAlgorithm.StorageCrc64 },
Download = { ChecksumAlgorithm = StorageChecksumAlgorithm.StorageCrc64 }
},
};
BlobServiceClient blobServiceClient = new BlobServiceClient(serviceUri, new DefaultAzureCredential(), options);

Availability

Comparison to MD5

MD5 is a cryptographic hashing algorithm that has historically been used for verifying integrity of data. Content-MD5 was one of the original HTTP headers for this purpose. While Blob Storage has supported HTTP headers for integrity including MD5, it’s less preferred than CRC64-NVME for data integrity due to its lower reliability in detection, high collision rate, inability for hardware acceleration, and higher computational cost. Recently, MD5 has even been disallowed by many security frameworks (e.g. FIPS) due to its lack of cryptographic security.

With MD5, Blob Storage has only been able to verify the data integrity of each partitioned HTTP request/response with MD5, not the entire blob. Additionally, for reading blobs with MD5 validation there is a 4MiB size limit on the data in HTTP responses, which can significantly increase the number of HTTP requests required to read data. In both reading and writing blobs, the SDKs require buffering or re-reading the data to compute MD5 in addition to assembling the HTTP request and unpacking the HTTP response.

With the latest implementation of CRC64-NVME, Azure Blob client libraries can verify data integrity on both the entire data and each HTTP request/response, can do so with any size of partitions on reading or writing data, and does so in only one pass over the data.

To migrate from MD5 to CRC64-NVME, simply change your ChecksumAlgorithm:

-ChecksumAlgorithm = StorageChecksumAlgorithm.MD5
+ChecksumAlgorithm = StorageChecksumAlgorithm.StorageCrc64

Next Steps

To begin using end-to-end Data Integrity with CRC64-NVME, upgrade to the latest SDK version of your preferred language and check out our developer guides and examples:

• .NET


• C++


• JavaScript


• Azure Storage Client Libraries