Thursday, May 28, 2009

New Windows Azure Storage Features – May 2009

by WindowsAzure via Windows Azure on 5/28/2009 8:58:20 PM

Today we released a new set of features for Windows Azure Storage. The main features in this release are:

• Entity Group Transactions for Windows Azure Table – It is now possible to perform a transaction across entities stored within the same table and same partition (i.e., entities having the same partition key value). An application can now atomically perform multiple Create/Update/Delete operations across multiple entities in a single batch request to the storage system, as long as the entities in the batch operation have the same partition key value and are in the same table.   This feature allows applications to now (a) perform batch inserts, updates and deletes (up to 100 entities per batch operation) in a single request to the system, and (b) perform atomic operations across multiple entities.
  
• Copy Blob for Windows Azure Blob – We now allow applications to copy a blob from a source blob to a destination blob within the same storage account.   This copies the whole blob, including the blob metadata, properties, and committed block list.   This enables applications to (a) create backup copies of blobs, (b) move blobs between containers and (c) rename blobs (copy to new name followed by delete of the old name).
  
• Get Block List for Windows Azure Blob – We have added the ability to GetBlockList to retrieve the uncommitted block list, as well as the committed block list. GetBlockList provides the ability to retrieve the list of blocks that have been uploaded as part of a blob.  There are two block lists maintained for a blob, and this function allows retrieval of either of the two or both:
  • Committed Block List - This is the list of blocks that have been successfully committed as part of a PutBlockList for a given blob. In doing a GetBlob, this is the list of blocks that constitute the readable version of the blob.
  • Uncommitted Block List - This is the list of blocks that have been uploaded using PutBlock for a blob since the last successful PutBlob or PutBlockList for the blob.  These blocks represent the temporary/uncommitted blocks that have not yet been committed.   One benefit of allowing applications to retrieve this list is that it helps the client more easily deal with continuation of upload if the client crashes. For instance, if a client that is uploading blocks crashes, when it restarts it can retrieve the list of uncommitted blocks that have already been uploaded, and continue uploading where it left off.
    
• API Versioning - For all of the Windows Azure Storage solutions we have introduced a new HTTP header called “x-ms-version”.     All changes to the storage APIs will be versioned by this header.  This allows prior versions of commands executed against the storage system to continue to work, as we extend the capabilities of the existing commands and introduce new commands.  The new version introduced with this release is “x-ms-version: 2009-04-14”.   We still support all of the PDC 2008 CTP commands without the version header, so everything will continue to work for existing storage customers.   In order to use the new functionality described above (Entity Group Transactions, Copy Blob, and Get Block List), the version header must be used as part of the REST request.
  
  Although it’s not currently required for older operations, we recommend that you begin including the x-ms-version header on all calls against non-public resources. Once Windows Azure is released commercially, versioning will be required on all operations, with the exception of anonymous operations against the Blob service.

In addition to the above, there are two additional changes to the semantics of Windows Azure Tables:

• Property Names – We now support C# Unicode characters for the property names. With this change we no longer support “-“, since it is not supported by ADO.NET data services. This is a versioned change, so the application must specify “x-ms-version: 2009-04-14” header to start using the new characters in the property name, and with this version “-“ is no longer supported as part of the property name.
  
• PartitionKey and RowKey sizes – The PartitionKey and RowKey have a limit of 1K of characters each.   Originally, we had documented that these could be up to 64KB, but there was a bug in the CTP release that actually limited these to less than 256 characters. With this release the limit is now 1K characters each.

Finally, we want to give a heads up to a future change for the max timeout allowed for some of the Windows Azure Storage operations. If the request specifies a timeout greater than this, we ignore the request timeout and instead use the max timeout specified below. If the request specifies a shorter timeout, we use the request’s timeout. Listed below are the current max timeouts that we support, along with the future change for List and Query operations.

• All Storage Timeouts – The following are the maximum timeouts allowed for Windows Azure Storage operations:
  • GetBlob – A get bob request is allowed to take up to 2 minutes per MByte transferred, before the server side times out.
  • PutBlob – A put bob request is allowed to take up to 10 minutes per MByte transferred, before the server side times out.
  • PutBlockList and GetBlockList – These have a max timeout of 60 seconds.
  • Table Query and All List Operations – Current max timeout is 60 seconds. In the future this will be changed to 30 seconds.
  • All other commands in Blobs, Tables and Queues have a max timeout of 30 seconds.

Note that the above features for this new release are currently only available via the Windows Azure Storage REST interface and ADO.NET client library. They are not yet part of the Windows Azure SDK or the sample storage client library that ships with the SDK. The reason for this is that our goal is to have an agile release process, and release the features as soon as the REST interfaces are ready, and then a future SDK release will include support for these changes.

For more information, details about these new features can be found in the MSDN documentation here:
http://msdn.microsoft.com/en-us/library/dd179355.aspx

In addition, May 2009 versions of the Windows Azure Storage Table and Blob White papers, with information about these features, will be pushed here soon:
http://msdn.microsoft.com/en-us/azure/cc994380.aspx

Steve Marx will have some examples on using the above new functionality on his blog soon:
http://blog.smarx.com/

As always, we appreciate any feedback you might have.

Brad Calder
Windows Azure Storage