Tuesday, April 13, 2010

Uploading Windows Azure Blobs from Silverlight – Part 3: Handling Big Files by Using the Block APIs

by Steve Marx via blog.smarx.com - cloud development blog on 4/13/2010 5:17:41 PM

In this series of blog posts, I’ll show you how to use Silverlight to upload blobs directly to Windows Azure storage. At the end of the series, we’ll have a complete solution that supports uploading multiple files of arbitrary size. You can try out the finished sample at http://slupload.cloudapp.net.

Part 3: Handling Big Files by Using the Block APIs

In Part 1, we saw how to handle authentication using Shared Access Signatures. In Part 2, we saw how to enable cross-domain access to blobs so we could upload data using Silverlight. In this post, we’ll see how to handle large blobs by breaking them into smaller blocks.

Using blocks

In Part 2, we used the Put Blob method to upload blobs using Silverlight. This operation uploads an entire blob in a single web request. For larger blobs, though, we may want to upload the blob in smaller chunks, called blocks. In fact, for blobs larger than 64MB, it’s mandatory that we use the block API (Put Block and Put Block List) to upload the blob in chunks.

Uploading a blob as a series of blocks generally gives us a number of advantages:

1. It allows us to upload blobs larger than 64MB.
2. It allows us to upload blocks in parallel.
3. It allows us to resume failed uploads by retrying only the blocks that weren’t already uploaded.

For our example, we’ll only be taking advantage of advantage #1.

The block API

Below is the relevant code from http://slupload.cloudapp.net.

This first snippet constructs the web request whether or not we’re using blocks. (We use blocks when the blob size exceeds 4MB.)

if (UseBlocks)
{
    // encode the block name and add it to the query string
    currentBlockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(Guid.NewGuid().ToString()));
    uriBuilder.Query = uriBuilder.Query.TrimStart('?') + 
        string.Format("&comp=block&blockid={0}", currentBlockId);
}

// with or without using blocks, we'll make a PUT request with the data
HttpWebRequest webRequest = (HttpWebRequest)WebRequestCreator.ClientHttp.Create(uriBuilder.Uri);
webRequest.Method = "PUT";
webRequest.BeginGetRequestStream(new AsyncCallback(WriteToStreamCallback), webRequest);

If we’ve used blocks, we need to commit the blob by calling Put Block List after we’ve uploaded all the blocks. This code constructs the web request:

HttpWebRequest webRequest = (HttpWebRequest)WebRequestCreator.ClientHttp.Create(
    new Uri(string.Format("{0}&comp=blocklist", UploadUrl)));
webRequest.Method = "PUT";
webRequest.Headers["x-ms-version"] = "2009-09-19"; // x-ms-version is required for put block list!
webRequest.BeginGetRequestStream(new AsyncCallback(BlockListWriteToStreamCallback), webRequest);

This code creates an XML document that lists the blocks (which we kept track of as we uploaded them) and sends this as the body of our Put Block List call:

HttpWebRequest webRequest = (HttpWebRequest)asynchronousResult.AsyncState;
Stream requestStream = webRequest.EndGetRequestStream(asynchronousResult);

var document = new XDocument(
    new XElement("BlockList",
        from blockId in blockIds
        select new XElement("Uncommitted", blockId)));
var writer = XmlWriter.Create(requestStream, new XmlWriterSettings() { Encoding = Encoding.UTF8 });
document.Save(writer);
writer.Flush();

requestStream.Close();

webRequest.BeginGetResponse(new AsyncCallback(BlockListReadHttpResponseCallback), webRequest);

Putting It All Together

Now that we’ve generated URLs using Shared Access Signatures, we’ve enabled cross-domain access to blob storage, and we’ve used the block API to handle large file uploads, we’re ready to put it all together with some nice UI. For the nice UI, I turned to the Silverlight Multi File Uploader project on Codeplex. I picked it because it looked nice and the source uses multiple upload methods already, so it was clear how to add another one.

You can download the full source (as a Visual Studio 2010 RC solution). I made the following modifications to the original Silverlight Multi File Uploader code:

• In Page.xaml.cs, I added the fields _WindowsAzureBlobUploader and _uploadContainerUrl, which are new parameters on the Silverlight control.
• In UserFile.cs, I added the corresponding fields and added a clause in Upload() to instantiate my new WindowsAzureBlobUploader when specified by the parameters.
• WindowsAzureBlobUploader.cs is a new file that started as a copy of HttpFileUploader.cs. This contains all the code from this post, and I tried to add comments in enough places that the code should be easily readable. Let me know if you take a look and don’t understand something.

Finally I added a web role with Default.aspx and View.aspx and their respective codebehinds. Those pages handle creating the Shared Access Signatures, instantiating the Silverlight control, and providing a viewer for the uploaded files.

Read the Whole Series

Read the entire three-part series to see how http://slupload.cloudapp.net was developed:

• Part 1 – Shared Access Signatures
• Part 2 – Enabling Cross-Domain Access to Blobs
• Part 3 – Handling Big Files by Using the Block APIs

You can download the full source code for the series as a Visual Studio 2010 RC solution here: http://cdn.blog.smarx.com/files/SilverlightUpload_source.zip.