Wednesday, August 03, 2011

Announcing Windows Azure Storage Analytics

by Windows Azure Storage Team via Windows Azure on 8/3/2011 10:57:00 PM

We are excited to announce the release of Windows Azure Storage Analytics.  This feature offers developers and operations the ability to track, analyze, and debug usage of Windows Azure Storage (Blobs, Tables and Queues).  You can use this data to analyze storage usage to improve the design of your applications and their access patterns to Windows Azure Storage.  Analytics data consists of:

• Logs
  • Provide trace of executed requests for Blobs, Tables and Queues
• Metrics
  • Provide summary of key capacity and request statistics for Blobs, Tables and Queues
    

Logs

This feature provides a trace of all executed requests for your storage accounts as block blobs in a special container called $logs. Each log entry in the blob corresponds to a request made to the service and contains information like request id, request URL, http status of the request, requestor account name, owner account name, server side latency, E2E latency, source IP address for the request etc.

This data now empowers you to analyze your requests much more closely. It allows you to run the following types of analysis:

• How many anonymous requests is my application seeing from a given range of IP address?
• Which containers are being accessed the most?
• How many times is a particular SAS URL being accessed and how?
• Who issued the request to delete a container?
• For a slow request –where is the time being consumed?
• I got a network error, did the request reach the server?

Metrics

Provide summary of key statistics for Blobs, Tables and Queues for a storage account. The statistics can be categorized as:

• Request information: Provides hourly aggregates of number of requests, average server side latency, average E2E latency, average bandwidth, total successful requests and total number of failures and more. These request aggregates are provided at a service level and per API level for APIs requested in that hour.  This is available for Blob, Table and Queue service.
• Capacity information: Provides daily statistics for the space consumed by the service, number of containers and number of objects that are stored in the service.  Note, this is currently only provided for the Windows Azure Blob service.

All Analytics Logs and Metrics data are stored in your user account and is accessible via normal Blob and Table REST APIs. The logs and metrics can be accessed from a service running in Windows Azure or directly over the Internet from any application that can send and receive HTTP/HTTPS requests. You can opt in to store either the log data and/or metric data by invoking a REST API to turn on/off the feature at a per service level.  Once the feature is turned on, the Windows Azure Storage stores analytics data in the storage account. Log data is stored as Windows Azure Blobs in a special blob container and metrics data is stored in special tables in Windows Azure Tables. To ease the management of this data, we have provided the ability to set a retention policy that will automatically clean up your analytics blob and table data.

Please see the following links for more information:

• MSDN Documentation
• Logging: Additional information and Coding Examples
• Metrics: Additional information and Coding Examples

Windows Azure Storage Team