AzureFeeds

Co-authors – Christoph Dreymann  –  Abul Azed  –  Shiva P.

Introduction

As organizations increase their cloud adoption to accelerate AI readiness, Microsoft Incident Response has observed the rise of cloud-based threats as attackers seek to access sensitive data and exploit vulnerabilities stemming from misconfigurations often caused by rapid deployments. In this blog series, Cloud Forensics, we share insights from our frontline investigations to help organizations better understand the evolving threat landscape and implement effective strategies to protect their cloud environments.

This blog post explores the importance of enabling and analyzing Microsoft Azure Key Vault logs in the context of security investigations. Microsoft Incident Response has observed cases where threat actors specifically targeted Key Vault instances. In the absence of proper logging, conducting thorough investigations becomes significantly more difficult.

Given the highly sensitive nature of the data stored in Key Vault, it is a common target for malicious activity. Moreover, attacks against this service often leave minimal forensic evidence when verbose logging is not properly configured during deployment. We will walk through realistic attack scenarios, illustrating how these threats manifest in log data and highlighting the value of enabling comprehensive logging for detection.

Key Vault

Key Vault is a cloud service designed for secure storage and retrieval of critical secrets such as passwords or database connection strings. In addition to secrets, it can store other information such as certificates and cryptographic keys.

To ensure effective monitoring of activities performed on a specific instance of Key Vault, it is essential to enable logging. When audit logging is not enabled, and there is a security breach, it is often difficult to ascertain which secrets were accessed without comprehensive logs. Given the importance of the assets protected by Key Vault, it is imperative to enable logging during the deployment phase.

How to enable logging

Logging must be enabled separately for each Key Vault instance either in the Microsoft Azure portal, Azure command-line interface (CLI) or Azure PowerShell. How to enable logging can be found here. Alternatively, it can be configured within the default log analytics workspace as an Azure Policy. How to use this method can be found here.

By directing these logs to a Log Analytics workspace, storage account, or event hub for security information and event management (SIEM) ingestion, they can be utilized for threat detection and, more importantly, to ascertain when an identity was compromised and which type of sensitive information was accessed through that compromised identity. Without this logging, it is difficult to confirm whether any material has been accessed and therefore may need to be treated as compromised.

NOTE: There are no license requirements to enable logging within Key Vault, but Log Analytics charges based on ingestion and retention for usage of that service (Pricing – Azure Monitor | Microsoft Azure)

Next, we will review the structure of the Audit Logs originating from the Key Vault instance. These logs are located in the AzureDiagnostics table.

Interesting fields

Below is a good starting query to begin investigating activity performed against a Key Vault instance:

 

AzureDiagnostics
| where ResourceType == ‘VAULTS’

 

The “operationName” field is of particular significance as it indicates the type of operation that took place. An overview of Key Vault operations can be found here.

The “Identity” field includes details about the identity responsible for an activity, such as the object identifier and UPN.

Lastly, the “callerIpAddress” shows which IP address the requests originate from.

The table below displays the fields highlighted and used in this article.

 

 

A complete list of fields can be found here.

To analyze further, we need to understand how a threat actor can access a Key Vault by examining the Access Policy and Azure role-based access control (RBAC) permission model used within it.

Access Policy permission model vs Azure RBAC

The Access Policy Permission Model operates solely on the data plane, specifically for Azure Key Vault. The data plane is the access pathway for creating, reading, updating, and deleting assets stored within the Key Vault instance. Via a Key Vault Access Policy, you can assign individual permissions and grant access to security principals such as users, groups, service principals, and managed identities, at the Key Vault scope with appropriate Control Plane privileges.

This model provides flexibility by granting access to keys, secrets, and certificates through specific permissions. However, it is considered a legacy authorization system native to Key Vault.

Note: The Access Policies permission model has privilege escalation risks and lacks Privileged Identity Management support. It is not recommended for critical data and workloads.

On the other hand, Azure RBAC operates on both Azure’s control and data planes. It is built on Azure Resource Manager, allowing for centralized access management of Azure resources. Azure RBAC controls access by creating role assignments, which consist of a security principal, a role definition (predefined set of permissions), and a scope (a group of resources or an individual resource). RBAC offers several advantages, including a unified access control model for Azure resources and integration with Privileged Identity Management. More information regarding Azure RBAC can be found here.

Now, let’s dive into how threat actors can gain access to a Key Vault.

How a threat actor can access a Key Vault

When a Key Vault is configured with Access Policy permission, privileges can be escalated under certain circumstances. If a threat actor gains access to an identity that has been assigned the Key Vault Contributor Azure RBAC role, Contributor role or any role that includes ‘Microsoft.KeyVault/vaults/write‘ permissions, they can escalate their privileges by setting a Key Vault access policy to grant themselves data plane access, which in turn allows them to read and modify the contents of the Key Vault.

Modifying the permission model requires ‘Microsoft.Authorization/roleAssignments/write‘ permission, which is included in the Owner and User Access Administrator roles. Therefore, a threat actor cannot change the permission model without one of these roles.

Any change to the authorization mode will be logged in the Activity Logs of the subscription, as shown in the figure below:

 

Figure 1: Activity Logs shows change of the authorization mode

If a new Access Policy is added, it will generate the following entry within the Azure Activity Log:

 

Figure 2: Activity Logs shows when a new Access Policy was added

When Azure RBAC is the permissions model for a Key Vault, a threat actor must identify an identity within the Entra ID tenant that has access to sensitive information or one capable of assigning such permissions. Information about Azure RBAC roles for Key Vault access, specifically those who can access Secrets, can be found here.

A threat actor that has compromised an identity with an Owner role is authorized to manage all operations, including resources, access policies, and roles within the Key Vault. In contrast, a threat actor with a Contributor role can handle management operations but does not have access to keys, secrets, or certificates. This restriction applies when the RBAC model is used within a Key Vault.

The following section will examine the typical actions performed by a threat actor after gathering permissions.

Attack scenario

Let’s review the common steps threat actors take after gaining initial access to Microsoft Azure. We will focus on the Azure Resource Manager layer (responsible for deploying and managing resources), as its Azure RBAC or Access Policy permissions determine what a threat actor can view or access within Key Vault(s).

Enumeration

Initially, threat actors aim to understand the existing organizations’ attack surface. As such, all Azure resources will be enumerated. The scope of this enumeration is determined by the access rights held by the compromised identity. If the compromised identity possesses access comparable to that of a reader or a Key Vault reader at the subscription level (reader permission is included in a variety of Azure RBAC roles), it can read numerous resource groups. Conversely, if the identity’s access is restricted, it may only view a specific subset of resources, such as Key Vaults. Consequently, a threat actor can only interact with those Key Vaults that are visible to them. Once the Key Vault name is identified, a threat actor can interact with the Key Vault, and these interactions will be logged within the AzureDiagnostics table.

List secrets / List certificates Operation

With the Key Vault Name, a threat actor could list secrets or certificates (Operation: SecretList and CertificateList) if they have the appropriate rights (while this is not the final secret, it indicates under which name the secret or certificate can be retrieved). If not, access attempts would appear as unsuccessful operations within the httpStatusCode_d field, aiding in detecting such activities. Therefore, a high number of unauthorized operations on different Key Vaults could be an indicator of suspicious activity as shown in the figure below:

 

Figure 3: Unauthorized operation on different Key Vaults

The following query assists in detecting potential unauthorized access patterns.

Query:

AzureDiagnostics

| where ResourceType == ‘VAULTS’ and OperationName != “Authentication”

| summarize MinTime = min(TimeGenerated), MaxTime = max(TimeGenerated), OperationCount=count(), UnauthorizedAccess=countif(httpStatusCode_d >= 400), OperationNames = make_set(OperationName), make_set_if(httpStatusCode_d, httpStatusCode_d >= 400), VaultName=make_set(Resource) by CallerIPAddress

| where OperationNames has_any (“SecretList”, “CertificateList”) and UnauthorizedAccess > 0

                   

When a threat actor uses a browser for interaction, the VaultGet operation is usually the first action when accessing a Key Vault. This operation can also be performed via direct API calls and is not limited to browser use.

High-Privileged account store 

Next, we assume a successful attempt to access a global admin password for Entra ID.

Analyzing Secret retrieval

When an individual has the identifier of a Key Vault and has SecretList and SecretGet access rights, they can list all the secrets stored within the Key Vault (OperationName SecretList). In this instance, this secret includes a password. Upon identifying the secret name, the secret value can be retrieved (OperationName SecretGet).

The image below illustrates what appears in the AzureDiagnostics table. The HTTP status code indicates that these actions were successful. The requestUri contains the name of the secret, such as “BreakGlassAccountTenant” for the SecretGet operation. With this information, one can ascertain what secret has been accessed.

 

Figure 4: Typical flow when a secret is retrieved

The requestUri_s format for the SecretGet operation is as follows:

{vaultBaseUrl}/secrets/{secret-name}/{secret-version}?api-version=7.4

 

When the browser accesses the Key Vault service through the Azure portal, additional API calls are often involved due to the various views within the Key Vault services in Azure. The figure below illustrates this process. 

 

Figure 5: Pattern of a secret retrieval via the Azure portal

When someone accesses a specific Key Vault via a browser, the VaultGet operation is followed by SecretList. To further distinguish actions, SecretListVersion will also be used, as the Key Vault service shows different versions of a Secret, which may indicate direct browser usage. The final SecretGet Operation retrieves the actual secret.

When using the Key Vault, SecretList operations can be accompanied by SecretGet operations. This is less common for emergency accounts since these accounts are infrequently used. Setting up alerts when certain secrets are retrieved can assist in identifying unusual activity.

Entra ID Application certificate store

In addition to storing secrets, certificates that provide access to Entra ID applications can also be managed within a Key Vault. When creating an Entra ID application with a certificate for authentication, you can automatically store that certificate within a Key Vault of your choice.  Access to such certificates could allow a threat actor to leverage the access rights of the associated Entra ID application and gain access to Entra ID. For instance, if the Entra ID application possesses significant permissions, the extracted certificate could be utilized to exercise those permissions. Various Entra ID roles can be leveraged to elevate privileges; however, for this scenario, we assume the targeted application holds the “RoleManagement.ReadWrite.Directory” permission. Consequently, the Entra ID application would have the capability to assign the Global Admin role to a user account controlled by the threat actor. We have also described this scenario here.

Analyzing Certificate retrieval 

The figure below outlines the procedure for a threat actor to download a certificate and its private key using the Key Vault API. First, the CertificateList operation displays all certificates within a Key Vault. Next, the SecretGet operation retrieves a specific certificate along with its private key (the SecretGet operation is required to obtain both the certificate and its private key).

 

Figure 6: Pattern of a certificate + private key retrieval

When a threat actor uses the browser through the Azure portal, the sequence of actions should resemble those in the figure below:

 

Figure 7: Pattern of a certificate + private key retrieval via the Azure portal

When a Certificate object is selected within a specific Key Vault view, all certificates are displayed (Operation: CertificateList). Upon selecting a particular certificate in this view, the operations CertificateGet and CertificateListVersions are executed. Subsequently, when a specific version is selected, the CertificateGet operation will be invoked again. 

 

Figure 8: Retrieval of certificate and private key via Azure portal

When “Download in PFX/PEM format” is selected, the SecretGet Operation downloads the Certificate and private key within the Browser.

With the downloaded certificate, the threat actor can sign in as the Entra application and utilize the assigned permissions.

Key Vault summary

Detecting misuse of a Key Vault instance can be challenging, as operations like SecretGet can be legitimate. A threat actor might easily masquerade their activities among legitimate users. Nevertheless, unusual attributes, such as IP addresses or peculiar access patterns, could serve as indicators. If an identity is known to be compromised and has utilized Key Vaults, the Key Vault logs must be checked to determine what has been accessed to respond appropriately.

Coming up next

Stay tuned for the next blog in the Cloud Forensics series. If you haven’t already, please read our previous blog about hunting with Microsoft Graph activity logs.