Tyson Garrett
Tyson Garrett

CTO, TrustOnCloud

How I bypassed the control plane in Azure OpenAI

While completing our ThreatModel for Azure OpenAI, I discovered a way to allow the management of Azure OpenAI deployments via the Data Plane, resulting in the loss of major security controls.

Prerequisite Knowledge

Before I outline this in more detail and what this means, let’s first cover some fundamentals around Azure security. 

Azure Control Plane and Data Plane Separation 

Unlike other cloud providers, Azure permissions are separated into Actions (Control Plane) and Data Actions (Data Plane). To quote Microsoft, “By adding these data properties, the separation between control plane and data plane is maintained.Control Plane actions are against the resource configuration whereas the Data Plane actions are against the data hosted by the resource. This is an awesome feature as, when correctly designed, it helps with segregation of duties and ensures that access to read/modify data is intentionally granted.

Control Plane APIs are associated with the management.azure.com endpoint. Actions against the Data Plane are associated with a ‘service instance endpoint’ – formatted as ‘RESOURCE_NAME.SERVICE.azure.com‘. In the case of Azure OpenAI, service instance endpoints are constructed as ‘RESOURCE_NAME.openai.azure.com‘.

APIs associated with management.azure.com come with built-in security benefits:

  1. Resource Manager Locks (aka management locks): Prevents the unintended deletion or modification of a resource. This is applied across all users and roles, regardless of your permission. Locks don’t restrict how a resource performs its functions. Actions taken against the data-plane are unaffected by resource locks assuming the caller has the appropriate permission.
  2. Azure Policy: Azure Policy can sit inline with APIs associated with management.azure.com. Azure Policy determines whether a resource is compliant or not by comparing the properties of a resource to the allowed values in the Azure Policy definition. Azure Policy will typically work in a Deny, Audit or Modify capacity. An example of an Azure Policy is “Permit only approved OpenAI models” which permits only certain types of OpenAI models to be deployed. Note that Azure Policy can work at more of a data-plane level for some services, but this is not the case with Azure OpenAI.

API’s associated with service endpoints still support Entra ID based authentication, but they also support a concept known as service keys (some services call these access keys). Service keys allow FULL access to all data plane operations for a given service. These keys are created when you create an Azure OpenAI Account. You can rotate them and you can apply IP restrictions on the Public Endpoint. You can also disable the public endpoint and require access to be via a private endpoint located in your virtual network.

image 1

Azure Role Definitions

In Azure Role Definitions, this Control Plane / Data Plane dichotomy is expressed as “Actions” for the Control Plane and “DataActions” for the Data Plane. An example policy illustrating this is below where you can see two control-plane APIs listed as “Actions”, the ability to read a Storage container and generate a user key delegation, but the ability to read the data is listed as a “DataAction”with “/containers/blob/read” as it is a data-plane API:

image 0

Azure OpenAI Control Plane Bypass

Azure OpenAI deployments specify the model type and version to be used in addition to content filters and rate limits to be used. Multiple deployments can reside within a single Azure OpenAI Account. An Azure OpenAI Account is represented as the service endpoint I referred to earlier.

image 3

Microsoft’s intention is for the management of Azure OpenAI Deployments to be performed under the Microsoft.CognitiveServices resource-manager (think management.azure.com) API’s using the path of https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.CognitiveServices/accounts/{accountName}/deployments

  • Deployments_List
  • Deployments_Create
  • Deployments_Get
  • Deployments_CreateUpdate
  • Deployments_Delete

The Action permissions associated with these API’s are:

  • Microsoft.CognitiveServices/accounts/deployments/read
  • Microsoft.CognitiveServices/accounts/deployments/write
  • Microsoft.CognitiveServices/accounts/deployments/delete

Management of the Azure OpenAI Deployment via this method provides the additional security benefits Resource Manager Locks, Azure Policy and Entra ID as previously described.

The 2023-03-15-preview, 2022-06-01-preview, 2022-03-01-preview, and 2022-12-01-stable OpenAI authoring APIs introduced the following operations via the service endpoint under the path of https://RESOURCE_NAME.openai.azure.com/deployments:

  • Deployments_List
  • Deployments_Create
  • Deployments_Get
  • Deployments_CreateUpdate
  • Deployments_Delete

These introduced APIs are associated with the following DataAction permissions:

  • Microsoft.CognitiveServices/accounts/OpenAI/deployments/read
  • Microsoft.CognitiveServices/accounts/OpenAI/deployments/write
  • Microsoft.CognitiveServices/accounts/OpenAI/deployments/delete

Which means we can now create/modify/delete the Azure OpenAI Deployment resource by the Data Plane bypassing the built in controls offered by the management.azure.com endpoint!

To make matters worse the Azure AI Developer built-in role has the following contents:

image 2

As these DataActions match the Microsoft.CognitiveServices/accounts/OpenAI/* wildcard permission the Azure AI Developer built-in role allows this bypass. To make it even worse, the description of this built-in role states, Can perform all actions within an Azure AI resource besides managing the resource itself” which I’d argue is not the case. This is another example of how wildcards in permissions can cause unintended consequences.

TLDR: I’m an Azure expert; tell me the issue

A set of Azure OpenAI authoring API’s enables the use of the service instance endpoint as opposed to management.azure.com to create, update, delete, and list/read the Azure OpenAI Deployment resource. This allows for bypass of Azure Policy for Deny/Modify effects, Resource Locks and allows the “Azure AI Developer” built-in role, which has a description of “Can perform all actions within an Azure AI resource besides managing the resource itself” the ability to manage the Deployment resource and finally provides you the option to use access keys instead of your Entra ID identity to do so.

We did notify Microsoft of this issue; however, it didn’t meet the MSRC bar; it was passed onto the Azure OpenAI team, and we have noted that Microsoft has announced that the 2023-03-15-preview and 2022-06-01-preview APIs will be retired on 2 Apr 2024; however, the 2022-12-01-stable API has not had any retirement notices posted so will presumably continue to allow this behavior. It appears that Microsoft did realize they doubled up here at some point as the 2023-07-01-preview and later API’s and the 2023-05-15 stable API removed these API operations. However, as you can specify the API version you wish to use when using Azure, you can still use the older ones until Microsoft either retires the API version or restricts this functionality.

Impacted API Versions

The following API versions are affected by this issue:

  • 2022-03-01-preview (retired)
  • 2023-03-15-preview (to be retired April 2, 2024)
  • 2022-06-01-preview (to be retired April 2, 2024)
  • 2022-12-01-stable

Recommended Controls

TrustOnCloud recommends customers not to use the Azure AI Developer built-in role and ensure any roles used for Microsoft.CognitiveServices namespace add the below operations to the NotDataActions section of any applicable Role Definitions.

  • Microsoft.CognitiveServices/accounts/OpenAI/deployments/read
  • Microsoft.CognitiveServices/accounts/OpenAI/deployments/write
  • Microsoft.CognitiveServices/accounts/OpenAI/deployments/delete

Additional controls are available in the TrustOnCloud ThreatModel for Azure OpenAI.

Vendor Notification Timeline

24 Oct 2023 10:04 – TrustOnCloud raised against MSRC 

24 Oct 2023 10:04 – Status changed to Review / Repo by MSRC

24 Oct 2023 10:09 – Status changed to New by MSRC

25 Oct 2023 05:01 – Status changed to Review / Repo by MSRC

31 Oct 2023 01:48 – Status changed from Review / Repo to New by MSRC

14 Nov 2023 06:12 – TrustOnCloud requests update

29 Nov 2023 17:59 – TrustOnCloud requests update due to no response

08 Dec 2023 09:06 – TrustOnCloud requests update due to no response

12 Dec 2023 17:41 – MSRC sets case with Complete responding with

“Thank you again for your submission to MSRC. Our engineers have investigated the report, and we have informed the appropriate team about the issues you reported. However, this case does not meet the bar for servicing by MSRC and we will be closing this case.”

Subscribe for Email Updates
Subscribe to TrustOnCloud's blog and newsletter for the latest insights on cloud security and industry best practices.

Want to see our Azure OpenAI ThreatModel in action?