AzureFeeds

This is part of my series on DNS in Microsoft Azure.

• DNS in Microsoft Azure – Azure-provided DNS
• DNS in Microsoft Azure – Azure Private DNS
• DNS in Microsoft Azure – Azure Private DNS Resolver
• DNS in Microsoft Azure – PrivateLink Private Endpoints
• DNS in Microsoft Azure – PrivateLink Private Endpoints and Private DNS
• DNS in Microsoft Azure – Private DNS in Business-To-Business Scenarios

Hello folks! I wanted to get at least one blog post in before 2025 so today I’m going to bring the conversation back to DNS once again. I’m going to be hitting on an advanced topic today, so if you’re unfamiliar with DNS in Azure, read up on my prior posts. I’m going to be skipping through much of the basics.

Today we’re going to talk about one of the challenges that tends to pop up when customers begin to heavily use PrivateLink Private Endpoints and Azure Private DNS. You will likely run into this challenge at some point (if you haven’t already) when you attempt to collaborate with another organization using Azure or even when using services like Azure Fabric where one BU (business unit) manages Azure and another manages Azure Fabric.

Brew your coffee, we’re about to dive into the weeds!

As I’ve covered in past posts, Microsoft provides you out-of-the-box DNS resolution for each VNet (virtual network) via the Azure-provided DNS service (I’m going to refer to it as the WireServer for the rest of this post). The WireServer can be reached at 168.63.129.16 from endpoints deployed to the virtual network and will route DNS queries to either Microsoft public DNS resolvers or to Private DNS zones. Private DNS Zones allow customers to host internally-facing DNS namespaces and are very commonly used with PrivateLink Private Endpoints for Microsoft PaaS (platform-as-a-service) services due automatic lifecycle management of the A records for the Private Endpoints. Thus our challenge begins to peek its ugly head.

Alrighty, I get it. You know all this and it’s boring you. Let’s get to the good stuff.

What if you need to collaborate with another organization and they also use Private Endpoints? How might this cause some issues?

Let’s take a scenario where Bob works for Contoso and Mary works for Fabrikam. Alice over at Fabrikam produces a daily dump of data from a financial system to an Azure Storage Account as a blob. Bob over at Contoso pulls that data down into his environment for analysis by employees of Contoso. Alice provides this dump to over a hundred customers. Due to this large volume of customers, she has opted to provide it over a public endpoint only.

This process has been working flawlessly for years and Bob’s life has been good. One day, Bob’s life isn’t good and his automation fails. After lots of troubleshooting involving both Contoso and Fabrikam, it’s determined that DNS resolution is failing when trying to resolve the name of the storage account.

As it turns out, Alice’s Information Security team made it a standard to use Private Endpoints and she turned on a Private Endpoint for the storage account. The creation of the Private Endpoint creates a CNAME for the storage account in public DNS for fabrikam.privatelink.blob.core.windows.net. Since Contoso has this Private DNS Zone configured in its environment, Bob’s query gets redirect to Contoso’s Private DNS Zone which doesn’t have the record and instead returns an NXDOMAIN.

Historically, this has been a pain to deal with. Customers have had to work around it by creating local host records (yuck), defining the FQDN (fully-qualified domain name) for the storage account as a zone, or creating conditional forwarders for specific FQDNs in their on-premises DNS service. While both will work, it can become a real headache at scale and can make troubleshooting resolution a complete nightmare. Yes, there is always the option of the 3rd party injecting a Private Endpoint into your virtual network, but I rarely see this occur across my customer base in situations where 3rd parties are servicing a large number of customers. Likely due to complexity and cost (yes Private Endpoints and the data transferring through them do have costs and can add up with large amounts of data).

At Ignite this year Microsoft announced the public preview of “Fallback to Internet for Private DNS” which seeks to address this problem once and for all. With this new feature customers can configure whether resolution should fallback to public DNS on a per virtual network link basis for each Private DNS Zone. This means you can pick which Private DNS Zones fallback to public DNS. Maybe you want to do it for privatelink.blob.core.windows not, but privatelink.database.windows.net. If you use different resolution paths (meaning separate virtual network links) for production and non-production, you can choose to fallback only for non-production while keeping today’s behavior for production. This gives you a ton of flexibility in how you handle resolution.

This feature is available for you to muck with today. In the Azure Portal you will see a new option in a virtual network link called Enable fallback to Internet. When you enable this option Azure DNS will fallback to public DNS resolution if it can’t find a record in a Private DNS Zone. Take note that if you’re doing this via command line you’ll need to update your az cli to the newest version. If using Terraform, plan to use azapi right now (not available in azurerm that I see) with an API version of 2024-06-01. The property in the API is called resolutionPolicy. With fallback off it’s set to the value of Default and if fallback is on it’s set to the value of NxDomainRedirect.

If we revisit Bob’s challenge. He can now resolve this by enabling fallback on the virtual network link used by his endpoint’s resolution path for the privatelink.blob.core.windows.net. When the WireServer receives back an NXDOMAIN, it will then try to resolve it via public DNS yielding the public endpoint IP Bob needs for Fabrikam’s storage account.

Yes folks, your life will become far less painful in the near future. Microsoft has yet to publish a public target GA (generally available) date, but I expect it will be live in the near future. Now is a great time to plan on how you’ll roll this out, especially which zones you want to fallback to public DNS and which zones you’d prefer to fail to resolve. There may be some zones you’d prefer return NXDOMAIN because resources using those zones will never have this type of use case. Personally, I would do this where it makes sense and leave it off for all other zones/links because it does add complexity to DNS resolution.

So your key takeaways:

1. The usage of Private Endpoints across organizations can create split-brain DNS-like scenarios that require lots of DNS record management overhead.
2. This feature will help to address those scenarios. You should use it where it makes sense, but it shouldn’t be your default.
3. This is still preview so don’t use this in production. When it goes GA, I’ll update this post.

Thanks for reading!