Golden Paths Are a Product. Treat Them Like One.
July 8, 2026Design a Ransomware-Resilient Backup Architecture by Using Azure Backup
July 8, 2026
Purpose
Regulated organizations – banks, insurers, healthcare providers – increasingly run Azure Machine Learning on compute instances that have no outbound internet access. Their data scientists still need modern Python versions and a rich, curated set of libraries. This document exists to close that gap. It captures a repeatable, field-tested pattern for delivering fully loaded Python 3.10-3.14 Jupyter kernels to an air-gapped Azure ML compute – without ever running a package resolver on the locked-down side. The technique itself is version-agnostic: the build, pack, ship, and unpack mechanism works for any Python version that conda can create. This guide uses Python 3.10-3.14 only as a representative range – 3.10 to match the built-in Azure ML kernel and 3.14 as today’s ceiling – but the same steps apply equally to 3.9, a future 3.15, or any other version. The only version-dependent part is the package list itself (for example, the Azure ML SDK v1 tops out at Python 3.11).
What This Document Is
- A hands-on runbook for building, packaging, shipping, and installing custom Python 3.10-3.14 Jupyter kernels on an air-gapped Azure ML compute instance.
- A build-once, ship-an-artifact pattern that uses conda-pack, an Azure Storage blob reached over a private endpoint, and the compute’s managed identity.
- A cleaned, ready-to-install package list – tuned per Python version – in which the known traps from the original Azure ML package export have already been fixed (conda internals, a CLI masquerading as a library, and standard-library-shadowing backports removed; wrong package names and duplicates corrected).
- A troubleshooting reference for the failures that actually happen: RBAC 403s, the azureml v1 Python ceiling, architecture and glibc mismatches, and conda-pack gotchas.
What This Document Is Not
- Not an introduction to Jupyter, conda, or Python packaging fundamentals.
- Not a guide to provisioning Azure ML workspaces, virtual networks, or private endpoints – those are assumed to already exist.
- Not for internet-connected computes, where a normal pip install or conda environment is all you need.
- Not a Windows or ARM recipe – the build box must be Linux x86_64 that matches the target compute.
- Not an official Microsoft product manual; it is a practitioner’s field guide.
Executive Summary
An air-gapped Azure ML compute presents two problems at the same time: it cannot reach PyPI or conda, and it ships with a single built-in Python 3.10 kernel. This guide solves both with one idea – build the environment where internet exists, freeze it into a single self-contained artifact, and ship that artifact across the air gap. The same mechanism works for any Python version conda can build – the 3.10-3.14 range used throughout is only a worked example, not a limit.
- Build once on an internet-connected Linux x86_64 box, then conda-pack the environment into a single relocatable tarball.
- Upload the tarball to a blob container the compute can reach over a private endpoint; on the compute, install_offline_kernel.py uses the managed identity to download and unpack it – with zero package resolution on the locked-down side.
- Clean the raw package list first: strip conda internals, a CLI masquerading as a library, and backports that shadow the standard library.
- Keep the azureml SDK v1 for Python 3.10 and 3.11; drop it for 3.12-3.14 and use azure-ai-ml (v2) instead.
- Blob upload and download need data-plane roles (Storage Blob Data Contributor or Reader) – control-plane Owner or Contributor is not enough.
- Custom kernels persist across compute stop/start; only deleting the compute removes them.
Here is a fun situation I keep running into at regulated customers. A data scientist opens a brand-new Azure ML compute instance inside a locked-down bank network, types pip install pandas, and gets… nothing. No wheels. No index. No error worth reading. Just a cursor blinking back with the quiet confidence of a machine that has never once touched the internet and does not intend to start now.
That is the environment we are dealing with: an air-gapped compute. And air-gapping is only the first of two problems. The second is that Azure ML Studio ships exactly one built-in kernel — Python 3.10. If you need 3.11, 3.12, 3.13, or 3.14, or you want that 3.10 kernel preloaded with your organisation’s curated data-science stack instead of the bare default, the only supported path is to register your own Jupyter kernel. On a normal box you’d pip your way there in a coffee break. Here you can’t, for the reason above.
This post is the pattern I use to solve both at once: build the environment somewhere that does have internet, freeze it into a single self-contained artifact with conda-pack, ship that artifact to a Storage blob the compute can reach over a private endpoint, and unpack it in place. No dependency resolution on the secure side. No firewall exceptions. No six-week change request that ends in “no.”
The short version
- Two constraints, one technique. Air-gapped compute (no PyPI/conda) and a single built-in 3.10 kernel. Custom kernels solve the version/packages gap; conda-pack solves the no-internet gap.
- Build once, ship a tarball. Create the env on an internet-connected Linux x86_64 box, conda-pack it, upload to blob, download + unpack on the compute. Zero resolver runs on the locked-down side.
- Don’t trust the raw package list. A curated AML image’s package dump contains conda internals, a CLI masquerading as a library, and two backports that shadow the standard library. Clean it first (Section 4).
- azureml SDK v1 caps you at Python 3.11. Keep azureml-* for 3.10/3.11; drop it on 3.12–3.14 and use azure-ai-ml (v2).
- “Owner” does not grant blob I/O. Uploads/downloads need a data-plane role (Storage Blob Data Contributor / Reader), not control-plane Owner or Contributor.
- Kernels survive stop/start. They live on the persistent user disk; only deleting the compute wipes them.
Before you start
Five things need to be true before any of the commands below will work. Confirm them now, not at Section 6:
- A Linux x86_64 build box with internet — Ubuntu 22.04 or similar. Must be x86_64 (see the arch note in Section 3) and ideally a similar/older OS than the compute (see the glibc note there too).
- An Azure Storage account reachable from the compute — over a private endpoint from the compute’s VNet, with a container to hold the tarball.
- A managed identity on the compute instance — system- or user-assigned, attached to the compute (not just the workspace), with Storage Blob Data Reader on that account.
- Rights to assign RBAC — Owner or User Access Administrator on the storage account, so you can grant the data-plane roles in Section 6. If you don’t have this, line up whoever does before you begin.
- Terminal access to the compute — via AML Studio, where you’ll run the installer as azureuser.
1. Why the obvious approaches don’t work
Before the solution, it’s worth being honest about why the easy options fail, because someone on the call will suggest all of them:
|
The tempting shortcut |
Why it doesn’t survive contact with an air-gapped compute |
|
pip install -r requirements.txt |
No route to PyPI. DNS or connect timeout. There is nothing to resolve against. |
|
conda create -n env python=3.12 … |
Can’t reach conda channels, and now also blocks on the Anaconda ToS endpoint it can’t contact. |
|
Just use the built-in kernel |
It’s Python 3.10 only, with the stock package set. No version choice, no curated stack. |
|
“Can we open the firewall briefly?” |
In a regulated tenant, that’s a security review measured in weeks and usually ends in a polite no. |
|
Copy a wheelhouse folder by hand |
Fine for pure-Python, but collapses the moment a package needs a compiled binary the target can’t build. |
The insight: stop trying to make the compute reach out. Build everything outside, freeze it into one self-contained artifact — interpreter, packages, and native .so files included — and hand it to the compute through the one channel it already trusts: its own VNet’s private endpoint to Azure Storage.
2. The approach: build, pack, ship, unpack
Four steps. Assemble the env on a machine with internet, compress it with conda-pack into one .tar.gz, upload it to a blob container, and run a small installer on the compute that downloads, unpacks, and registers the kernel.
Why conda-pack instead of rebuilding from a pip freeze on the target? Because a freeze from a curated AML image is not a clean, reinstallable manifest. That env was assembled by conda plus internal feeds, so its pins are internally contradictory — they will not resolve on a fresh Python. conda-pack sidesteps all of it by shipping the exact binaries that already work. The target never runs a solver, so there is nothing left to conflict.
|
📌 Note A conda-pack tarball is a frozen snapshot of a working env — same site-packages, same shared objects, same interpreter. Once conda-unpack rewrites the embedded absolute paths, it runs exactly as if it had been built on the compute. No network, no channels, no ToS prompt. |
3. Part 1 — Build the environment (Linux x86_64)
Where: an internet-connected Linux x86_64 box (Ubuntu 22.04 or similar). The architecture is not optional. Build on ARM64 or a Mac and the compute rejects the binary with Exec format error. There is no cross-arch fallback; the interpreter simply won’t run.
|
⚠️ Gotcha Match more than just the CPU. conda-pack ships compiled binaries, so they link against the build box’s system libraries — glibc in particular. Build on a newer OS than the compute (e.g. Ubuntu 24.04 for an older compute image) and you can hit GLIBC_2.xx not found at import time. Build on an OS that’s the same or older than the target, and match the Python patch line, to stay safe. |
3.1 Install Miniconda and accept the ToS (once)
|
curl -fSL -o miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh bash miniconda.sh -b -p $HOME/miniconda source $HOME/miniconda/etc/profile.d/conda.sh conda init bash && exec bash
# Accept the ToS HERE, on the box that can actually reach the endpoint: conda tos accept –override-channels –channel https://repo.anaconda.com/pkgs/main || true conda tos accept –override-channels –channel https://repo.anaconda.com/pkgs/r || true conda install -n base -c conda-forge conda-pack -y |
3.2 Create the env (ipykernel is mandatory)
|
# The env name follows through the whole workflow. Pick your version: conda create -n aml-py310 -c conda-forge python=3.10 ipykernel pip setuptools wheel -y conda create -n aml-py314 -c conda-forge python=3.14 ipykernel pip setuptools wheel -y |
Include ipykernel. The installer on the compute runs a preflight check for it and aborts with a rebuild message if it’s missing. It comes from conda here, which is why it does not appear in the pip requirements files. Leave it out and you’ll do the whole build, upload, and download before finding out nothing registers.
3.3 Install the curated package set
A note on the requirements file: there’s no magic requirements-py310.txt attached to this post — it stands in for your own curated package list. Bring whatever set your team needs; Section 4 shows how to sanitise it and Section 5 covers the per-version differences. The one-file-per-Python-version layout below is just how I keep the versions straight.
|
conda activate aml-py310 pip install -r requirements/requirements-py310.txt pip check # surface residual conflicts BEFORE you pack
# Freeze exact versions so a rebuild next month is byte-for-byte repeatable: pip freeze > requirements/requirements-py310.lock.txt |
On 3.12 / 3.13 / 3.14 this set is built around azure-ai-ml (v2); the deprecated azureml-* v1 family is intentionally absent. Section 5 explains why.
3.4 Pack it
Run this from base (where conda-pack lives), not from inside the env:
|
conda deactivate mkdir -p ~/aml-offline && cd ~/aml-offline conda pack -n aml-py310 -o aml-py310.tar.gz –ignore-missing-files ls -lh aml-py310.tar.gz # ~150 MB minimal … several GB with the full DS stack |
|
⚠️ Gotcha –ignore-missing-files is required for these envs. Because the env mixes conda with a large pip layer, pip upgrades a few conda-installed packages (e.g. packaging) and rewrites their dist-info. Without the flag, conda-pack aborts with CondaPackError: Files managed by conda were found to have been deleted/overwritten. The flag packs what’s actually on disk — the consistent pip versions. Add –force if the output file already exists. |
3.5 Smoke-test before shipping
|
mkdir -p /tmp/test && tar -xzf aml-py310.tar.gz -C /tmp/test /tmp/test/bin/conda-unpack /tmp/test/bin/python -c “import sys, ipykernel; print(sys.version, ipykernel.__version__)” file /tmp/test/bin/python # MUST say: ELF 64-bit LSB executable, x86-64 rm -rf /tmp/test |
Run the file check every single time. If it doesn’t say x86-64, stop and rebuild now. Shipping it anyway just moves the same failure to the compute, where it takes longer to spot.
4. Cleaning the curated package list – (Depends on the package list , Here we can see some examples)
A raw package list dumped from a curated AML image looks fine at a glance. Installed as-is, it either fails the build outright or, worse, silently breaks the env in ways you won’t notice until a notebook misbehaves three weeks later. I’ve hit six recurring problems in these lists. Here they are, with the fix for each.
|
Offender |
The problem |
The fix |
|
conda, conda-content-trust, conda-package-handling, pycosat, boltons, ruamel-yaml-conda |
These are components of the base conda install, not project libraries. pycosat needs a C toolchain, and a pip-installed conda can shadow the real one. |
Remove all six. Keep ruamel.yaml (the real PyPI build) for YAML. |
|
azure-cli |
It’s a command-line app, not a notebook library, and it pins hundreds of exact azure-mgmt-* / azure-storage-* versions. Single biggest source of conflicts. |
Remove it. Install the az CLI separately if you need it — don’t let it dictate your SDK versions. |
|
azureml-core + the azureml-* v1 family |
Deprecated (EOL June 2026) and version-capped. Binary deps (azureml-dataprep-rslex, dotnetcore2) only ship wheels to CPython 3.11. |
Keep for 3.10/3.11 only. On 3.12–3.14, drop it and use azure-ai-ml (v2). |
|
brotlipy |
A conda-image artifact superseded by Brotli. On newer Python it often has no wheel and wants a C compiler. |
Remove. requests / urllib3 pull in Brotli automatically when needed. |
|
dataclasses==0.6, uuid==1.30 |
Ancient backports that shadow the standard library on modern Python. These cause real, subtle breakage — not just noise. |
Remove on sight. |
|
azure-storage-filedatalake (and duplicates) |
That name 404s on PyPI — it’s the import name. azure-identity and databricks-sql-connector also appear twice. |
Correct to azure-storage-file-datalake (dashes). Deduplicate the rest. |
|
💡 Tip Do the resolution out in the open, on the build box, where pip check and error messages are visible and cheap to fix. The whole point of shipping a tarball is that the locked-down compute never runs a solver — so make sure the solver runs cleanly before you pack. |
5.The azureml v1 version ceiling
This one deserves its own section because it silently dictates which Python versions you can even offer. The azureml-* v1 SDK — azureml-core and its binary friends azureml-dataprep-rslex, azureml-dataprep-native, dotnetcore2 — is deprecated (EOL June 2026) and only ships wheels for CPython 3.8–3.11 (rslex reaches 3.12, but not 3.13/3.14). On 3.13/3.14 the install simply cannot resolve. So the strategy is version-aware:
|
Python |
Requirements file |
azureml-* v1 |
Notebook SDK |
|
3.10 |
requirements-py310.txt |
Included |
azureml-core |
|
3.11 |
requirements-py311.txt |
Included |
azureml-core |
|
3.12 |
requirements-py312.txt |
Dropped |
azure-ai-ml |
|
3.13 |
requirements-py313.txt |
Dropped |
azure-ai-ml |
|
3.14 |
requirements-py314.txt |
Dropped |
azure-ai-ml |
The v2 azure-ai-ml SDK is tested on Python 3.8–3.14 and covers workspace, compute, datastore, job, pipeline, and AutoML operations — so notebooks keep working on the newer interpreters without the deprecated stack.
The filenames in this table are just a per-version naming convention for your own package lists — not files you download from anywhere. Keep one list per Python version and the rest of the workflow is identical.
6. Part 2 — Upload the tarball to blob
The tarball is packed and tested. Now drop it in a blob container the compute can reach over its private endpoint. This is where a role-assignment subtlety trips up almost everyone.
6.1 Upload options
Portal (easiest): Storage account → Containers → your container → + Add Directory offline-kernels → Upload the .tar.gz. The Portal uses the account key, so it sidesteps the RBAC issue below entirely.
azcopy with the build VM’s managed identity:
|
curl -fSL -o azcopy.tar.gz https://aka.ms/downloadazcopy-v10-linux tar -xzf azcopy.tar.gz sudo cp azcopy_linux_amd64_*/azcopy /usr/local/bin/ && sudo chmod +x /usr/local/bin/azcopy
azcopy login –identity # add –identity-client-id for user-assigned azcopy copy “$HOME/aml-offline/aml-py310.tar.gz” “https://.blob.core.windows.net//offline-kernels/aml-py310.tar.gz” |
6.2 The 403 that ignores your title
You’re a subscription Owner. Azure is unimpressed ☹. The upload still fails with 403 AuthorizationPermissionMismatch. Owner and Contributor are control-plane roles; they let you manage the storage account. Reading and writing blobs is a data-plane operation, and it needs a data-plane role the control-plane ones don’t include.
I lost most of an afternoon to this during a bank onboarding earlier this year. I was sure the role assignment just hadn’t propagated yet, so I kept re-running the copy and waiting. The real problem was that I’d been granted Contributor on the account and never the data-plane role. About ninety seconds after the right role landed, the upload went through. Check the role before you blame propagation.
|
Operation |
Role you actually need |
Plane |
|
Upload the tarball (build side) |
Storage Blob Data Contributor |
Data-plane |
|
Download the tarball (compute side) |
Storage Blob Data Reader |
Data-plane |
|
Owner / Contributor |
Manage the account — not sufficient for blob I/O |
Control-plane |
sid=$(az storage account show -n –query id -o tsv)
az role assignment create
–assignee
–role “Storage Blob Data Contributor”
–scope “$sid”
# wait ~1–2 min for RBAC propagation, then retry the copy
⚠️ Gotcha
Whatever account you upload to must be the same one the installer points at, and the compute’s own managed identity needs Storage Blob Data Reader on it. Mismatch the accounts and the download 403s the same way, except now it fails on the compute instead of your build box.
7. Part 3 — Install the kernel on the compute
On the compute’s terminal (AML Studio → the compute → Terminal, where you’re azureuser), point install_offline_kernel.py at the right env and run it. The full script is in the Appendix at the end of this post.
|
# Constants at the top of install_offline_kernel.py: STORAGE_ACCOUNT = “” CONTAINER = “data” BLOB_PREFIX = “offline-kernels” ENV_NAME = “aml-py310” # aml-py311 / py312 / py313 / py314 DISPLAY_NAME = “Python 3.10 (AML Custom)” |
|
# Only for a user-assigned MI, so IMDS knows which identity to use: export AZURE_CLIENT_ID=””
python3 install_offline_kernel.py |
What the script does:
- Bootstraps the SDK — installs azure-identity + azure-storage-blob into the user site (~5 MB, one-time). It also supports a local tarball or a SAS URL, so it works with zero Azure auth if you pre-stage the file.
- Authenticates via DefaultAzureCredential — picks up the compute’s managed identity from IMDS.
- Downloads offline-kernels/aml-py310.tar.gz over the private endpoint.
- Extracts into /anaconda/envs/aml-py310 (sudo only if the path isn’t user-writable).
- Runs conda-unpack — rewrites the embedded absolute paths so the env works in its new home.
- Preflight-checks that the env can import ipykernel, and aborts with an exact rebuild command if not.
- Registers the kernel as “Python 3.10 (AML Custom)” for azureuser. Idempotent — it skips the unpack if the env already exists.
Force a clean rebuild after re-uploading a new tarball:
|
sudo rm -rf /anaconda/envs/aml-py310 rm -rf ~/.local/share/jupyter/kernels/aml-py310 python3 install_offline_kernel.py |
Verify: in AML Studio open a notebook → kernel picker → select “Python 3.10 (AML Custom)” and run import sys; print(sys.version). On 3.12+, reach for the v2 SDK: from azure.ai.ml import MLClient.
8. When it doesn’t work
The short list I check first, in order of how often each one is the culprit:
|
Symptom |
Cause |
Fix |
|
Exec format error: …/bin/python |
Tarball built on the wrong arch/OS (ARM64 or macOS). |
Rebuild on Linux x86_64 (Section 3), re-upload, retry. |
|
ERROR: … cannot import ipykernel |
Env packed without ipykernel. |
Recreate the env WITH ipykernel, repack, re-upload, clean-rebuild. |
|
DefaultAzureCredential failed … |
No MI on the compute, or MI lacks the data role. |
Attach an MI to the compute (not the workspace) + Storage Blob Data Reader. |
|
HTTP 403 from blob |
MI lacks the data-plane role on this account/container. |
Add Storage Blob Data Reader on the same account you uploaded to. |
|
HTTP 404 from blob |
Wrong account/container/blob path in the script. |
Re-check the constants against the actual blob path in the Portal. |
|
Multiple identities from IMDS |
User-assigned MI; the SDK can’t guess which one. |
export AZURE_CLIENT_ID=”” first. |
⚠️ Gotcha
The single most common face-plant is the workspace-vs-compute identity mix-up. Granting a role to the AML workspace identity does nothing for IMDS calls coming from the compute. The managed identity must be assigned to the compute instance itself. Check that before you check anything else.
9. Why this actually works with no internet
Four details make the whole thing hold together. They’re also exactly what a security reviewer will ask about, so it helps to have the answers ready.
- IMDS is not the internet. Managed-identity auth works with zero outbound access because IMDS lives at 169.254.169.254 — a link-local address served by the Azure host, not a public endpoint. The blob download then rides the Storage account’s private endpoint inside the VNet. Nothing touches the public internet.
- No Anaconda ToS prompt on the secure side. You accepted the ToS on the build box. The compute only ever sees a plain tarball, so no conda channel is ever contacted and the ToS gate never appears.
- Workspace MI ≠ compute MI. RBAC granted to the workspace identity does not authorize IMDS calls from the compute. The identity must live on the compute instance. This is the top cause of a mysterious 403 — worth repeating.
- Kernels persist across stop/start. The kernelspec and env live on the persistent user disk (~/.local/share/jupyter/kernels/ and /anaconda/envs/). A stop/start keeps them; only deleting the compute wipes them — at which point you just re-run the installer.
10. conda-pack gotchas on the build box
The build box has its own small collection of traps. Sharing the scars so you can skip them:
|
Trap |
What you see |
The fix |
|
python -m conda_pack … |
package cannot be directly executed |
The module has no __main__. Use the console script conda-pack … or the conda pack subcommand. |
|
conda pack |
invalid choice: ‘pack’ |
conda-pack isn’t installed in that conda. conda install -n base -c conda-forge conda-pack -y. |
|
Packing a mixed conda+pip env |
Files managed by conda … deleted/overwritten |
Add –ignore-missing-files; add –force if the .tar.gz exists. |
|
conda-pack script missing |
conda-pack: command not found |
python -m pip install conda-pack, then hash -r, then conda-pack -p -o out.tar.gz. |
Wrapping up
You don’t need to punch a hole in the firewall to give a locked-down Azure ML compute a rich, multi-version Python environment. You need to do all the messy dependency resolution outside, where you can see it, freeze the result with conda-pack, ship one tarball over a private endpoint, and register it as a kernel on the other side. Build once, pack once, upload once, install once.
The parts that bite are all predictable once you know they’re there: the arch mismatch, the azureml version ceiling, the data-plane 403, and the workspace-vs-compute identity. Now you know, so they should cost you minutes instead of an afternoon.
Appendix — install_offline_kernel.py
This is the installer referenced in Section 7 — the only script that runs on the air-gapped compute. It fetches the tarball three ways in priority order (a local file, a SAS URL, or blob download via the compute’s managed identity), unpacks it, runs conda-unpack, preflight-checks ipykernel, and registers the kernel. Set the constants at the top, copy it onto the compute, and run python3 install_offline_kernel.py.
📌 Note
Replace the placeholders. The storage account, subscription id, and CLI user below are placeholders — swap in your own values before running. Everything else is ready to use as-is.
|
#!/usr/bin/env python3 “”” Install an offline conda-packed Python kernel on an AML compute instance.
Gets a conda-pack tarball (in priority order): 1. a local tarball on this box (LOCAL_TARBALL, or ~/aml-offline/.tar.gz, or ./.tar.gz) – no Azure auth needed, 2. a SAS URL (BLOB_SAS_URL) – no RBAC data role needed, 3. Azure Blob via DefaultAzureCredential (compute managed identity). Then unpacks it and registers it as a Jupyter kernel for the current user.
Prerequisites (only for option 3 – blob download via managed identity): – A managed identity (system- or user-assigned) attached to the compute instance. – That identity has “Storage Blob Data Reader” on the storage account (DATA-plane role; control-plane Contributor/Owner is NOT sufficient). – For user-assigned MI, set env var AZURE_CLIENT_ID to its client id. “””
from __future__ import annotations
import os import subprocess import sys from pathlib import Path
# —- Configuration ——————————————————— STORAGE_ACCOUNT = “” CONTAINER = “data” BLOB_PREFIX = “offline-kernels”
ENV_NAME = “aml-py310” DISPLAY_NAME = “Python 3.10 (AML Custom)”
# If the tarball is already on this machine (e.g. you built it here), the script # uses it and skips the blob download entirely – no Azure auth / RBAC needed. # Leave “” to auto-search ~/aml-offline/.tar.gz and the current dir, # or set an explicit path. LOCAL_TARBALL = “”
# Optional full SAS URL to the blob, e.g. # https://.blob.core.windows.net///.tar.gz?sv=… # If set, it is used instead of DefaultAzureCredential and bypasses the # “Storage Blob Data Reader” RBAC requirement. Leave “” for managed-identity auth. BLOB_SAS_URL = “”
# Azure auth for the blob download (only used if no local tarball and no SAS URL). # CREDENTIAL_MODE = “cli” -> use your signed-in Azure CLI user; first run: # az login # device-code on a compute # az account set –subscription $SUBSCRIPTION_ID # CREDENTIAL_MODE = “managed” -> use the compute’s managed identity # CREDENTIAL_MODE = “default” -> DefaultAzureCredential chain (MI, then CLI, …) SUBSCRIPTION_ID = “” AZURE_USERNAME = “” # expected CLI user (informational) CREDENTIAL_MODE = “cli”
TARGET_DIR = Path(f”/anaconda/envs/{ENV_NAME}”) TARBALL_PATH = Path(f”/tmp/{ENV_NAME}.tar.gz”) # —————————————————————————
def run(cmd: list[str], **kwargs) -> None: “””Run a subprocess command, streaming output and raising on failure.””” print(f”+ {‘ ‘.join(cmd)}”, flush=True) subprocess.run(cmd, check=True, **kwargs)
def ensure_sdk_packages() -> None: “””Install azure-identity and azure-storage-blob into the current interpreter.””” try: import azure.identity # noqa: F401 import azure.storage.blob # noqa: F401 return except ImportError: pass
print(“Installing azure-identity and azure-storage-blob …”, flush=True) run([ sys.executable, “-m”, “pip”, “install”, “–quiet”, “–user”, “azure-identity”, “azure-storage-blob”, ])
def find_local_tarball() -> Path | None: “””Return an existing local tarball to use instead of downloading, or None.””” candidates = [] if LOCAL_TARBALL: candidates.append(Path(LOCAL_TARBALL).expanduser()) candidates.append(Path.home() / “aml-offline” / f”{ENV_NAME}.tar.gz”) candidates.append(Path.cwd() / f”{ENV_NAME}.tar.gz”) for c in candidates: if c.is_file(): return c return None
def get_credential(): “””Return an Azure credential according to CREDENTIAL_MODE.””” if CREDENTIAL_MODE == “cli”: from azure.identity import AzureCliCredential # Uses the user signed in via `az login` (e.g. ). print(f”Using Azure CLI credential (expected user: {AZURE_USERNAME}).”, flush=True) return AzureCliCredential() if CREDENTIAL_MODE == “managed”: from azure.identity import ManagedIdentityCredential client_id = os.environ.get(“AZURE_CLIENT_ID”) print(“Using the compute’s managed identity.”, flush=True) return ManagedIdentityCredential(client_id=client_id) if client_id else ManagedIdentityCredential() from azure.identity import DefaultAzureCredential print(“Using DefaultAzureCredential chain.”, flush=True) return DefaultAzureCredential()
def download_blob(env_name: str, dest: Path) -> None: “””Download the conda-pack tarball via SAS URL or an Azure credential.””” from azure.storage.blob import BlobClient
dest.parent.mkdir(parents=True, exist_ok=True)
if BLOB_SAS_URL: print(f”Downloading via SAS URL -> {dest}”, flush=True) client = BlobClient.from_blob_url(BLOB_SAS_URL) else: ensure_sdk_packages() credential = get_credential()
blob_name = f”{BLOB_PREFIX}/{env_name}.tar.gz” url = f”https://{STORAGE_ACCOUNT}.blob.core.windows.net/{CONTAINER}/{blob_name}” print(f”Downloading {url} -> {dest}”, flush=True) client = BlobClient.from_blob_url(url, credential=credential)
try: with open(dest, “wb”) as f: stream = client.download_blob(max_concurrency=4) stream.readinto(f) except Exception as exc: # noqa: BLE001 msg = str(exc) if “AuthorizationPermissionMismatch” in msg or “not authorized” in msg: sys.exit( “ERROR: 403 AuthorizationPermissionMismatch downloading the tarball.n” f”The credential in use (CREDENTIAL_MODE='{CREDENTIAL_MODE}’) lacks then” f”DATA-plane role ‘Storage Blob Data Reader’ on storage accountn” f”‘{STORAGE_ACCOUNT}’. (azcopy login / control-plane Contributor do NOT grant this.)n” “Fix ONE of:n” f” 1. Grant that role to the principal (e.g. {AZURE_USERNAME} for CLI mode,n” ” or the compute’s managed identity for managed mode), orn” ” 2. Set BLOB_SAS_URL to a SAS link (no RBAC needed), orn” f” 3. Put the tarball at ~/aml-offline/{env_name}.tar.gz to skip download.nn” f”Original error:n{msg}” ) raise print(f”Downloaded {dest.stat().st_size:,} bytes”, flush=True)
def ensure_target_dir(target: Path) -> None: “””Create target dir, using sudo if necessary, and chown to current user.””” if target.exists(): return
user = os.environ.get(“USER”, “azureuser”) try: target.mkdir(parents=True, exist_ok=True) except PermissionError: print(f”Need sudo to create {target}”, flush=True) run([“sudo”, “mkdir”, “-p”, str(target)]) run([“sudo”, “chown”, “-R”, f”{user}:{user}”, str(target)])
def install_kernel() -> None: if (TARGET_DIR / “bin”).is_dir(): print(f”Env {ENV_NAME} already exists at {TARGET_DIR}, skipping unpack.”, flush=True) else: local = find_local_tarball() if local is not None: print(f”Using local tarball {local}; skipping blob download.”, flush=True) src = local downloaded = False else: download_blob(ENV_NAME, TARBALL_PATH) src = TARBALL_PATH downloaded = True
ensure_target_dir(TARGET_DIR)
print(f”Extracting {src} -> {TARGET_DIR}”, flush=True) run([“tar”, “-xzf”, str(src), “-C”, str(TARGET_DIR)])
print(“Running conda-unpack …”, flush=True) run([str(TARGET_DIR / “bin” / “conda-unpack”)])
# Only remove the temp download, never a user-provided local tarball. if downloaded: try: TARBALL_PATH.unlink() except FileNotFoundError: pass
print(f”Registering Jupyter kernel ‘{ENV_NAME}’ …”, flush=True) env_python = str(TARGET_DIR / “bin” / “python”)
# Sanity-check: env must actually contain ipykernel, otherwise the tarball # was built without it and needs to be rebuilt on the Linux build box. check = subprocess.run( [env_python, “-c”, “import ipykernel, sys; print(ipykernel.__version__)”], capture_output=True, text=True, ) if check.returncode != 0: sys.exit( f”ERROR: {env_python} cannot import ipykernel.n” f”The conda-pack tarball was built without ipykernel. Rebuild on the Linux box with:n” f” conda create -n {ENV_NAME} -c conda-forge python= ipykernel pip setuptools wheel -yn” f” conda-pack -n {ENV_NAME} -o {ENV_NAME}.tar.gz –ignore-missing-filesn” f”then re-upload and clean up: sudo rm -rf {TARGET_DIR}nn” f”stderr from check:n{check.stderr}” ) print(f”ipykernel {check.stdout.strip()} found in env.”, flush=True)
run([ env_python, “-m”, “ipykernel”, “install”, “–user”, “–name”, ENV_NAME, “–display-name”, DISPLAY_NAME, ])
def main() -> int: print(f”Starting offline {ENV_NAME} kernel setup on AML compute instance …”, flush=True) install_kernel()
kernels_dir = Path.home() / “.local” / “share” / “jupyter” / “kernels” if kernels_dir.is_dir(): print(f”nInstalled kernels in {kernels_dir}:”, flush=True) for k in sorted(kernels_dir.iterdir()): print(f” – {k.name}”)
print(“nOffline kernel setup completed.”, flush=True) return 0
if __name__ == “__main__”: sys.exit(main()) |
References
- conda-pack documentation — packaging conda envs into relocatable tarballs.
- Azure ML: identity-based access for compute — attaching a managed identity to the compute instance.
- Authorize access to blobs with Microsoft Entra ID — why data-plane roles, not Owner, grant blob I/O.
- Azure Instance Metadata Service (IMDS) — the link-local endpoint that makes MI auth work offline.
- azure-ai-ml (SDK v2) — the supported SDK on Python 3.12–3.14.
- Azure ML CLI/SDK v1 deprecation — background on the v1 version ceiling.