AzureFeeds

For two years, the AI conversation in healthcare fixated on a single question: which large language model is best? That question is now out of date. The frontier didn’t just get smarter, it got plural. The most capable healthcare AI systems being built today are not one model but a portfolio: large reasoning models, small efficient models, and specialized models for voice, images, and video, each doing the part of the job it does best.

Microsoft Foundry makes that portfolio practical. Its model catalog spans over 1,900 models from Microsoft, OpenAI, Anthropic, Meta, Mistral, xAI, DeepSeek, Hugging Face, and others, together with the tools to compare, ground, evaluate, and govern them in one place. For healthcare and life sciences (HLS) leaders, the strategic shift is from *pick the model* to *design the portfolio*.

This post walks through one concrete, high-value pattern, turning a clinician’s spoken encounter into a verified, structured note, and uses it to show how a model portfolio works end to end, what it costs at scale, and how to keep it safe. Several of the newest capabilities referenced here are in preview; confirm current naming and availability before you build.

Why “model” is now a plural

A modern model strategy spans two axes: capability size and modality. Getting both right is the whole game.

On the size axis, Microsoft Foundry presents a clear menu. GPT-5-class models are the most capable for complex, multi-step reasoning and multimodal scenarios. GPT-4.1 strikes a balance of capability and cost for production workloads, and GPT-4.1 mini is tuned for the lowest-latency, highest-throughput jobs. Alongside them sit Microsoft’s Phi small language models, Phi-4, Phi-4-mini, and Phi-4-multimodal, designed for strong reasoning with efficient deployment, including on-device through Foundry Local.

On the modality axis, the catalog reaches well beyond text. Microsoft’s MAI (Microsoft AI) family includes MAI-Voice-1, an expressive neural text-to-speech model for natural, long-form English speech, and MAI-Transcribe, a speech-recognition model tuned for both accuracy and efficiency. Phi-4-multimodal accepts text, images, and audio in a single model. And a dedicated set of healthcare AI models, such as MedImageInsight, MedImageParse, and CXRReportGen, brings multimodal reasoning to medical imaging, pathology, and radiology. The point isn’t to use them all; it’s that the right answer to almost any real HLS workflow is several of them, composed.

A multimodal model portfolio in Microsoft Foundry: diverse inputs, the smallest capable model per task, governed outputs.

The business problem: documentation, education, and unread data

Start with the problem clinicians feel most acutely: documentation. Writing notes, orders, and after-visit summaries consumes hours that clinicians would rather spend with patients, and it is a leading contributor to burnout. Ambient AI scribing, listening to the visit and drafting the note, is one of the clearest near-term wins in healthcare AI.

But documentation is only the first domino. The same portfolio that drafts a note can generate patient-education material in plain language and natural voice, summarize a specialist’s imaging findings for a referring physician, and make dense clinical text understandable for caregivers. Much of healthcare’s most valuable data, audio, images, and scanned documents, has historically gone unread by software. Multimodal models change that.

The catch is economics. If every one of these tasks calls the largest available model, the combined cost and latency make the program impossible to scale. Right-sizing, matching each task to the smallest model that can do it well, is what turns a compelling demo into a sustainable service.

A technical walkthrough: from voice to verified note

Consider the ambient-documentation workflow in detail. It is a useful template because it touches every part of the portfolio, speech, small models, large models, retrieval, evaluation, and human review, in a single pass.

The inputs

The pipeline starts with two inputs: the ambient audio of the encounter, captured with patient consent, and structured context from the electronic health record (EHR), the patient’s problem list, current medications, and recent results. Related images or documents can come along for the ride when the visit calls for them.

Which model does what

Here is where the portfolio earns its keep. No single model touches the whole job:

• Speech-to-text. A speech model (Azure AI Speech, or MAI-Transcribe for high accuracy and efficiency) converts the conversation into an accurate, timestamped transcript.


• Extraction and structuring. A small Phi model parses the transcript into discrete elements, problems, medications, and follow-ups, at a fraction of the cost and latency of a frontier model, and it can run close to the data.


• Grounding (RAG). A retrieval step injects only approved context, the organization’s clinical guidelines and the specific patient’s record, so the draft reflects real, current, permissioned information rather than the model’s memory.


• Drafting. A larger GPT-class model composes the narrative note, reconciling the transcript, the structured extraction, and the grounded context into clean clinical prose.


• Patient-facing media. For the after-visit summary, MAI-Voice-1 can render a plain-language version as natural speech, and avatar or video generation can produce short education clips for patients or providers.


• Imaging insights. When images are involved, healthcare models such as MedImageInsight or CXRReportGen surface findings for a specialist to review, in supported, non-diagnostic configurations.

Orchestration, evaluation, and the human checkpoint

Foundry orchestrates these calls, and two stages matter as much as the models themselves. Evaluation runs automated checks on every draft, grounding and faithfulness, safety, and completeness, and routes low-confidence outputs for extra scrutiny. Human review is the non-negotiable final gate: the clinician reads, edits, and signs off before anything reaches the chart. The model drafts; the clinician decides.

From voice to verified note: each stage uses the smallest capable model, and a clinician signs off before anything reaches the chart.

1. Capture with consent. Record ambient audio and pull the relevant EHR context at the point of care.


2. Transcribe. Convert audio to text with a speech model sized for accuracy and throughput.


3. Extract. Use a small Phi model to structure the transcript into problems, medications, and actions.


4. Ground. Retrieve only approved guidelines and the patient’s own record as context (RAG).


5. Draft. Have a larger model compose the note from transcript, structure, and grounded context.


6. Evaluate. Score each draft for faithfulness, safety, and completeness; flag the doubtful ones.


7. Review and sign off. The clinician edits and approves; the edits feed back to improve prompts and routing.

The economics at scale: right-sizing the portfolio

The difference between a pilot and a platform is cost-to-serve. A right-sized portfolio attacks it on several fronts at once.

Match model to task. Most steps above, transcription, extraction, classification, routing, are high-volume but not cognitively hard. They belong on small models like Phi, which deliver strong results at far lower cost and latency. Reserve the large, expensive models for the genuinely difficult step: composing and reconciling the final note. Spending frontier-model dollars on routine extraction is the single most common way AI programs blow their budget.

Cache and batch. Identical or near-identical requests, a common guideline lookup, a standard education snippet, can be cached and reused. Non-urgent work, overnight summarization or cohort processing, can be batched for throughput rather than instant response.

Run small models close to the data. Through Foundry Local, eligible Phi workloads can run on-device or at the edge, cutting per-call cost and keeping sensitive audio and text on local infrastructure, valuable in a hospital where latency and data residency both matter.

Make unit economics predictable. For specialized work, Microsoft’s premium healthcare models are offered as pay-as-you-go serverless endpoints with predictable per-image economics, so finance can model cost per study rather than guess at infrastructure. The net effect: cost and latency scale with the difficulty of the work, not with raw volume.

One large model for everything versus a right-sized portfolio: matching model size to task difficulty is what changes the economics.

Governance: safety, responsible AI, and human oversight

In healthcare, governance is not a layer you add at the end; it is the foundation. A right-sized portfolio actually makes governance easier, because each model has a narrower, better-understood job.

• Responsible AI by design. Foundry includes built-in evaluation, content safety, and observability, so teams can test models against their own data and monitor them in production.


• Data boundaries. The pipeline runs within your tenant, under your existing identity, permissions, and data-protection controls; grounding restricts the model to approved, permissioned sources.


• Human in the loop. Microsoft’s healthcare models are explicitly designed to support, never replace qualified professionals; many are in preview and intended for research and development, not autonomous clinical decisions. A clinician signs off on every clinically meaningful output.


• Provenance and feedback. Capturing edits and approvals creates an audit trail and a continuous-improvement loop for prompts, routing, and model selection.

Responsibility stays with the deploying organization: you remain accountable for verifying outputs, meeting applicable healthcare regulations, and obtaining any clearances required before a model informs clinical decision-making. The portfolio approach supports that accountability by keeping each step inspectable and each consequential decision human.

How to start: a 30-day mini-playbook

1. Pick one painful, high-volume workflow. Ambient documentation, referral summaries, or patient-education drafting are strong first candidates.


2. Map the modalities. List the inputs (audio, text, images) and outputs (note, summary, voice) so you know which model types you need.


3. Draft the portfolio, smallest-first. Assign each step to the smallest capable model and reserve large models for the hardest step only.


4. Ground it. Connect approved guidelines and the relevant record through retrieval before you optimize anything else.


5. Stand up evaluation early. Define faithfulness, safety, and completeness checks, and a human-review step, from day one, not after launch.


6. Measure cost and latency per task, then tune routing, caching, and batching against real usage.


7. Expand by reuse. Once the portfolio works for one workflow, most of it, speech, extraction, grounding, governance, carries straight over to the next.

The organizations that win with healthcare AI over the next year won’t be the ones who picked the single “best” model. They’ll be the ones who designed the best portfolio, matching capability and modality to each task, grounding every output, and keeping clinicians firmly in the loop. Right-sizing isn’t a cost-cutting afterthought; it is the architecture that makes AI affordable, fast, and safe enough to use everywhere.

Subscribe to the Microsoft Healthcare & Life Sciences blog for weekly, practical deep dives on building AI that scales. And one question to leave you with: if you mapped your top clinical workflow today, how many of its steps actually need a frontier model, and how many are quietly overpaying for one?

Sources

• What is Microsoft Foundry?


• Microsoft Foundry Models overview


• Healthcare AI models in Microsoft Foundry


• What is MAI-Voice?


• MAI-Transcribe in Azure Speech


• Phi open models

Related links

• Microsoft Foundry — Build, evaluate, and deploy AI across a catalog of 1,900+ models.


• Phi small language models — Microsoft’s efficient open models, including Phi-4-multimodal.


• Healthcare AI models in Microsoft Foundry — Multimodal foundation models for medical imaging and more.


• Azure AI Speech — Speech-to-text, text-to-speech, and the MAI voice and transcribe models.


• Microsoft Responsible AI — Principles and practices for building AI responsibly.


• Healthcare & Life Sciences Tech Community — More HLS deep dives and announcements.