AzureFeeds

This article walks you through an application for Procure-to-Pay anomaly detection using Azure AI Foundry Agent Service. It analyzes purchase invoice images to detect procurement anomalies and compliance issues. The key capabilities of Azure AI Foundry it showcases are:

1. Agent Service that performs a multi-step business process with only natural language instructions. The Application has little to no business logic in the code.


2. Invocation of a variety of automated tool actions required during the multi steps business process. It showcases the recently announced turnkey integration with MCP Server, Azure Logic Apps. In addition, it uses a) visual reasoning of images using gpt-4o models, b) turnkey vector search, and c) reasoning to apply business rules and perform p2p anomaly detection.


3. The ability of Agent Service in Azure AI Foundry to:
   
   
   
   • most importantly, all the steps in the business process are performed through a single call from the Application to the Agent Service. The Agent orchestrates the steps autonomously, based on the instructions in Natural language. The Client application itself contains no business logic in code.
   
   
   • handle application state between different tool calls. It determines what to extract from the output of one tool call to pass as input to the next.
   
   
   • orchestrating the business process from start to end, sequencing them to achieve the business end.
   
   
   
   

Architecture of the Solution

Shown below is the architecture of the Solution. The Client Application is a python-based console App that calls the P2P Anomaly Detection Agent running in the Azure AI Foundry Agent Service.

Solution Components

 

Some of the key aspects of the Solution are:

1) There is only one call from the Client Application to the Agent in Azure AI Foundry. The rest of the steps are performed by the Agent Service autonomously. The code snippet below shows the client application passing the user provided Invoice image and user instruction to the Agent in Azure AI Foundry.

2) Observe that there is no business logic in the code whatsoever, and there is no code written to perform any of the tool actions either. These are autonomously done by the Agent itself.

 

def process_anomaly_detection(self, user_prompt: str, image_path: str, verbose: bool = True) -> str:
“””
Process the anomaly detection request with proper error handling and retry logic.

Args:
user_prompt: User’s prompt for anomaly detection
image_path: Path to the invoice image

Returns:
Response from the AI agent

Raises:
Exception: For various processing errors
“””
try:
# Create message content
self._show_progress(“Creating message content…”, verbose)
content_blocks = self.create_message_content(user_prompt, image_path)

# Create a new thread for this conversation
self._show_progress(“Creating conversation thread…”, verbose)
thread = self.client.threads.create()
logger.info(f”Created thread, ID: {thread.id}”)

# Create the message
self._show_progress(“Sending message to agent…”, verbose)
message = self.client.messages.create(
thread_id=thread.id,
role=”user”,
content=content_blocks
)

# Run the agent with proper error handling
self._show_progress(“Processing with AI agent (this may take a moment)…”, verbose)
run = self.client.runs.create_and_process(
thread_id=thread.id,
agent_id=self.agent.id
)

if run.status == “failed”:
error_msg = f”Agent run failed: {run.last_error}”
logger.error(error_msg)
raise Exception(error_msg)

# Retrieve and process messages
self._show_progress(“Retrieving analysis results…”, verbose)
messages = self.client.messages.list(
thread_id=thread.id,
order=ListSortOrder.ASCENDING
)

# Extract the agent’s response
agent_response = “”
for message in messages:
if message.role == “assistant” and message.text_messages:
agent_response = message.text_messages[-1].text.value
break

if not agent_response:
raise Exception(“No response received from the agent”)

self._show_progress(“Analysis complete!”, verbose)
logger.info(“Successfully processed anomaly detection request”)
return agent_response

except Exception as e:
logger.error(f”Error processing anomaly detection: {e}”)
raise

 

Note: The steps to configure the Solution is covered in the GitHub Repo link shared later in this article, and hence not discussed here.

Agent Instructions

As mentioned already, the Client application does not contain any Business logic. The latter is provided as natural language instructions to the Foundry Agent. Show below are the instructions configured on the P2P Agent.

This is a Procure to Pay process. You will be provided with the Purchase Invoice image as input.

Note the sequence of execution you must adhere to strictly:
– Step 2 must be performed only after Step 1 is performed
– Step 3 below must be performed only after Step 1 and Step 2 are completed.
– Step 5 must be performed only after Step 1, Step 2, Step 3 and Step 4 have been performed successfully.
– Step 6 must be performed only after Step 5 is performed

Step 1: As a first step, you will extract the Contract ID and Supplier ID from the Purchase Invoice image along with all the line items from the Invoice in the form of a table.
Step 2: You will then use the function tool by passing the Contract ID and Supplier ID to retrieve the contract details.
Step 3: You will then use the file search tool to retrieve the business rules applicable to detection of anomalies in the Procure to Pay process.
Step 4: Then, apply the retrieved business rules to match the invoice line items with the contract details fetched from the system, and detect anomalies if any.
Step 5: Prepare a detailed ‘p2p verification report’ in markdown with the following content:
– Verdict: Whether the Purchase Invoice complies with the Contract?
– Purchase Invoice details: Invoice Header and Invoice Lines, in Markdown Table Format
– Contract Details: Contract Header and Contract Lines, in Markdown Table Format
– P2P Business Rules: The Rules that were retrieved using the File Search Tool
– Reasoning behind the verdict: Provide a detailed reasoning why you think the Invoice aligns with the Contract yes or no. Use a Markdown Table format to compare each item in the Invoice with the Contract and indicate the basis for your judgement. Use icons/images to embellish the report and make it easy for comprehension by Business users. They must be able to quickly give this a once over and commit the Invoice into the System
Step 6: You will use the MCP tool available with you to upload the ‘p2p verification report’ created in Step 5.
– Choose a container with name ‘p2p-anomaly-detection-outcomes’ to upload to. If the container with this name does not exist, create one.
– The name of the Report must start with the Invoice Number, appended with a hyphen and appended with a guid. E.g. Invoice001-001.md
– Secure the name of the Report document uploaded to the Blob Storage account through the MCP Tool
Step 7: Wait till Step 6 completes, then return the content of the Markdown document that was uploaded to Blob Storage.

 

Output from the Run

(.venv) PS C:Userssansriagentic-ai-service-samplesp2p-anomaly-detection-agent> python p2pagent.py
2025-07-04 09:04:22,149 – __main__ – INFO – Successfully initialized Azure AI Agents client
2025-07-04 09:04:22,149 – INFO – Successfully initialized Azure AI Agents client

=== P2P Anomaly Detection Agent ===
This tool analyzes invoice images for procurement anomalies.

Enter the path to your invoice image: data_files/Invoice-002.png

Enter your analysis prompt (or press Enter for default):
can you perform the procure to pay anomaly detection based on the instructions you have been provided with and give me a detailed response if this Purchase Invoice Image attached aligns with the Contract?

Processing image: C:Userssansriagentic-ai-service-samplesp2p-anomaly-detection-agentdata_filesInvoice-002.png
Analyzing with Azure AI Agent…
🔄 Creating message content…
2025-07-04 09:04:46,774 – __main__ – INFO – Successfully converted image to base64: C:Userssansriagentic-ai-service-samplesp2p-anomaly-detection-agentdata_filesInvoice-002.png
2025-07-04 09:04:46,774 – INFO – Successfully converted image to base64: C:Userssansriagentic-ai-service-samplesp2p-anomaly-detection-agentdata_filesInvoice-002.png
2025-07-04 09:04:46,777 – __main__ – INFO – Successfully created message content blocks
2025-07-04 09:04:46,777 – INFO – Successfully created message content blocks
🔄 Creating conversation thread…
2025-07-04 09:04:47,231 – __main__ – INFO – Created thread, ID: thread_Lz8ViyvcnDMG9i9TGkgE7jq6
2025-07-04 09:04:47,231 – INFO – Created thread, ID: thread_Lz8ViyvcnDMG9i9TGkgE7jq6
🔄 Sending message to agent…
🔄 Processing with AI agent (this may take a moment)…
🔄 Retrieving analysis results…
🔄 Analysis complete!
2025-07-04 09:05:55,543 – __main__ – INFO – Successfully processed anomaly detection request
2025-07-04 09:05:55,543 – INFO – Successfully processed anomaly detection request

==================================================
ANOMALY DETECTION RESULTS
==================================================
### P2P Verification Report

#### **Verdict:**
The Purchase Invoice **does not comply** with the Contract.

—

#### **Purchase Invoice Details**

| **Attribute** | **Value** |
|———————–|——————————————————————————|
| **Invoice Number** | INV001002 |
| **Contract Reference**| CON000002 |
| **Supplier ID** | SUP0008 |
| **Total Invoice Value** | 113130.16 USD |
| **Invoice Date** | 2023-06-15 |

—

#### **Invoice Line Items**

| **Item ID** | **Quantity** | **Unit Price (USD)** | **Total Price** | **Description** |
|————-|————–|———————–|—————–|—————————————|
| ITEM0040 | 116 | $136.75 | $15863.00 | Description for ITEM0040 |
| ITEM0082 | 116 | $554.62 | $64335.92 | Description for ITEM0082 |
| ITEM0011 | 36 | $398.09 | $14331.24 | Description for ITEM0011 |
| ITEM0031 | 36 | $475.00 | $17100.00 | Description for ITEM0031 |
| ITEM9999 | 10 | $150.00 | $1500.00 | Extra item not in contract |

—

#### **Contract Details**

| **Attribute** | **Value** |
|——————————–|——————————————————————————|
| **Contract ID** | CON000002 |
| **Supplier ID** | SUP0008 |
| **Contract Date** | 2022-10-19 |
| **Expiration Date** | 2023-01-07 |
| **Contract Total Amount** | 66543.39 USD |
| **Status** | Expired |
| **Currency** | USD |

| **Item ID** | **Quantity** | **Unit Price (USD)** | **Total Price** | **Delivery Date** | **Description** |
|————-|————–|———————–|—————–|——————-|—————————-|
| ITEM0040 | 78 | $136.75 | $10666.50 | 2023-01-01 | Description for ITEM0040 |
| ITEM0082 | 57 | $479.87 | $27352.58 | 2022-11-26 | Description for ITEM0082 |
| ITEM0011 | 21 | $398.09 | $8359.89 | 2022-11-29 | Description for ITEM0011 |
| ITEM0031 | 47 | $429.03 | $20164.41 | 2022-12-09 | Description for ITEM0031 |

—

#### **P2P Business Rules**
– Invoice date in Purchase Invoice must be within the Contract term【6:0†source】.
– The Contract must be valid; ensure it is not expired【6:0†source】.
– Invoice Total Value should stay within the Contract value【6:0†source】.
– Items in the Invoice must be strictly from the Contract item list, with correct quantities, descriptions, and unit prices【6:0†source】.
– Minor rounding differences in amounts should be ignored【6:0†source】.

—

|——————|————————————————-|————————————————————————————————————|
| ITEM0040 | Quantity: 78, Unit Price: $136.75 | Invoice quantity exceeds Contract quantity. |
|——————|————————————————-|————————————————————————————————————|
| ITEM0040 | Quantity: 78, Unit Price: $136.75 | Invoice quantity exceeds Contract quantity. |
| ITEM0082 | Quantity: 57, Unit Price: $479.87 | Invoice unit price differs and quantity exceeds Contract quantity. |
| ITEM0011 | Quantity: 21, Unit Price: $398.09 | Invoice quantity exceeds Contract quantity. |
| ITEM0031 | Quantity: 47, Unit Price: $429.03 | Invoice quantity exceeds Contract quantity and Contract total is lower than Invoice total. |
| ITEM9999 | Not listed in Contract | Extra item not defined in Contract—anomalous. |

### Additional Observations:
– **Contract is Expired:** The Contract expired on 2023-01-07. Invoice dated 2023-06-15 falls outside the Contract’s validity term.
– **Invoice exceeds Contract Value:** Contract value capped at $66543.39 USD, while Invoice totals $113130.16 USD.
– **Currency Matches:** Invoice and Contract both indicate prices in USD.

—

### Professional Guidance and Suggested Next Steps:
– The anomalies detected prevent this Invoice from fully complying with the Contract terms.
– **Recommendation:** Rectify discrepancies (quantities, unit price adjustments, or removal of extra items) and adhere to valid Contract before committing Invoice to the System.

—

### Uploading Report Content
Proceeding to upload this Markdown report into Blob Storage.
==================================================

 

MCP Actions

As an outcome from the P2P evaluation, a Markdown document is created by the P2P Agent which then gets uploaded to the Azure Blob Storage, using the MCP Tool action. See the screenshot below from the Azure Portal.

 

References

Here is the GitHub Repo that contains the code for the solution covered in this article.