AzureFeeds

 

 

 

 

Today, we announced a significant expansion of Microsoft Sentinel’s capabilities through the introduction of Sentinel data lake, now rolling out in public preview. Security teams cannot defend what they cannot see and analyze. With exploding volumes of security data, organizations are struggling to manage costs while maintaining effective threat coverage. Do-it-yourself security data architectures have perpetuated data silos, which in turn have reduced the effectiveness of AI solutions in security operations. With Sentinel data lake, we are taking a major step to address these challenges.

Microsoft Sentinel data lake enables a fully managed, cloud-native, data lake that is purposefully designed for security, right inside Sentinel. Built on a modern lake architecture and powered by Azure, Sentinel data lake simplifies security data management, eliminates security data silos, and enables cost-effective long-term security data retention with the ability to run multiple forms of analytics on a single copy of that data. Security teams can now store and manage all security data. This takes the market-leading capabilities of Sentinel SIEM and supercharges it even further. Customers can leverage the data lake for retroactive TI matching and hunting over a longer time horizon, track low and slow attacks, conduct forensics analysis, build anomaly insights, and meet reporting & compliance needs. By unifying security data, Sentinel data lake provides the AI ready data foundation for AI solutions. Let’s look at some of Sentinel data lake’s core features.

Simplified onboarding and enablement inside Defender Portal: Customers can easily discover and enable the new data lake from within the Defender portal, either from the banner on the home page or from settings. Setting up a modern data lake now is just a click away, empowering security teams to get started quickly without a complex setup.

Figure 1: Simplified Onboarding to Sentinel data lake

Simplified security data management: Sentinel data lake works seamlessly with existing Sentinel connectors. It brings together security logs from Microsoft services across M365, Defender, Azure, Entra, Purview, Intune plus third-party sources like AWS, GCP, network and firewall data from 350+ connectors and solutions. The data lake supports Sentinel’s existing table schemas while customers can also create custom connectors to bring raw data into the data lake or transform it during ingestion. In the future, we will enable additional industry-standard schemas. The data lake expands beyond just activity logs by including a native asset store. Critical asset information is added to the data lake using new Sentinel data connectors for Microsoft 365, Entra, and Azure, enabling a single place to analyze activity and asset data enriched with Threat intelligence.

A new table management experience makes it easy for customers to choose where to send and store data, as well as set related retention policies to optimize their security data estate. Customers can easily send critical, high-fidelity security data to the analytics tier or choose to send high-volume, low fidelity logs to the new data lake tier. Any data brought into the analytics tier is automatically mirrored into the data lake at no additional charge, making data lake the central location for all security data.

Figure 2: Flexible Data Management in Sentinel data lake

Advanced data analysis capabilities over data in the data lake: Sentinel data lake stores all security data in an open format to enable analysts to do multi-modal security analytics on a single copy of data. Through the new data lake exploration experience in the Defender portal, customers can leverage Kusto query language to analyze historical data using the full power of Kusto. Since the data lake supports the Sentinel table schema, advanced hunting queries can be run directly on the data lake. Customers can also schedule long-running jobs, either once or on a schedule, that perform complex analysis on historical data for in-depth security insights. These insights generated from the data lake can be easily elevated to analytics tier and leveraged in Sentinel for threat investigation and response.

Figure 3: Advanced Data Analytics

Additionally, as part of the public preview, we are also releasing a new Sentinel Visual Studio Code extension that enables security teams to easily connect to the same data lake data and use Python notebooks, as well as spark and ML libraries to deeply analyze lake data for anomalies. Since the environment is fully managed, there is no compute infrastructure to set up. Customers can just install the Visual Studio Code extension and use AI coding agents like GitHub Copilot to build a notebook and execute it in the managed environment. These notebooks can also be scheduled as jobs and the resulting insights can be elevated to analytics tier and leveraged in Sentinel for threat investigation and response.

Figure 4: Python Notebooks in new Sentinel Visual Studio Code Extension

Flexible business model: Sentinel data lake enables customers to separate their data ingestion and retention needs from their security analytics needs, allowing them to ingest and store data cost effectively and then pay separately when analyzing data for their specific needs.  

Let’s put this all together and show an example of how a customer can operationalize and derive value from the data lake for retrospective threat intelligence matching in Microsoft Sentinel. Network logs are typically high-volume logs but can often contain key insights for detecting initial entry point of an attack, command and control connection, lateral movement or an exfiltration attempt. Customers can now send these high-volume logs to the data lake tier. Next, they can create a python notebook that can join latest threat intelligence from Microsoft Defender Threat Intelligence to scan network logs for any connections to/from a suspicious IP or domain. They can schedule this notebook to run as a scheduled job, and any insights can then be promoted to analytics tiers and leveraged to enrich ongoing investigation, hunts, response or forensics analysis. All this is possible cost-effectively without having to set up any complex infrastructure, enabling security teams to achieve deeper insights.

This preview is now rolling out for customers in Defender portal in our supported regions. To learn more, check out our Mechanics video and our documentation or talk to your account teams. Also learn more about updates to Azure Monitor that provides one of the underlying services for the Sentinel data lake – more information coming soon at http://aka.ms/AuxiliaryLogsEnhancements.

Get started today 

Join us as we redefine what’s possible in security operations: 

• Onboard Sentinel data lake: https://aka.ms/sentineldatalakedocs


• Explore our pricing: https://aka.ms/sentinel/pricingblog


• For the supported regions, please refer to https://aka.ms/sentinel/datalake/geos


• Learn more about our MDTI news: http://aka.ms/mdti-convergence