AzureFeeds

Overview of container network logs

Container network logs offer a comprehensive way to monitor network traffic in AKS clusters. Two modes of support, stored-logs and on-demand logs, provides debugging flexibility with cost optimization. The on-demand mode provides a snapshot of logs with queries and visualization with Hubble CLI UI for specific scenarios and does not use log storage to persist the logs. The stored-logs mode when enabled continuously collects and persists logs based on user-defined filters. Logs can be stored either in Azure Log Analytics (managed) or locally (unmanaged).

• Managed storage: Logs are forwarded to Azure Log Analytics for secure, scalable, and compliant storage. This enables advanced analytics, anomaly detection, and historical trend analysis. Both basic and analytics table plans are supported for storage.


• Unmanaged storage: Logs are stored locally on the host nodes under /var/log/acns/hubble. These logs are rotated automatically at 50 MB to manage storage efficiently. These logs can be exported to external logging systems or collectors for further analysis.

Use cases

1. Connectivity monitoring: Identify and visualize how Kubernetes workloads communicate within the cluster and with external endpoints, helping to resolve application connectivity issues efficiently.


2. Troubleshooting network errors: Gain deep granular visibility into dropped packets, misconfigurations, or errors with details on where and why errors are occurring (TCP/UDP, DNS, HTTP) for faster root cause analysis.


3. Security policy enforcement: Detect and analyze suspicious traffic patterns to strengthen cluster security and ensure regulatory compliance.

How it works

Container network logs use eBPF technology with Cilium to capture network flows from AKS nodes. Log collection is disabled by default. Users can enable log collection by defining custom resources (CRs) to specify the types of traffic to monitor, such as namespaces, pods, services, or protocols. The Cilium agent collects and processes this traffic, storing logs in JSON format. These logs can either be retained locally or integrated with Azure Monitoring for long-term storage and advanced analytics and visualization with Azure managed Grafana.

Fig1: Container network logs overview

If using managed storage, users will enable Azure monitor log collection using Azure CLI or ARM templates. Here’s a quick example of enabling container network logs on Azure monitor using the CLI:

az aks enable-addons -a monitoring –enable-high-log-scale-mode -g $RESOURCE_GROUP -n $CLUSTER_NAME

az aks update –enable-acns
    –enable-retina-flow-logs
    -g $RESOURCE_GROUP
    -n $CLUSTER_NAME

 Key benefits

• Faster issue resolution: Detailed logs enable quick identification of connectivity and performance issues.


• Operational efficiency: Advanced filtering reduces data management overhead.


• Enhanced application reliability: Proactive monitoring ensures smoother operations.


• Cost optimization: Customized logging scopes minimize storage and data ingestion costs.


• Streamlined compliance: Comprehensive logs support audits and security requirements.

Observing logs in Azure managed Grafana dashboards

Users can visualize container network logs in Azure managed Grafana dashboards, which simplify monitoring and analysis:

1. Flow logs dashboard: View internal communication between Kubernetes workloads. This dashboard highlights metrics such as total requests, dropped packets, and error rates.


2. Error logs dashboard: Easily zoom in only on the logs which show errors for faster log parsing.


3. Service dependency graph: Visualize relationships between services, detect bottlenecks, and optimize network flows.

These dashboards provide filtering options to isolate specific logs, such as DNS errors or traffic patterns, enabling efficient root cause analysis. Summary statistics and top-level metrics further enhance understanding of cluster health and activity.

Fig 2: Azure managed Grafana dashboard for container network logs

Conclusion

Container network logs for AKS offer a powerful and cost optimized way to monitor and analyze network activity, enhance troubleshooting, security, and ensure compliance. To get started, enable Advanced Container Networking Services in your AKS cluster and configure custom resources for logging. Visualize your logs in Grafana dashboards and Azure Log Analytics to unlock actionable insights. Learn more here.