AzureFeeds

Drasi, an open-source Data Change Processing Platform, simplifies the creation of change-driven systems because it provides a consistent way of thinking about, detecting, and reacting to change. Sometimes, you need to detect and react when data doesn’t change. Drasi provides an approach to detecting the absence of change and makes building such systems easy.  

When there is no change  

In the world of change-driven systems, certain scenarios challenge conventional response mechanisms. Among these challenges is the subtle yet complex problem of responding to the absence of change rather than the arrival of an individual event. This nuanced requirement often arises in monitoring systems, IoT devices, and other applications where a condition must persist for a given duration to warrant a reaction.  

Consider an example: a freezer’s temperature sensor emits an event when the temperature changes, and at one point, the temperature registers above 32°F. While this measurement is significant, the system should only react if the freezer’s temperature remains above 32°F for at least 15 minutes. There is, however, no explicit event that confirms this persistence. The difficulty lies in establishing a reliable mechanism to track and respond to sustained states without direct event notification of their continuity.  We’ll describe Polling and Timers, which are traditional solutions, and then describe how Drasi solves this problem. 

Traditional solutions

Polling 

To solve this, polling often serves as a standard approach. In this method, the system would periodically scan the last 15 minutes of data to determine if the temperature was above the threshold continuously for 15 minutes. This approach is inherently limited by its non-real-time nature, as the system only identifies qualifying conditions during scheduled intervals. Consequently, there may be delays in detecting and responding to critical conditions, especially in scenarios where timely action is paramount. Furthermore, polling can lead to increased computational overhead, especially in large-scale systems, as it requires frequent queries to ensure no conditions are missed. 

Timers 

An alternative to polling involves leveraging the initial event that triggers a state change to start a timer. In this approach, the system initiates a countdown the moment a condition arises, such as the temperature rising above 32°F. If the condition persists for the defined threshold (15 minutes for the freezer), the system initiates the required response. Conversely, if the condition is resolved before the timer expires, the timer is canceled. 

While this approach addresses some limitations of polling by introducing real-time responsiveness, it introduces its own complexities and overhead. Managing timers at scale is not trivial, particularly in distributed systems with thousands of tracked conditions. Each timer must be initiated, monitored, and terminated. 

To implement initiation, monitoring, and termination effectively, a specialized timer management service must be built or adopted. This service needs to handle the management of timers, ensure high reliability, and scale to volumes. Ensuring failover and recovery mechanisms for timers, particularly in distributed systems, introduces further complexity. For example, if a node managing active timers fails, the system must ensure that no timer is lost or incorrectly reset, which often requires sophisticated state replication and recovery strategies. 

Ultimately, this timer-based approach necessitates the deployment and management of custom-built services. These services bring inherent costs not only in terms of development and maintenance but also in operational overhead. As such, while this method can deliver superior responsiveness compared to polling, its implementation comes with a steep tradeoff in system complexity and costs. 

Drasi to detect the absence of change

Central to Drasi is the Continuous Query Pattern, implemented using the openCypher graph query language. A Continuous Query runs perpetually, fed by change logs from one or more data sources, maintaining the current query result set and generating notifications when those results change. Unlike producer-defined event streams, this pattern empowers consumers to specify the relevant properties and their relationships using a familiar database-type query. Drasi solves the “absence of change” problem through a suite of “future” functions, within a Continuous Query.

Verifying Sustained Conditions with Drasi: A Freezer Monitoring Example

The freezer example can be expressed as a simple openCypher query, using the “trueFor” function unique to Drasi. The “trueFor” function takes an expression that must evaluate to “true” for the duration specified, if this expression holds true for the entire length of the duration specified, the WHERE clause will resolve to true and only then will a notification be emitted that a new item has been added to the result set. 

MATCH  
  (f:Freezer) 
WHERE drasi.trueFor(f.temp > 32, duration( { minutes: 15 } )) 
RETURN 
  f.id AS id, 
  f.temp AS temp 

Under the hood 

To achieve this, internally Drasi leverages a specialized priority queue with unique access patterns that is ordered by future timestamps.  

When the WHERE clause is first evaluated, some metadata about the associated graph elements is pushed into the priority queue, this metadata can later be used to surgically re-evaluate a given condition using cached indexes. The position in the queue will be determined by the future timestamp at which the condition can be re-evaluated. 

The “trueFor” function takes a condition and a duration of how long the condition needs to be true.  The function will only return ‘true’ when the condition has held true continuously for the specified duration. 

Let’s consider the freezer example with the following temperature changes: 

At 12:00 – The freezer temp is 35  

At 12:01 – The freezer temp is 36  

At 12:02 – The freezer temp is 30  

At 12:14 – The freezer temp is 34  

Given the value of 30 at 12:02 and the value of 34 at 12:14, the alert should not fire until 12:29. To achieve this, the time at which the freezer crosses 32 degrees needs to be tracked so that it can be determined if the condition has been true for at least 15 minutes. 

When the query engine first evaluates this function, it will test the “temp > 32” expression passed to it. If the condition resolves true, then the element metadata is added to the queue, only if it is not already on the queue. 

If the condition resolves false, and if that metadata is already on the queue, it is removed from the queue, because continuity has been broken. 

If that metadata reaches the head of the queue and its timestamp elapses, the element is reprocessed through the query, and the function returns a “true” result which triggers a reaction.  

The priority queue would look as follows for each change (where “f1” represents the metadata for “Freezer 1”):

Future-Time Evaluation with Drasi: A Payment Authorization Example

The continuity feature of the “trueFor” function may not be desired in every use case. Take another example: an online payment system, where a payment is first authorized and the customer funds are put on hold to secure an order. If the order is not completed within fifteen minutes, then the funds must be released, and the reserved inventory must be made available again.  

This example can also be expressed as a simple openCypher query, using the “trueLater” function. This function takes an expression that must evaluate to “true” at a given future time. If it evaluates to “true” at the given future time, the WHERE clause will resolve to true and only then will a notification be emitted that a new item has been added to the result set. 

MATCH  
  (p:Payment) 
WHERE drasi.trueLater(p.status = ‘auth’, p.exipres_at) 
RETURN 
  p.id, 
  p.amount, 
  p.customer 

Under the hood

When the WHERE clause is first evaluated, if the timestamp provided to the function is in the future, the function will push the element metadata to the priority queue and return an “AWAITING” result, which is the equivalent of false, and in the payment example the WHERE clause filters out this potential result. 

If the provided timestamp is in the past, the function will return the result of evaluating the condition. 

Try out the “Absence of Change” tutorial to see these functions in action.

Conclusion 

Detecting the absence of change in change-driven systems is a subtle yet critical challenge, often complicated by the inefficiencies of traditional approaches like polling or the complexities of managing timers at scale. Drasi revolutionizes this process with the Continuous Query Pattern and powerful functions like “trueFor” and “trueLater”, enabling developers to build responsive, scalable systems with ease. By leveraging familiar openCypher queries, Drasi eliminates the need for cumbersome custom services, delivering real-time reactions with minimal overhead. Drasi offers a streamlined, elegant solution. Ready to simplify your change-driven systems? Explore Drasi today, experiment with its Continuous Queries, and join the conversation to share your insights! 

Further reading: Reference | Drasi Docs

Join the Drasi community

If you’re a developer interested in solving real-world problems, exploring modern architectures, or just looking to contribute to something meaningful, we’d love to have you onboard. You can check out the code on our GitHub organization, dig into the technical details on our documentation site, and join our developer community on Discord.