DCS: A Policy Framework for the Detection of Correlated Data Streams

Rakan Alseghayer, Daniel Petrov, Panos K. Chrysanthis, Mohamed Sharaf, Alexandros Labrinidis

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

There is an increasing demand for real-time analysis of large volumes of data streams that are produced at high velocity. The most recent data needs to be processed within a specified delay target in order for the analysis to lead to actionable result. To this end, in this paper, we present an effective solution for detecting the correlation of such data streams within a micro-batch of a fixed time interval. Our solution, coined DCS, for Detection of Correlated Data Streams, combines (1) incremental sliding-window computation of aggregates, to avoid unnecessary re-computations, (2) intelligent scheduling of computation steps and operations, driven by a utility function within a micro-batch, and (3) an exploration policy that tunes the utility function. Specifically, we propose nine policies that explore correlated pairs of live data streams across consecutive micro-batches. Our experimental evaluation on a real world dataset shows that some policies are more suitable to identifying high numbers of correlated pairs of live data streams, already known from previous micro-batches, while others are more suitable to identifying previously unseen pairs of live data streams across consecutive micro-batches.

Original languageEnglish
Title of host publicationReal-Time Business Intelligence and Analytics - International Workshops, BIRTE 2015, BIRTE 2016, BIRTE 2017, Revised Selected Papers
EditorsMalu Castellanos, Panos K. Chrysanthis, Konstantinos Pelechrinis
PublisherSpringer
Pages191-210
Number of pages20
ISBN (Print)9783030241230
DOIs
Publication statusPublished - 2019
Event9th International Workshop on Business Intelligence for the Real-Time Enterprise, BIRTE 2015, 10th International Workshop on Enabling Real-Time Business Intelligence, BIRTE 2016 and 11th International Workshop on Real-Time Business Intelligence and Analytics, BIRTE 2017 held in conjunction with the International Conference on Very Large Data Bases, VLDB 2017 - Munich, Germany
Duration: Aug 28 2017Sep 1 2017

Publication series

NameLecture Notes in Business Information Processing
Volume337
ISSN (Print)1865-1348
ISSN (Electronic)1865-1356

Conference

Conference9th International Workshop on Business Intelligence for the Real-Time Enterprise, BIRTE 2015, 10th International Workshop on Enabling Real-Time Business Intelligence, BIRTE 2016 and 11th International Workshop on Real-Time Business Intelligence and Analytics, BIRTE 2017 held in conjunction with the International Conference on Very Large Data Bases, VLDB 2017
Country/TerritoryGermany
CityMunich
Period8/28/179/1/17

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Management Information Systems
  • Business and International Management
  • Information Systems
  • Modelling and Simulation
  • Information Systems and Management

Fingerprint

Dive into the research topics of 'DCS: A Policy Framework for the Detection of Correlated Data Streams'. Together they form a unique fingerprint.

Cite this