File(s) under permanent embargo
Resource scheduling and provisioning for processing of dynamic stream workflows under latency constraints
journal contributionposted on 2023-05-21, 14:39 authored by Alexander Brown, Saurabh GargSaurabh Garg, Erin MontgomeryErin Montgomery, Ujjwal K C
There has been an increasing demand for large-scale processing of stream workflows. Many streaming data sources, such as video and audio, differ in terms of the type and density of information they contain over time, which require dynamic workflow structures where paths change in response to data variations. Additionally, information is often only inferred from examining discrete chunks and hence, cannot be arbitrarily parallelised like other streaming data sources. Stream processes using these data sources often operate as dynamic workflows, where different paths are selected based on prior processing information, relating to differences in the nature of the data itself. To better enable low latency analysis in such scenarios, this work investigates strategies to schedule and provision streaming dynamic workflow scenarios of various complexities and degrees of unpredictability. It is found that, as scenarios become increasingly uncertain and unpredictable, it is better to use simpler strategies which do not make assumptions about workflow structures or completion times.
Publication titleFuture Generation Computer Systems
Department/SchoolSchool of Information and Communication Technology
PublisherElsevier Science Bv
Place of publicationPo Box 211, Amsterdam, Netherlands, 1000 Ae
Rights statement© 2022 Elsevier B.V. All rights reserved.