Home
Resource
Autonomic management of data streaming and in-transit processing for data intensive scientific workflows
PDF
PDF format is widely accepted and good for printing.
Plug-in required
PDF-1(2.29 MB)
Citation & Export
View Usage Statistics
Staff View

Citation & Export
Hide

Simple citation

Bhat, Viraj N.. Autonomic management of data streaming and in-transit processing for data intensive scientific workflows. Retrieved from https://doi.org/doi:10.7282/T3XS5VRH

Export

Click here for information about Citation Management Tools at Rutgers.

Statistics
Hide

Description

Uniform TitleAutonomic management of data streaming and in-transit processing for data intensive scientific workflows
NameBhat, Viraj N. (author); Parashar, Manish (chair); Marsic, Ivan (internal member); Pompili, Dario (outside member); Marty, Christopher (outside member); Klasky, Scott (outside member); Rutgers University; Graduate School - New Brunswick
Date Created2008
Other Date2008-05 (degree)
SubjectElectrical and Computer Engineering, Streaming technology (Telecommunications), Data flow computing
Extentxvii, 145 pages
DescriptionHigh-performance computing is playing an important role in science and engineering and is enabling highly accurate simulations, which provide insights into complex physical phenomena. A key challenge is managing the enormous data volumes and high data rates associated with these applications, so as to have minimal impact on the execution of the simulations. Furthermore these applications are based on seamless interactions and coupling between multiple and potentially distributed computational, data and information services. This requires addressing the natural mismatches in the ways data is represented in different workflow components and on a variety of machines, and being able to "outsource" the required data manipulation and transformation operations to less expensive commodity resources "in-transit". Satisfying these requirements is challenging, especially in large-scale and highly dynamic in-transit environments with shared computing and communication resources, resource heterogeneity in terms of capability, capacity, and costs, and where application behaviors, needs, and performance are highly variable.
In this research we address these requirements by developing a data streaming and in-transit data manipulation framework that provides mechanisms as well as the management strategies for large scale and wide-area data intensive scientific and engineering workflows. The main objectives of this research are: (1) developing an end-to-end QoS management framework for data intensive applications so that it is able to provide robust underlying support for asynchronous, high-throughput, low-latency data streaming, and (2) effectively and opportunistically utilize resources in-transit for data processing, to match data mismatches between application entities executing in scientific workflows.
In this thesis, we address problem at two levels, the first or application level deals with satisfying QoS goals at the end points. Specifically, it ensures that the data is delivered in a timely manner, with no loss at the source or destination, and with minimal storage requirements at the end-points. The solution couples model-based limited look-ahead controllers (LLC) with rule-based managers to satisfy data streaming requirements under various operating conditions. The second or in-transit level focuses on scheduling in-transit computations and data transfer in an opportunistic manner on the in-transit overlay resources taking into account the higher level QoS goals of the source and the sink. Additionally the in-transit level management is coupled with the application level management at end points to manage QoS of grid workflows.
This research is driven by the requirements of the Fusion Simulation Project (FSP), which forms the basis of a predictive plasma edge simulation capability to support next-generation burning plasma experiments such as the International Thermonuclear Experimental Reactor (ITER). These scientific workflows require in-transit data manipulation and streaming in a wide area environment. The self-managing data streaming service developed using this approach for the FSP workflow minimizes streaming overheads on the executing simulation to about 2% of the simulation execution time, reduces buffer occupancy at the source and thus prevents data loss. Additionally experiments with self-managing data streaming and in-transit processing demonstrates that adaptive processing using this service during network congestions decreases average idle time per data block from 25% to 1%, thereby increasing utilization at critical times. Furthermore, coupling end-point and in-transit level management during congestion reduces average buffer occupancy at in-transit nodes from 80% to 60.8%, thereby reducing load and potential data loss.
NotePh.D.
NoteIncludes bibliographical references (p. 136-143).
Genretheses, ETD doctoral
Persistent URLhttps://doi.org/doi:10.7282/T3XS5VRH
LanguageEnglish
CollectionGraduate School - New Brunswick Electronic Theses and Dissertations
Organization NameRutgers, The State University of New Jersey
RightsThe author owns the copyright to this work.
Version 8.5.5
Rutgers University Libraries - Copyright ©2024