A study on the selection of resources from XSEDE supercomputers and the open science grid
Description
TitleA study on the selection of resources from XSEDE supercomputers and the open science grid
Date Created2018
Other Date2018-10 (degree)
Extent1 online resource (52 pages) : illustrations
DescriptionThe effective selection of resources on supercomputers and grids improves workload schedul-
ing and reduces workload time-to-completion. Lower workload time-to-completions allow scien-
tists to gain scientific insights from simulations more quickly. For example, molecular dynamics
(MD) simulations are revolutionizing the development of new therapeutics. In the past 6 years,
at least 4 billion core-hours consumed on XSEDE machines were from MD software packages
alone. Given the increasing number of MD simulations executed and the increasing amounts of
computation they require, there is a need for greater efficiency in resource utilization. Thus, we
investigate how resources from XSEDE supercomputers and OSG can be effectively selected to
reduce workload time-to-completion.
Effective resource selection on grids and supercomputers is difficult. Grids have heterogeneous
and transient pools of resources, whose availability and performance vary over time. This makes
it difficult to collect information, such as benchmarking results, application profiles and hardware
capabilities, used by techniques like application performance modeling and benchmarking to best
select grid resources. While the performance of supercomputing resources are easier to assess
for grid resources, the acquisition of these resources requires waiting on a queue, sometimes for
long periods of time. However, accurately predicting queue waiting time remains difficult.
In this thesis, we studied how to effectively select resources from XSEDE supercomputers and
OSG in the presence of limited information. We developed a formalism that allows us to model
the cost of task execution based on the information available from XSEDE supercomputers and
XSEDE OSG. On the base of our formalism, we constructed the Limited Information Model
(LIM) to predict the execution times of compute-intensive, single-threaded, single-process tasks.
We evaluated the accuracy of these predictions and the resources selected using these predictions
iito gain insight into what information, if any, would be needed to operate better predictions and
resource selections. To overcome the difficulty of selecting resources using queue waiting times,
we also developed the resource re-selection process, by which tasks are re-assigned to different,
acquired resources at runtime. Resource re-selection uses task execution times and resource
acquisition times, but not queue waiting times. We show that tasks can be effectively re-assigned
even when using inaccurate execution time predictions.
Experimental validation of LIM shows that LIM’s predictions are within 157–171% error
on XSEDE supercomputers and 18–31% on OSG. By accounting for the differences in software
configurations on XSEDE supercomputers and OSG, LIM’s predictions can still be used correctly
rank XSEDE supercomputers and OSG. Experiments also show that workloads executed using
resources selected with LIM’s predictions have 67–78% lower workload time-to-completion than
those executed using randomly selected resources. However, executing workloads on resources
selected using LIM’s predictions contributed to ∼29–99% of the reduction in workload time-
to-completion. We found that the queue waiting times largely influences workload time-to-
completion and should also be considered to effectively select resources from supercomputers
and grids.
Finally, experiments show that performing resource re-selection allows tasks to avoid expe-
riencing large queue waiting times on XSEDE supercomputers by executing on OSG instead.
Despite using inaccurate task execution time predictions and no queue waiting time predictions,
experiments show that resource re-selection reduces task queue waiting times by up to 99%
and workload time-to-completion by up to 73% when the queue waiting time experienced on
XSEDE supercomputers are high. However, resource re-selection shows little to no reductions
in task queue waiting times and workload time-to-completions when the queue waiting times
experienced on XSEDE supercomputers are small.
NoteM.S.
NoteIncludes bibliographical references
Noteby Ming Tai Ha
Genretheses, ETD graduate
Languageeng
CollectionSchool of Graduate Studies Electronic Theses and Dissertations
Organization NameRutgers, The State University of New Jersey
RightsThe author owns the copyright to this work.
Statistical Profile