A Generic Approach to Scheduling and Checkpointing Workflows

TitleA Generic Approach to Scheduling and Checkpointing Workflows
Publication TypeConference Paper
Year of Publication2018
AuthorsHan, L., V. Le Fèvre, L-C. Canon, Y. Robert, and F. Vivien
Conference Name The 47th International Conference on Parallel Processing (ICPP 2018)
Date Published2018-08
PublisherIEEE Computer Society Press
Conference LocationEugene, OR

This work deals with scheduling and checkpointing strategies to execute scientific workflows on failure-prone large-scale platforms. To the best of our knowledge, this work is the first to target failstop errors for arbitrary workflows. Most previous work addresses soft errors, which corrupt the task being executed by a processor but do not cause the entire memory of that processor to be lost, contrarily to fail-stop errors. We revisit classical mapping heuristics such as HEFT and MinMin and complement them with several checkpointing strategies. The objective is to derive an efficient trade-off between checkpointing every task (CkptAll), which is an overkill when failures are rare events, and checkpointing no task (CkptNone), which induces dramatic re-execution overhead even when only a few failures strike during execution. Contrarily to previous work, our approach applies to arbitrary workflows, not just special classes of dependence graphs such as M-SPGs (Minimal Series-Parallel Graphs). Extensive experiments report significant gain over both CkptAll and CkptNone, for a wide variety of workflows.

External Publication Flag: