RAPS: Recovery Aware Parallel computing Systems

With rapid advances in processing technology, along with the emerging multi-core processors and specialized co-processors, parallel processing is permeating almost every aspect of our lives, from high-end computing to commodity deployment. Production systems in the next few years are expected to contain hundreds of thousands of processors, with each processor containing dozens of cores. Fueled by the ever-growing scale and complexity of parallel systems, these systems often fail in unpredictable ways. Studies have shown that in production systems, failure rates range from 20 to more than 1000 per year, and depending on root cause of the problem, the system-level mean time to repair (MTTR) ranges from a couple of hours for failures caused by human errors to nearly 100 hours for failures due to hardware problems. Years of intense research have focused on pre-failure prediction and tolerance by predicting failures and taking precaution actions before failure occurrence. Nevertheless, despite research progress on failure prediction, unexpected failures frequently occur in practice, especially in modern parallel systems with unprecedented sizes and complexities. Hence, relying on pre-failure prediction and tolerance alone is insufficient for fault tolerance due to the inevitability of failures. Just as failures need to be carefully avoided and tolerated, post-failure diagnosis and recovery (i.e., a procedure taken after failures) is of equal importance and has a profound impact on almost every aspect of parallel computing.

The goal of this research project is to develop RAPS, a Recovery Aware Parallel computing System for post-failure diagnosis and recovery. The research focuses on how to quickly and effectively resume parallel computing after a failure has occurred. The ultimate goal is to seamlessly integrate post-failure diagnosis and recovery with pre-failure prediction and tolerance as a compound fault management solution for parallel computing. The approach consists of (1) development of new diagnosis mechanisms for fast failure detection and root cause analysis, (2) development of system-wide orchestration for recovery coordination, (3) design of new recovery techniques for quick restoration of parallel applications, and (4) a comprehensive evaluation. The results of this project can significantly improve the productivity of parallel systems