When HPC meets cloud computing
The rise of global population along with the increased demand for high-quality food push efficient resource usage for agriculture industry. Technologies, e.g. IoT, are adopted to meet this challenge. Massive data processing raises the convergence of HPC, big data and cloud computing.
On cloud, containerization technology is widely-used as it enables massive and efficient application deployment, e.g. Google runs Gmail service, Youtube, Search etc., in containers. Containerization is basically an OS virtualization approach. A container incorporates files, libraries, environment variables, etc. to run its desired software without an entire OS image. This makes it lightweight, in contrast with VM which runs with the guest OS. Docker and singularity are the most-adopted containers. Nevertheless, containerized applications can become complicated, e.g. thousands of separate containers may be required in production. The production can benefit from container orchestrators that manage and scale the container-based workloads easily. An example of the container orchestrator is Kubernetes.
HPC systems are traditionally adopted to perform financial and engineering simulation, which demands low-latency and high-throughput. HPC has a legacy software stack steered towards bare metal deployment (C/C++, batch jobs). The emerging areas, e.g. machine learning, are not yet readily supported due to their complementary software and runtime requirements, i.e. different programming languages, libraries, frameworks and workflows. Containerization could be the potential solution to allow such complex software stacks on HPC. However, packaging workloads into containers are foreign to HPC users. Traditional HPC workloads are managed by the work load managers, such as Torque, SLURM. Therefore, it is interesting to develop the plugins/applications which can link HPC workload managers and container orchestrators. This is under development in CYBELE project. More information of bridging HPC and container orchestrators can be found here: https://buff.ly/2kQLTSh