The Interface Launcher (iLauncher) technology automates the submission of HPC jobs and provides a mechanism for rapidly prototyping web interfaces from the user’s desktop to powerful capabilities running on back-end high performance computing (HPC) resources, including Amazon Web Services (AWS) GovCloud, distributed clusters of heterogeneous nodes with multiple graphics processing units (GPUs) per node running the Slurm batch queuing software, and Department of Defense (DoD) supercomputers running the Portable Batch Scheduling (PBS) software. We present some of the latest advancements in iLauncher plugin development, particularly in the use of channels to make deployment of plugins easier for groups of users. We also describe our latest plugin for using PostgreSQL with the TimescaleDB and PostGIS extensions in a Singularity container with the pgAdmin and Jupyter Notebook web interfaces for use on these HPC resources.
We present some of the latest advancements in the development of the Interface Launcher (iLauncher), along with the application of this technology to the development of plugins that support distributed PyTorch deep learning workflows across a diversity of computing resources including Amazon Web Services (AWS) GovCloud, distributed clusters of heterogeneous nodes with multiple graphics processing units (GPUs) per node running the Slurm batch queuing software, and Department of Defense (DoD) high performance computing (HPC) supercomputers running the Portable Batch Scheduling (PBS) software. The iLauncher technology automates the submission of HPC jobs and provides a mechanism for rapidly prototyping web interfaces from the user’s desktop to powerful capabilities running on the HPC nodes. We describe the extension of previous work to show the development of the client-side plugin JavaScript Object Notation (JSON) description, the underlying server-side scripts for running distributed PyTorch deep learning models on various platforms with different queuing systems, and the recipes for the software along with all dependencies in an all-inclusive software packaging technology called a container. Finally, we show a representative use case running distributed PyTorch in a Jupyter Notebook through iLauncher on the various backend platforms along with some guidance on when each one may be beneficial for a range of scenarios based on models and data.
This work builds on another effort described in Application of Jupyter Notebook interfaces and iLauncher to deep learning work ows on HPC systems.22 We describe a complex work ow application which generates millions of images in parallel on an HPC system via web interfaces using ipywidgets in Jupyter Notebooks and the Interface Launcher (iLauncher). Some computations are so complicated, taking many millions of HPC hours, that only a few subject matter experts are able to generate information efficiently. We present our custom application that walks the user through a work flow to include: target selection, configuration of the target, radar phase history simulation, and finally SAR image generation. The interface requests the user to enter a minimal set of parameters while other variables essential to computations are generated on the y and provides status updates on work ow computations. Additionally, the ability to download any data component or view images interactively is provided. This application can be disconnected from the HPC system and reconnected at any time without slowing down the computations on the work ow submitted. Although typically a maximum run time must be specified when submitting a job to the queuing interface on an HPC system, this application uses the HPC-GPS tool to allow users to extend run times even after the initial request is submitted. Our new application helps to reduce the barrier to entry for both complicated physics-based simulations and using HPC systems.
We present information on what users need in order to support complex deep learning workflows on DoD HPC systems with web interfaces using ipywidgets in Jupyter Notebooks and the Interface Launcher (iLauncher), a tool that automates the submission of HPC jobs and provides a mechanism for rapidly prototyping web interfaces from the user’s desktop to powerful capabilities running on the HPC nodes. We detail a representative use case for a PyTorch deep learning workflow and show how to include the underlying software along with all dependencies in an all-inclusive software packaging technology called a Singularity container. We then show how to use ipywidgets in a Jupyter Notebook and convert it to a full-fledged web interface using the Voila server. Finally, we outline how to create the iLauncher plugin to run the web interface on DoD HPC system nodes to provide a complete user interface workflow solution that does not require special privileges to create, deploy, or use. Using Jupyter Notebooks, ipywidgets, iLauncher, and Singularity containers together provide an explosion of accessible capabilities that were previously inconceivable in the restrictive DoD environment.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.