Speaker
Description
Authors
Seung-Jin Sul*, Mario Melara, Ramani Kothadia, Ludovico Bianchi, Joshua Boverhof, Nick Tyler, Daniela Cassol, Mike Sneddon, Setareh Sarrafan, Kjiersten Fagnan
Lawrence Berkeley National Laboratory, Berkeley, CA, USA
Abstract
Scientific workflows at the DOE Joint Genome Institute (JGI) increasingly need to run across multiple high-performance computing facilities, but each site exposes a different mix of schedulers, queue policies, container runtimes, data-transfer paths, and security constraints. The JGI Analysis Workflow Service (JAWS) addresses this problem by separating the scientific workflow interface from the facility-specific execution backend. Users submit Workflow Description Language (WDL) workflows through a common client and JAWS Central service, while each JAWS Site provides the local backend configuration needed to execute those workflows on a particular compute environment.
This talk focuses on an introduction to JAWS and the backend architecture that makes that portability practical, with emphasis on HTCondor and the JAWS Pool Manager. In JAWS, Cromwell orchestrates WDL workflows, and HTCondor serves as the execution backend that receives task submissions, maintains the job queue, and dispatches work to compute pools at the target site. The JAWS Pool Manager implements the HTCondor glidein: it monitors HTCondor job queue, determines whether the pool has capacity, and provisions worker nodes from the site's native batch system (Slurm or PBS) to scale the HTCondor pool up and down with demand. The JAWS and HTCondor logs and metadata support performance reporting for CPU use, memory efficiency, runtimes, and failure modes. This design enables JAWS run task-by-task execution over heterogeneous facilities while presenting a consistent operational model to users and workflow developers.
Keywords
JAWS, HTCondor, workflow management, WDL, Cromwell, pool manager, HPC, DOE computing
Acknowledgement
The work conducted by the U.S. Department of Energy Joint Genome Institute, a DOE Office of Science User Facility, is supported by the Office of Science of the U.S. Department of Energy operated under Contract No. DE-AC02-05CH11231.