Jul 10 – 14, 2023
Fluno Center on the University of Wisconsin-Madison Campus
America/Chicago timezone

Questions about attending, speaking, accommodations, and other concerns

HTCondor and Dask Gateway for elastic CMS Data analysis at Fermilab

Not scheduled
20m
Howard Auditorium (Fluno Center on the University of Wisconsin-Madison Campus)

Howard Auditorium

Fluno Center on the University of Wisconsin-Madison Campus

601 University Avenue, Madison, WI 53715-1035

Availability of the Speaker

I will NOT be available on July 14th

Summary (2-4 sentences)

EAF is a JupyterHub-based scientific analysis facility hosted at Fermilab. With more than 35 different environments, it provides access to specialized hardware and cutting-edge machine learning, scientific software and grid utilities for Fermilab experiments, including remote HTCondor submission to our CMS LPC anlaysis cluster. EAF has enabled the use of Dask Gateway, a secure, multi-tenant server for managing distributed python analysis clusters through a hybrid model where task schedulers are executed as Kubernetes pods and task workers are submitted as HTCondor jobs. Once jobs reach the HTCondor worker node, a Dask worker payload is executed inside an apptainer container, which automatically announces itself to the scheduler. This enables EAF analyzers to add large chunks of resources on-the-fly by leveraging the power of HTCondor and Dask for fast, reliable analysis.

Primary author

Maria Acosta (Fermilab)

Presentation materials

There are no materials yet.