Throughput Computing 2025

Jun 2, 2025, 6:30 AM → Jun 6, 2025, 11:15 PM America/Chicago

Howard Auditorium (Fluno Center on the University of Wisconsin-Madison Campus)

Event Images

Monday, June 2

7:30 AM → 9:00 AM Registration, Coffee, and Light Breakfast

9:00 AM → 10:20 AM OSG at 20: Growing the Throughput Community

9:00 AM Navigating the PATh forward

Speaker: Miron Livny (UW-Center for High Throughput Computing)

Miron-Opening-Remarks-HTC25.pptx

9:20 AM Expanding Facilitation Impacts

Speaker: Christina Koch (UW Madison)

Koch-Monday-Slides-HTC25.pdf

9:40 AM State of OSG

Speaker: Frank Wuerthwein (UC San Diego)

Wuerthwein-PDF-HTC25.pdf

Wuerthwein-Slides-HTC25.pptx

10:00 AM The Pelican in Flight: Delivering Data Services

Speaker: Brian Bockelman (Morgridge Institute for Research)

HTC25 - In Flight.pdf

HTC25 - In Flight.pptx

10:20 AM → 10:50 AM Break

10:50 AM → 12:30 PM Erik Wright Keynote Presentation, David Swanson Awardee Presentation

10:50 AM Keynote Introduction

Speaker: Christina Koch (UW Madison)

10:55 AM Erik Wright's Keynote Presentation: Biological Discovery at an Unfathomable Scale

Recent technological advances have revealed an enormous diversity of lifeforms by sequencing their genomes. There are now millions of available genomes, each comprised of thousands of genes. The universe of newly discovered genes is expanding far faster than our ability to study them in the laboratory. Here, I will present how high-throughput computing is unlocking the function of novel genes at an unfathomable scale.

Speaker: Dr Erik Scott Wright (University of Pittsburgh)

Wright-Slides-HTC25

12:00 PM Introduction of David Swanson Award

Speaker: Ronda Swanson (University of Nebraska-Lincoln)

12:05 PM David Swanson Awardee Presentation: Reconstructing Spider Webs from Behavioral Tracking.

Speaker: Brandi Pessman (University of Nebraska-Lincoln)

Pessman-Slides-HTC25.pptx

12:30 PM → 1:30 PM Lunch

1:30 PM → 2:45 PM Throughput Computing in the Biology & Life Sciences Community

1:30 PM Using OSG to learn the rules of biological evolution (Remote Presentation)

Speaker: Oana Carja (Carnegie Mellon University)

1:55 PM Computational challenges in metagenomics and small molecule biosynthesis

Speaker: Jason Kwan (UW-Madison)

2:20 PM Understanding Gene Regulatory Networks

Speaker: Prakriti Garg (University of Wisconsin-Madison)

Garg-Slides-HTC25.pptx

2:45 PM → 3:15 PM Break

3:15 PM → 4:50 PM Reenvisioning Identity in Distributed Services

3:15 PM Our Current Vision for Trustworthy Long-Term CILogon Operations

Speaker: Jim Basney (University of Illinois Urbana-Champaign)

Basney-Slides-HTC25.pdf

3:40 PM Placement Tokens

Speaker: Matyas Selmeci (UW-Madison CHTC)

Selmeci - Placement Tokens - 2025.pdf

4:00 PM Can I have my data... please? Authorization in Pelican

Speaker: Justin Hiemstra (Morgridge Institute for Research)

Can I Have My Data... Please?

4:25 PM Fermilab’s Transition to Token Authentication

Fermilab is the first High Energy Physics institution to transition from X.509 user certificates to authentication tokens in production systems. All the experiments that Fermilab hosts are now using JSON Web Token (JWT) access tokens in their grid jobs. The tokens are defined using the WLCG Common JWT Profile. Many software components have been either created or updated for this transition, and the changes to those components are described. Most of the software is available to others as open source. There have been some glitches and learning curve issues but in general the system has been performing well and is being improved as operational problems are addressed.

Speaker: Dave Dykstra (Fermilab)

Dykstra-Slides-HTC25.pdf

4:50 PM → 5:05 PM CHTC Fellow Lightning Talks

Kashika-Lightning-Talk-HTC25 ppt.pptx

Sandy-Lightning-Talk-HTC25.pptx

William-Lightning-Talk-HTC25.pptx

5:05 PM → 5:15 PM Closing Remarks

Tuesday, June 3

6:30 AM → 7:30 AM Lakeshore Run

8:00 AM → 9:00 AM Coffee, Pastries and Registration

9:00 AM → 10:35 AM Campus Impacts on the National Science Community via the OSDF and OSPool

9:00 AM NSF Campus Cyberinfrastructure (CC*) (Remote Presentation)

Speaker: Kevin Thompson (NSF)

Thompson-Slides-HTC25.pdf

9:20 AM Translational Computer Science Panel Discussion

Panel Discussion led by Miron LIvny.

Speakers: Brian Bockelman (Morgridge Institute for Research), Douglas Thain (University of Notre Dame), Ewa Deelman (USC ISI), Manish Parashar (University of Utah), Miron Livny (UW-Center for High Throughput Computing), Peter Couvares (LIGO Laboratory - Caltech)

Miron-Panel-Slides-HTC25.pptx

9:55 AM University of Montana and Contributing to the OSPool

Speaker: Michael Couso (University of Montana)

Couso-Slides-HTC25.pptx

10:15 AM Experiences of a Small, Primarily Undergraduate Institution in Servicing OSPool Compute Jobs

Speaker: Dr Stephen Wheat (Oral Roberts University)

Wheat-Slides-HTC25.pptx

10:35 AM → 11:00 AM Break

11:00 AM → 12:30 PM HTC at Work at Campuses and in Research

11:05 AM Scaling Up Research: Integrating CENVAL-ARC resources with OSG and Expanding User Access (Remote Presentation)

Speaker: Sarvani Chadalapaka (University of California - Merced)

Chadalapaka-Slides-HTC25.pptx

11:25 AM Supporting Research Computing @ Syracuse University

Speaker: Peter Pizzamenti (Syracuse University)

Pizzimenti-Slides-HTC25.pptx

11:50 AM Supporting MicrobiologyResearch at Scale: Experiences and Perspectives

Speaker: Patricia Tran (UW-Madison)

Tran-PDF-HTC25.pdf

12:10 PM Integrating NSF NCAR’s data infrastructure with OSDF

NSF NCAR’s labs and programs collectively cover a breadth of research topics in Earth system science, from the effects of the Sun on Earth's atmosphere to the role of the ocean in weather and climate prediction, as well as supporting and training the next-generation of Earth system scientists. However, with the current legacy `download and analyze’ model followed by most of our remote users, we are not realizing the full research potential of NCAR’s wealth of datasets. Our goal is to integrate NCAR’s curated data collections with the OSDF data and compute fabric to broaden access capabilities. In this talk, we present progress on this collaboration and demonstrate geoscience workflows which ingest data from NCAR’s Research Data Archive using pelicanFS via OSDF caches.

Speaker: Harsha Hampapura (NSF National Center for Atmospheric Research)

Hampapura-Slides-HTC25.pptx

12:30 PM → 1:30 PM Lunch

1:30 PM → 2:00 PM Doors Open for 40 Years of The Condor Project: A Celebration (In-Person Event Only)

2:00 PM → 6:30 PM 40 Years of The Condor Project Commemorative Gathering (In-Person Event Only)

2:00 PM Welcome to 40 Years of The Condor Project

Speaker: Todd Tannenbaum (University of Wisconsin)

2:05 PM Reflections from Guri Sohi

2:15 PM Reflections from Rajesh Raman

2:25 PM Reflections from Ewa Deelman

2:35 PM Condor in Europe

Speaker: Helge Meinhard (CERN)

2:50 PM Reflections from Manish Parashar

3:00 PM Reflections from Peter Couvares

3:10 PM Reflections from Jim Basney

3:20 PM Reflections from Brian Bockelman

Speaker: Brian Bockelman (Morgridge Institute for Research)

3:30 PM Reflections from Remzi Arpaci-Dusseau

3:40 PM Reflections from Doug Thain

3:50 PM Reflections from the Floor

4:00 PM Reflections from Miron Livny

4:30 PM Reception

6:30 PM → 8:30 PM Evening on the Memorial Union Terrace

Wednesday, June 4

8:00 AM → 9:00 AM Coffee, Pastries and Registration

9:00 AM → 9:30 AM Advancing Technologies Through Community

9:00 AM Building Communities Around OSDF and OSPool Contributors

Speakers: Christina Koch (UW Madison), Tim Cartwright (University of Wisconsin–Madison, OSG)

Koch-Cartwright-Slides-2-HTC25.pdf

9:30 AM → 10:45 AM Campus Track: Campus/CC* Contributors (to OSPool & OSDF, current & future): Meet the Team and Stump the Experts

9:30 AM → 10:45 AM Interactive Access and Administration

9:30 AM Integration of MINCER with Open Science Grid

The Monitoring Infrastructure for Network and Computing Environment Research (MINCER) project aims to provide a foundation for in-depth insight and analysis of distributed heterogeneous computing environments, supporting and enhancing research and education in computer and network systems. Our approach is to work in conjunction with the Open Science Grid (OSG) by providing a set of MINCER containers for integration, including the necessary software to obtain measurements for different types of system experiments on available platforms. These containers will be accompanied by detailed instructions and examples illustrating their use, and will be submitted to the OSG repository for broader community access.
To ensure portability and reproducibility, we will follow best practices for constructing lightweight containers, providing examples that illustrate how to robustly specify dependencies not directly included in the container environment. Our current focus includes containers that enable performance and power measurement capabilities for workflows running on GPUs. In parallel, we are developing a separate profiling tool that leverages cyPAPI to extract compute and memory metrics from Python applications, generating roofline models for performance analysis. This tool will also be containerized to facilitate streamlined deployment and analysis across varied platforms, further supporting comprehensive insights into GPU-based workflows.
Furthermore, we are developing a container-based strategy for enabling OSG sites contributing to OSG’s OSPool to collect more detailed job-specific data using tools such as DCGM and LDMS. DCGM (Data Center GPU Manager) is an NVIDIA tool that tracks GPU metrics like utilization, memory use, and power. LDMS (Lightweight Distributed Metric Service) is a low-overhead system for collecting performance data from clusters. Using these tools will allow OSG sites to export more comprehensive data to the OSG dashboard if desired. They can also use this job-level data to better tune their systems to meet workload demands.

Speaker: Irvin Lopez-Audetat (University of Texas at El Paso)

Lopez-Audetat-Slides-HTC25.pptx

9:50 AM Monitoring and diagnostics to support scaling up radio astronomy imaging workflows (Remote Presentation)

The Algorithms Research and Development Group (ARDG) at the National Radio Astronomy Observatory (NRAO) has been using HTCondor and compute resources at the Open Science Grid (OSG) to significantly improve throughput in radio astronomy imaging by up to 2 orders of magnitude in single imaging workflows, and we are currently putting efforts towards extending these imaging capabilities to multiple imaging workflows. Besides developing and maintaining software to efficiently process data and manage the imaging workflows, we found that monitoring and diagnosing workflow executions is critical to achieve and maintain high throughput. We will present and discuss some of the tools and methodology that we use to assess workflow health and efficiency, which enable us to visualize and solve problems, eventually also contributing to optimizing and advancing capabilities of HTCondor and the OSG.

Speaker: Felipe Madsen (National Radio Astronomy Observatory)

fmadsen_HTC25.pdf

10:10 AM Managing, Maintaining, and Monitoring a large HTC system

Speaker: Tom Smith (Brookhaven National Laboratory)

Smith-Slides-2-HTC25.pdf

10:30 AM A Tape Robot for the MIT Tier-2 Center

Since its establishment in 2006, the MIT Tier-2 computing center has been a long-standing contributor to CMS computing efforts. Recently, an opportunity arose to take part in the usage of a shared tape storage system operated by Harvard University. In the context of a pilot project to explore this system we acquired tape cartridges with a total capacity of 15 PB and successfully integrated them for use by CMS. A key challenge was the lack of direct access to the tape libraries. To address this, we developed a novel setup that is currently unique within the CMS infrastructure. In this talk, we present our technical approach and share insights that may benefit other sites considering participation in shared tape storage systems.

Speaker: Maxim Goncharov (MIT)

mit-tape-general.pdf

10:45 AM → 11:15 AM Break

11:15 AM → 12:30 PM Campus Track: Campus/CC* Technical Lightning Talks and Discussion

11:15 AM → 12:30 PM Tutorials and Workflows Track: Building Your Toolbox: HTCSS Overviews

11:15 AM Duct Tape, DAGs, and Determination: Snakemake at the Edge of HTCondor

Speaker: Justin Hiemstra (Morgridge Institute for Research)

Hiemstra Slides

11:40 AM Wrangling Complex Notebook Workflows with Floability

Computational notebooks have become a critical tool of scientific discovery, by wrapping together code, results, and visualization into a common package. However, moving complex notebooks between different facilities is not so easy: complex workflows require precise software stacks, access to large data, and large backend computational resources. The Floability project aims to connect these two worlds, making it possible to specify, share, and execute computational workflows through the familiar notebook interface. This talk will introduce the Floability concept of a workflow "backpack" and demonstrate applications in high energy physics, machine learning, and geosciences.

Speaker: Douglas Thain (University of Notre Dame)

Thain-Douglas-Slides-2-HTC25.pptx

12:05 PM EWMS in Action: A User’s Guide to Adaptive, Extreme-Scale Workflows

The Event Workflow Management System (EWMS) enables previously impractical scientific workflows by transforming how HTCondor is used for massively parallel, short-runtime tasks. This talk explores what’s now possible from a user’s perspective. Integrated into IceCube’s Realtime Alert pipeline and powered by OSG’s national-scale compute resources, EWMS’s debut application delivers directional reconstructions of high-energy neutrinos within minutes. The system’s user-first design streamlines scientific workflows, while built-in tools provide reliable administrative control. This talk will showcase how EWMS is accelerating discovery today and explore how its capabilities could unlock new research across domains—from astrophysics to protein modeling, large-scale text mining, and beyond.

Speaker: Ric Evans (UW-Madison / IceCube)

Evans-Slides-HTC25.pdf

12:30 PM → 1:30 PM Lunch

1:30 PM → 2:45 PM Tutorials and Workflows Track: Workloads and Workflows: Debugging and Observability

1:30 PM Monitoring in the OSDF

Speaker: Patrick Brophy (Morgridge)

Brophy-Slides-HTC25.pptx

1:55 PM Disk Usage of Jobs at the EP

Speaker: Cole Bollig (University of Wisconsin-Madison)

Bollig-Slides-HTC25.pptx

2:20 PM Hollistic cost analysis of running a computing center

The MIT Tier-2 computing center, established in 2006, has been a long-standing contributor to CMS computing. As hardware ages and computing demands evolve, we are undertaking a major redesign of the center’s infrastructure. In this talk, we present a holistic cost analysis that includes not only hardware purchases but also power consumption, cooling, and rack space—factors often excluded from conventional cost models. Using power measurements under typical CMS workloads, we evaluate the cost-effectiveness of maintaining aging hardware versus timely replacement, and propose optimal hardware retirement and procurement policies.

Speaker: Zhangqier Wang (Massachusetts Inst. of Technology (US))

sustain_T2_htc_wangz.pdf

sustain_T2_htc_wangz.pptx

2:45 PM → 3:05 PM Break

3:05 PM → 4:50 PM Scientific Collaborations Track: Collaborations Group Discussions

3:05 PM IceCube and PATh Services

Speaker: Benedikt Riedel (UW-Madison)

HTC25.pdf

3:25 PM Computing Requirements and Challenges for Muon Collider Full Simulation

Speaker: Mark Larson (University of Chicago)

Computing Requirements and Challenges for Muon Collider Detector Simulation.pdf

Computing Requirements and Challenges for Muon Collider Detector Simulation.pptx

3:45 PM Status of REDTOP and preliminary results from the 2025 Montecarlo campaign (Remote Presentation)

Speaker: Corrado Gatto (INFN and NIU)

HTC25_20250602_REDTOP.pdf

4:05 PM Data processing in XENONnT dark matter project (Remote Presentation)

Speaker: Dacheng Xu (Columbia University)

XENONnT_outsource_compressed.pdf

4:25 PM Collaboration Services & Usage

Speaker: Pascal Paschos (University of Chicago)

Collaborations Services & Usage

3:05 PM → 4:50 PM Tutorials and Workflows Track: Objects, Objects Everywhere: Using the OSDF in your Science

3:05 PM Data Everywhere: Using and Sharing Scientific Data with Pelican

While there are perhaps hundreds of petabytes of datasets available to researchers, instead of swimming in seas of data there is often a feel of sitting in a data desert: there’s a mismatch between what sits in carefully curated repositories around the world versus what’s accessible at the computational resources locally available. The Pelican Project (https://pelicanplatform.org/) aims to bridge the gap between repositories and compute by providing a software platform to connect the two sides. Pelican’s flagship instance, the Open Science Data Federation (OSDF), serves billions of objects and more than a hundred petabytes a year to national-scale resources. This tutorial, targeted at end-user data consumers and data providers, will cover the data access model of Pelican, guide participants to access and share data through an existing data federation, and consider how data movement via Pelican and the OSDF can enable their research computing.

Speaker: Dr Andrew Owen (UW-Madison/CHTC)

Owen-Interactive-Environment

Owen-Materials

Owen-Slides

4:30 PM Unbreaking the bird: Debugging Pelican client failures

Speaker: Brian Bockelman (Morgridge Institute for Research)

HTC25 - Debugging Pelican.pdf

HTC25 - Debugging Pelican.pptx

4:50 PM → 5:00 PM Closing Remarks

6:00 PM → 7:00 PM Evening Bike Ride

Thursday, June 5

8:00 AM → 9:00 AM Coffee, Pastries and Registration

9:00 AM → 10:30 AM Joint CMS and US-ATLAS Session: Monitoring and Measuring

We have a common notes and action items document at:
https://docs.google.com/document/d/1y3V4HKGQ8EMUTxH9_MDCqegtA8sGeNtu6UmFk2fRyLQ/edit?usp=sharing

Conveners: Andrew Melo (Vanderbilt University), Shawn McKee (University of Michigan)

Agenda notes and action items

9:00 AM Networking: perfSONAR, SciTags and WLCG Site network monitoring

Speaker: Shawn McKee (University of Michigan)

Networking: perfSONAR, SciTags and WLCG Site Network Monitoring

USATLAS-USCMS Networking Status & Discussion.pdf

9:15 AM Tape Archival at Tier-2s

Experiences with running tape storage systems at ATLAS and CMS Tier-2s

Speakers: Maxim Goncharov (MIT), Rafael Coelho Lopes de Sa (University of Massachusetts Amherst)

mit-tape-cms.pdf

Rafael_HTC25_NET2tape.pdf

9:40 AM Discussion on next steps and joint work

Speakers: Andrew Melo (Vanderbilt University), Shawn McKee (University of Michigan)

Meeting Notes and Action Items

9:00 AM → 10:30 AM Running ML, AI and GPU Workflows

9:00 AM GPU Access and AI workflows in CHTC and the OSPool

Speaker: Amber Lim (UW Madison)

Lim-Slides-HTC25.pptx

9:25 AM ARA Distributed Inference Experiments: Flying HTCondor Over a Field of Wireless Dreams

We present the initial design and proposed implementation for a series of long-baseline, distributed inference experiments leveraging ARA --- a platform for advanced wireless research that spans approximately 500 square kilometers near Iowa State University, including campus, the City of Ames, local research and producer farms, and neighboring rural communities in central Iowa. These experiments aim to demonstrate, characterize, and evaluate the use of distributed inference for computer vision tasks in rural and remote regions where high-capacity, low-latency wireless broadband access and backhaul networks enable edge computing devices and sensors in the field to offload compute-intensive workloads to cloud and high-performance computing systems embedded throughout the edge-to-cloud continuum. In each experiment, a distributed implementation of the MLPerf Inference benchmarks for image classification and object detection will measure standard inference performance metrics for an ARA subsystem configuration under different workload scenarios. Real-time network and weather conditions will also be monitored throughout each experiment to evaluate their impact on inference performance. Here, we highlight the role of HTCondor as the common scheduler and workload manager used to distribute the inference workload across ARA and beyond. We also discuss some of the unique challenges in deploying HTCondor on ARA and provide an update on the current status of the project.

Speaker: Martin Kandes (San Diego Supercomputer Center)

adix-flying-htcondor-over-a-field-of-wireless-dreams.pdf

9:50 AM Pegasus WMS Deployments in ACCESS and NAIRR Pilot

Pegasus is a widely used scientific workflow management system built on top of HTCondor DAGMan. This talk will highlight how Pegasus is deployed within the NSF ACCESS ecosystem and the NAIRR Pilot. We will cover access point deployments, including the hosted ACCESS Pegasus platform (Open OnDemand and Jupyter), workflow execution nodes in HPC environments, and a JupyterLab-based access point within the Purdue Anvil Composable Subsystem. On the execution side, we will discuss several provisioning strategies, including HTCondor Annex, custom virtual machines on the Jetstream2 cloud, simple glidein configurations for campus clusters, and a dynamically autoscaled TestPool environment designed for workflow development and testing.

Speaker: Mats Rynge (USC / ISI)

htc25-rynge-pegasus.pdf

10:15 AM Experiments and expansions: Leveraging PATh tools and NAIRR resources in ML workflows

Speaker: Ian Ross (U. Wisconsin)

Ross-Slides-HTC25.pptx

10:30 AM → 11:00 AM Break

11:00 AM → 12:30 PM AI-driven Science with PATh Services

11:00 AM Contouring the Audio Fovea with Pinna Cues for Spatial Speech Perception (Remote Presentation)

Speaker: Bujji Selagamsetty (University of Wisconsin-Madison)

Selagamsetty-Slides-HTC25.pdf

11:25 AM Mapping the Zymomonas mobilis Interactome

Speaker: Sameer DCosta (GLBRC / WEI)

D'Costa-Slides-HTC25.pdf

11:50 AM Large Scale Dataset Curation and Model Evaluation

Speaker: John Peters (UW-Madison)

Peters-Slides-HTC25.pdf

12:10 PM Utilizing HTCondor, Pelican, and DAGman workflows for high-throughput phenotyping in dairy cattle (Remote Presentation)

Speaker: Ariana Negreiro (University of Wisconsin-Madison)

Negreiro-Slides-HTC25.pptx

11:00 AM → 12:30 PM Joint CMS and US-ATLAS Session: WLCG (Mini) Data Challenges

We have a common notes and action items document at:
https://docs.google.com/document/d/1y3V4HKGQ8EMUTxH9_MDCqegtA8sGeNtu6UmFk2fRyLQ/edit?usp=sharing

Conveners: Andrew Melo (Vanderbilt University), Shawn McKee (University of Michigan)

Agenda notes and action items

11:00 AM Capacity Challenges

Discuss tools and options for the next capacity challenge.

Speaker: Hironori Ito (Brookhaven National Laboratory)

Throughput Capacity Challenge OSG 2025.pdf

11:15 AM Capability Challenge: SENSE/Rucio

Discuss plans for SENSE/Rucio testing for USATLAS/USCMS

Speakers: Diego Davila (UCSD), Justas Balcas (ESnet)

SENSE_Rucio - HTC25.pdf

11:30 AM Capability Challenge: Netbird/Wireguard

Speaker: Lincoln Bryant (University of Chicago)

WireGuard Capability Testing.pdf

11:45 AM Discussion & Planning

Discuss capacity and capability challenges. Which are we interesting in pursuing? Who will participate? When to schedule?

Speakers: Andrew Melo (Vanderbilt University), Shawn McKee (University of Michigan)

12:30 PM → 1:30 PM Lunch

1:30 PM → 2:45 PM Joint CMS and US-ATLAS Session: AI/ML, Tools and Operations

We have a common notes and action items document at:
https://docs.google.com/document/d/1y3V4HKGQ8EMUTxH9_MDCqegtA8sGeNtu6UmFk2fRyLQ/edit?usp=sharing

Conveners: Andrew Melo (Vanderbilt University), Shawn McKee (University of Michigan)

Agenda notes and action items

1:30 PM HEPCDN: Exploring NGINX for Content Distribution

Speaker: Andrew Melo (Vanderbilt University)

NGINX

1:45 PM CANCELED: Data Streaming as a Service

Presentation/discussion canceled.

Speaker: Alexei Klimentov (Brookhaven National Lab)

2:00 PM AI/ML Introduction and Discussion

Quick overview of AI/ML in WLCG so far
Needs for AI/ML for Infrastructure AND Infrastructure for AI/ML
Funding opportunities
Next step, areas on common interest/effort?

Speaker: Ilija Vukotic (University of Chicago)

AI supporting science.pdf

AI Supporting Scientific Computing

1:30 PM → 2:45 PM Tutorial: Mastering Debugging (for admins)

1:30 PM Tracking HTCondor Uptime

While the DaemonStartTime and MonitorSelfAge attributes of HTCondor daemons provide a slice of insight as to the uptime and availability of the service, they're not well-suited for tracking longer-term up/down-time stats over the course of days, weeks, or months.

One illustration of this limitation is that if a malfunctioning node or service restarts every five minutes, the values are reset to zero each time and there's no accumulation of the total uptime across the restarts.

Longer-time-period uptime statistics are essential for contractual Service Level Agreement (SLA) management, and are an important aspect of monitoring the overall health of large HTCondor pools.

Using straightforward scripting, the start daemon's Cron system, and an external file to store an uptime-centered ClassAd, long-term statistics can be maintained and easily queried.

Speaker: Michael Pelletier

Tracking HTCondor Uptime TC25.pptx

1:55 PM Rolling HTCondor Upgrades Without Rolling Over

Speaker: Tim Theisen (UW-Madison CHTC)

Theisen-Slides-HTC25.pdf

2:20 PM Tighter HTCondor and Kubernetes interplay for better glideins

HTCondor is the leading system for building a dynamic overlay batch scheduling system on resources managed by any scheduling system, by means of glideins. One fundamental property of these setups is the use of late binding of containerized user workloads. From a resource provider point of view, a compute resource is thus claimed before the user container image is selected. Kubernetes allows for both multi-container requests and dynamic updates to the container image being used. In this talk we show how HTCondor can exploit these features to both increase the effectiveness and the security of gildeins running on top of Kubernetes-managed resources.

Speakers: Igor Sfiligoi (University of California San Diego), Jaime Frey (Center for High-Throughput Computing)

htc25_k8s_glideins_rc3.pdf

2:45 PM → 3:05 PM Break

3:05 PM → 3:55 PM HTCondor: New Things You Should Know

3:05 PM Functional Throughput Computing

Speaker: Greg Thain (Center for High Throughput Computing)

FunctionalDistributedProgramming.pptx

3:30 PM What's New, What's Upcoming

Speaker: Todd Tannenbaum (University of Wisconsin)

WhatsNew2025.pptx

3:05 PM → 3:55 PM Joint CMS and US-ATLAS Session: Analysis Facilities (AFs)

We have a common notes and action items document at:
https://docs.google.com/document/d/1y3V4HKGQ8EMUTxH9_MDCqegtA8sGeNtu6UmFk2fRyLQ/edit?usp=sharing

Conveners: Hironori Ito (Brookhaven National Laboratory), Lincoln Bryant (University of Chicago)

Agenda notes and action items

3:05 PM Introduction and Overview for AFs in WLCG

Speaker: Lincoln Bryant (University of Chicago)

Analysis Facilities.pdf

3:20 PM AF Discussion

Shared development and prototyping for AFs?
Storage technologies to support AFs (Carlos Gamboa ?, 10 minutes)
Joint AFs: Can we “share” AFs between experiments: Belle II and ATLAS or Dune and CMS, etc ? (Hiro/Ofer/Lincoln ?)
Can we agree on a minimum baseline for AFs?
Standard “login”

Speakers: Hironori Ito (Brookhaven National Laboratory), Lincoln Bryant (University of Chicago)

3:55 PM → 4:10 PM Break

4:10 PM → 4:55 PM Tools for Production Pools

4:10 PM From backend to interactive based on HTCondor

New requirements from HEP data analysis includes limited access to login nodes, resource needed from the experiments program rather than the login nodes and efficient data access for collaborative workflows etc. We have developed an Interactive aNalysis worKbench (INK), a web-based platform leveraging the HTCondor cluster. INK transforms traditional batch-processing resources into a user-friendly, web-accessible interface, enabling researchers to leveage the cluster computing and storage resources directly via their browsers.
A loosely coupled architecture with token-based authentication to ensures thesecurity, while fine-grained permission management allows customizable access for users and experimental applications. Universal public interfaces abstract the heterogeneity of underlying resource heterogeneity. Following the first version released in Mar.2025, user feedback has been highly positive.

Speaker: Jingyan Shi (INSTITUTE OF HIGH ENERGY PHYSICS, Chinese Academy of Science)

From backend to interactive based on HTCondor_v2.pptx

4:35 PM Operating a Federated HTCondor Infrastructure: Monitoring and Management for CMS Computing

The Compact Muon Solenoid (CMS) experiment at CERN generates and processes vast volumes of data requiring significant computing capacity. To meet these demands, CMS has adopted a federated throughput computing model distributed across a global infrastructure based on HTCondor, the CMS Submission Infrastructure. Seamless integration of heterogeneous resources from multiple sites allows for operating a unified, virtualized pool. This infrastructure currently provides access to over 500,000 CPU cores, enabling CMS to efficiently execute a wide variety of data processing and simulation workloads.

This federation, however, comes with substantial operational challenges, notably, the need for robust and scalable monitoring. To ensure reliability, performance, and rapid diagnosis of issues, we have developed a comprehensive monitoring ecosystem that spans job execution, resource availability, and system health across the entire pool. This talk will present the architecture of the CMS federated compute infrastructure, detail the role of HTCondor in enabling global workload distribution, and highlight recent developments in monitoring that are critical to operating such a large-scale system effectively.

Speaker: Bruno Coimbra (Fermilab)

Operating a Federated HTCondor Infrastructure_ Monitoring and Management for CMS Computing -1.pdf

4:55 PM → 5:05 PM Closing Remarks

6:00 PM → 7:30 PM Sunset Kayak or Paddleboard

8:45 PM → 10:45 PM Karaoke Night

Friday, June 6

8:00 AM → 9:00 AM Coffee, Pastries and Registration

9:00 AM → 10:25 AM Throughput Computing Science Impacts

9:00 AM High-resolution Imaging of the Multi-phase Interstellar Medium with CHTC

Speaker: Nickolas Pingel (University of Wisconsin-Madison)

Pingel-Slides-HTC25.pptx

9:25 AM Processing Mouse Brain Data on CHTC using Research Drive Integration

Speaker: Aydan Bailey (UW-Madison)

Bailey-Slides-HTC25.pptx

9:45 AM Visual proteomics, powered by CHTC

Speaker: Raison D'Souza (University of Wisconsin-Madison)

10:05 AM High Throughput Computing for Comparative Genomics on Large Public Datasets

Speaker: Conor Bendett (University of Wisconsin-Madison)

Bendett-Slides-HTC25.pdf

10:25 AM → 10:45 AM Break

10:45 AM → 12:30 PM Managed and Staging Objects: Caches and Execution Points

10:45 AM Files Common Across Jobs and How To Transfer Them

Speaker: Todd Miller (CHTC)

Miller-Slides-HTC25.pptx

11:10 AM condor_who (are you)

Speaker: John Knoeller (University of Wisconsin, Madison)

HTC25-condor_who-are-you.pptx

11:30 AM Kingfisher: Toward Explicit Space Management

Speaker: Justin Hiemstra (Morgridge Institute for Research)

Hiemstra Kingfisher

11:55 AM Using the National Data Platform Endpoint to Improve Access to Science Data

I describe the process of deploying the National Data Platform Endpoint (formerly Point of Presence / POP) on local infrastructure to provide a data streaming service for a published science dataset where the data origin is located in Hawaii. From the perspective of a software engineer I will cover the process of deploying the endpoint into a Kubernetes cluster or using Docker Compose. I will explain how I integrated access to local data with the endpoint and describe some useful lessons that were learned. Finally, from the perspective of a science user, I will demonstrate how to use the deployed endpoint to preprocess data and stream the results to a Jupyter notebook to visualize the data.

Speaker: Curt Dodds (Institute for Astronomy, University of Hawaii)

Dodds-Slides-HTC25.pptx

12:20 PM Closing Remarks

Speaker: Miron Livny (UW-Center for High Throughput Computing)

Article to display during Friday Closing Remarks

12:30 PM → 1:30 PM Lunch