Sign in to confirm you’re not a bot
This helps protect our community. Learn more

Managing Dropbox Processing Workflows with HTCondor

Partnership to Advance Throughput Computing
125 subscribers
46 views 4 years ago
Learn how to manage Dropbox processing workflows with HTCondor. Advantages of using Condor as a Dropbox workflow framework include: high-quality vetted implementations of essential capabilities, simple commands for easy monitoring and management, protection of input and output data integrity, the ability to transparently distribute work across multiple machines, a large global user community, and responsive developer support. Presented on May 26 by Micheal Pelletier as part of HTCondor Week 2021. Slide decks are available here: https://agenda.hep.wisc.edu/event/157... Questions? We're happy to answer questions about the PATh Project and help researchers, campus computing providers, and research collaborations leverage both CHTC and OSG resources! Visit our contact page: https://path-cc.io/contact/ and contact us at support@opensciencegrid.org with any questions. ...more
...more
These chapters are auto-generated

Intro

0:00

What is a Dropbox Processing Workflow?

0:56

What Capabilities are Needed?

1:46

Pitfalls of a Conventional Cron-Based Approach

3:17

Small Gains via a systemd Service Approach

4:30

Workflow System Layout

6:15

Introduction to HTCondor

7:07

HTCondor Submit Description File

7:34

Alternative for Periodic Jobs: Crondor

9:10

Holding a Job in Event of a Runtime Error

9:34

E-mail Notification

10:36

The Watchdog Job

11:37

Input Data Integrity Protection

12:41

The Watchdog's Mailer Job

14:03

Job Startup and Shutdown

15:31

Visualizing the Workflow

16:41

HTCondor is a Great Dropbox Workflow Framework

18:48
Managing Dropbox Processing Workflows with HTCondor
1Likes
46Views
2021Jun 16
Learn how to manage Dropbox processing workflows with HTCondor. Advantages of using Condor as a Dropbox workflow framework include: high-quality vetted implementations of essential capabilities, simple commands for easy monitoring and management, protection of input and output data integrity, the ability to transparently distribute work across multiple machines, a large global user community, and responsive developer support. Presented on May 26 by Micheal Pelletier as part of HTCondor Week 2021. Slide decks are available here: https://agenda.hep.wisc.edu/event/157... Questions? We're happy to answer questions about the PATh Project and help researchers, campus computing providers, and research collaborations leverage both CHTC and OSG resources! Visit our contact page: https://path-cc.io/contact/ and contact us at support@opensciencegrid.org with any questions.

Follow along using the transcript.

Partnership to Advance Throughput Computing

125 subscribers