Speaker
Douglas Thain
(University of Notre Dame)
Description
A number of modern programming frameworks encourage end users to write concurrent functional programs that are expanded into task graphs, and then executed using local parallelism. While providing an elegant user experience, these systems struggle when presented with large programs that generate million-node graphs and must run on heterogeneous systems. We demonstrate a new framework, VineReduce, that allows for the transformation of these programs into an adaptive hierarchy that can effectively harness distributed, heterogeneous HTCondor pools.