An efficient framework for Java data processing systems in HPC environments

Aidan Fries; Javier Castañeda; Yago Isasi; Guillermo L. Taboada; Jordi Portell de Mora; Raül Sirvent

doi:10.1117/12.898217

12 October 2011 An efficient framework for Java data processing systems in HPC environments

Aidan Fries, Javier Castañeda, Yago Isasi, Guillermo L. Taboada, Jordi Portell de Mora, Raül Sirvent

Proceedings Volume 8183, High-Performance Computing in Remote Sensing; 818306 (2011) https://doi.org/10.1117/12.898217
Event: SPIE Remote Sensing, 2011, Prague, Czech Republic

Abstract

Java is a commonly used programming language, although its use in High Performance Computing (HPC) remains relatively low. One of the reasons is a lack of libraries offering specific HPC functions to Java applications. In this paper we present a Java-based framework, called DpcbTools, designed to provide a set of functions that fill this gap. It includes a set of efficient data communication functions based on message-passing, thus providing, when a low latency network such as Myrinet is available, higher throughputs and lower latencies than standard solutions used by Java. DpcbTools also includes routines for the launching, monitoring and management of Java applications on several computing nodes by making use of JMX to communicate with remote Java VMs. The Gaia Data Processing and Analysis Consortium (DPAC) is a real case where scientific data from the ESA Gaia astrometric satellite will be entirely processed using Java. In this paper we describe the main elements of DPAC and its usage of the DpcbTools framework. We also assess the usefulness and performance of DpcbTools through its performance evaluation and the analysis of its impact on some DPAC systems deployed in the MareNostrum supercomputer (Barcelona Supercomputing Center).

Citation Download Citation

Aidan Fries, Javier Castañeda, Yago Isasi, Guillermo L. Taboada, Jordi Portell de Mora, and Raül Sirvent "An efficient framework for Java data processing systems in HPC environments", Proc. SPIE 8183, High-Performance Computing in Remote Sensing, 818306 (12 October 2011); https://doi.org/10.1117/12.898217

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available