DataConnector: A Data processing framework integrating hadoop and a grid middleware OGSA-DAI for cloud environment

Abstract

Recent mobile internet services make use of computing resources provided in forms of Cloud computing. However, conventional data management framework faces performance problems when importing external heterogeneous data and processing the vast amount of data with Cloud computing technology. In this paper, we propose a data processing framework for cloud applications based on OGSA-DAI (Open Grid Service Architecture-Data Access & Integration) for heterogeneous external data importing and MapReduce for big data processing. We designed and implemented a framework called DataConnector extending OGSA-DAI middleware which can access and integrate distributed data in a heterogeneous environment, and we deployed DataConnector into a Cloud environment. We conducted various experiments for evaluation and showed that our approach can be used for fast heterogeneous external data access and efficient large data processing with negligible or no system overhead. © 2013 International Information Institute.