Srinath Perera

Explore the Hadoop MapReduce v2 ecosystem to gain insights from very large data-sets.

In the last few years, we have seen much interest in data processing technologies. Although initial interests focused on batch processing technologies like MapReduce, people have realized the need for more responsive technologies such as stream processing and complex event processing. Complex event processing has been historically used within a single node or a cluster of tightly interconnected nodes. However, to be effective with Big Data use-cases, CEP technologies need to be able to scale up… Show more

In the last few years, we have seen much interest in data processing technologies. Although initial interests focused on batch processing technologies like MapReduce, people have realized the need for more responsive technologies such as stream processing and complex event processing. Complex event processing has been historically used within a single node or a cluster of tightly interconnected nodes. However, to be effective with Big Data use-cases, CEP technologies need to be able to scale up to handle large use-cases. This paper presents several approaches to scale complex event processing by distributing it across several nodes. Wihidum discusses how to balance the workload among nodes efficiently, how complex event processing queries can be broken up into simple sub queries, and how queries can be efficiently deployed in the cluster. The paper focuses on three techniques used for scaling queries: pipelining, partitioning and distributed operators. Then it discusses in detail the distribution of few CEP operators: filters, joins, pattern matching, and partitions. Empirical results show that the techniques followed in Wihidum have improved the performance of the CEP solution. Show less

The DEBS Grand Challenge is an annual event in which different event-based systems compete to solve a real-world problem. The 2014 challenge is to demonstrate scalable real- time analytics using high-volume sensor data collected from smart plugs over a one and a half month period. This paper aims to show how a general-purpose commercially available event-based system - the WSO2 Complex Event Processor (WSO2 CEP) - was used to solve this problem. We achieved 400K throughput for single node and… Show more

The DEBS Grand Challenge is an annual event in which different event-based systems compete to solve a real-world problem. The 2014 challenge is to demonstrate scalable real- time analytics using high-volume sensor data collected from smart plugs over a one and a half month period. This paper aims to show how a general-purpose commercially available event-based system - the WSO2 Complex Event Processor (WSO2 CEP) - was used to solve this problem. We achieved 400K throughput for single node and came close to one million throughput with four nodes. In addition, we explore areas where we created extensions to the WSO2 CEP engine to better solve the challenge. Show less

Mediation is an important tool used in systems built by aggregating many components together, and it is considered to be a key element of SOA. Typically, each message mediated by an ESB undergoes several parse and copy operations thereby incurring a severe performance penalty.
This paper presents several optimizations to ESB design that enable lower-overhead message mediation by reducing processing requirements, eliminating buffer copies, and facilitating network-level data transfers… Show more

Mediation is an important tool used in systems built by aggregating many components together, and it is considered to be a key element of SOA. Typically, each message mediated by an ESB undergoes several parse and copy operations thereby incurring a severe performance penalty.
This paper presents several optimizations to ESB design that enable lower-overhead message mediation by reducing processing requirements, eliminating buffer copies, and facilitating network-level data transfers. We also detail the performance gains that we have been able to achieve – close to two orders of magnitude in some cases – in a redesign of a popular implementation.
Show less

Instant MapReduce Patterns – Hadoop Essentials How-to is a concise introduction to Hadoop and programming with MapReduce. It is aimed to get you started and give you an overall feel for programming with Hadoop so that you will have a well-grounded foundation to understand and solve all of your MapReduce problems as needed.

Instant MapReduce Patterns – Hadoop Essentials How-to will start with the configuration of Hadoop before moving on to writing simple examples and discussing MapReduce… Show more

Instant MapReduce Patterns – Hadoop Essentials How-to is a concise introduction to Hadoop and programming with MapReduce. It is aimed to get you started and give you an overall feel for programming with Hadoop so that you will have a well-grounded foundation to understand and solve all of your MapReduce problems as needed.

Instant MapReduce Patterns – Hadoop Essentials How-to will start with the configuration of Hadoop before moving on to writing simple examples and discussing MapReduce programming patterns.

We will start simply by installing Hadoop and writing a word count program. After which, we will deal with the seven styles of MapReduce programs: analytics, set operations, cross correlation, search, graph, Joins, and clustering. For each case, you will learn the pattern and create a representative example program. The book also provides you with additional pointers to further enhance your Hadoop skills. Show less

A one-stop guide to processing large and complex data sets using the Hadoop ecosystem. "Hadoop MapReduce Cookbook" introduces you to simple examples and then dives deep to solve in-depth big data use cases.

"Hadoop MapReduce Cookbook" presents more than 50 ready-to-use Hadoop MapReduce recipes in a simple and straightforward manner, with step-by-step instructions and real world examples. Start with how to install, then configure, extend, and administer Hadoop. Then write simple examples,… Show more

A one-stop guide to processing large and complex data sets using the Hadoop ecosystem. "Hadoop MapReduce Cookbook" introduces you to simple examples and then dives deep to solve in-depth big data use cases.

"Hadoop MapReduce Cookbook" presents more than 50 ready-to-use Hadoop MapReduce recipes in a simple and straightforward manner, with step-by-step instructions and real world examples. Start with how to install, then configure, extend, and administer Hadoop. Then write simple examples, learn MapReduce patterns, harness the Hadoop landscape, and finally jump to the cloud.

The book deals with many exciting topics such as setting up Hadoop security, using MapReduce to solve analytics, classifications, on-line marketing, recommendations, and searching use cases. You will learn how to harness components from the Hadoop ecosystem including HBase, Hadoop, Pig, and Mahout, then learn how to set up cloud environments to perform Hadoop MapReduce computations. Show less

NoSQL storage provides a flexible and scalable alternative to relational databases, and among many such storages, Cassandra is one of the popular choices. Move beyond the well-known details and explore the less obvious details associated with Cassandra. You'll examine the Cassandra data model, storage schema design, architecture, and potential surprises associated with Cassandra.

A combination of factors: expanding user bases, the ubiquity of mobile communications, and newer technologies such as cloud computing and multi-core computing, are pushing todays systems to grow larger and larger. With their loosely coupled nature, distributed messaging systems often play a key role in such architectures. However, just like other parts of the architecture, those messaging systems also need to scale up, and they need to do so in three dimensions: quantity of messages, number of… Show more

A combination of factors: expanding user bases, the ubiquity of mobile communications, and newer technologies such as cloud computing and multi-core computing, are pushing todays systems to grow larger and larger. With their loosely coupled nature, distributed messaging systems often play a key role in such architectures. However, just like other parts of the architecture, those messaging systems also need to scale up, and they need to do so in three dimensions: quantity of messages, number of users, and size of messages. Although most current systems handle the first two dimensions, few of them efficiently support the third dimension. This paper proposes a novel method to implement a scalable and persistent broker that supports a publish/subscribe model and distributed queues using a NoSQL database and a coordination framework. We will discuss the design that uses recent advances in scalable database management and distributed coordination middleware, and we will compare the proposed models with other distributed message brokers. Show less

The out put of the work done building WSO2 Cloud Gateway.

Complex event processing typically requires studying the query language of a CEP engine and writing the required queries by hand. This is an overwhelming task, and requires extensive prior knowledge. In addition to that, the coding process becomes complicated with complex requirements and multiple steps involved. Therefore a visual editor which lets users construct CEP programs visually is a highly desirable addition to the CEP community. In this paper, we present an approach to creating a… Show more

Complex event processing typically requires studying the query language of a CEP engine and writing the required queries by hand. This is an overwhelming task, and requires extensive prior knowledge. In addition to that, the coding process becomes complicated with complex requirements and multiple steps involved. Therefore a visual editor which lets users construct CEP programs visually is a highly desirable addition to the CEP community. In this paper, we present an approach to creating a Visual Editor for construction of CEP programs. Show less

Today there are so much data being available from sources like sensors (RFIDs, Near Field Communication), web activities, transactions, social networks, etc. Making sense of this avalanche of data requires efficient and fast processing. Processing of high volume of events to derive higher-level information is a vital part of taking critical decisions, and Complex Event Processing (CEP) has become one of the most rapidly emerging fields in data processing. e-Science use-cases, business… Show more

Today there are so much data being available from sources like sensors (RFIDs, Near Field Communication), web activities, transactions, social networks, etc. Making sense of this avalanche of data requires efficient and fast processing. Processing of high volume of events to derive higher-level information is a vital part of taking critical decisions, and Complex Event Processing (CEP) has become one of the most rapidly emerging fields in data processing. e-Science use-cases, business applications, financial trading applications, operational analytics applications and business activity monitoring applications are some use-cases that directly use CEP. This paper discusses different design decisions associated with CEP Engines, and proposes some approaches to improve CEP performance by using more stream processing style pipelines. Furthermore, the paper will discuss Siddhi, a CEP Engine that implements those suggestions. We present a performance study that exhibits that the resulting CEP Engine--Siddhi--has significantly improved performance. Primary contributions of this paper are performing a critical analysis of the CEP Engine design and identifying suggestions for improvements, implementing those improvements through Siddhi, and demonstrating the soundness of those suggestions through empirical evidence. Show less

This paper is based on developing mathematical models to describe molecular geometries of graphene and carbon nanotubes, examining their geometry dependent electronic band structures and visually representing those relations in a 3D modelling and simulation software. Paper describes equations to derive the coordinates of the atoms, algorithms to derive the orientation of the bonds of the molecular structures, implementation of those algorithms in java3D and derivation of electronic band… Show more

This paper is based on developing mathematical models to describe molecular geometries of graphene and carbon nanotubes, examining their geometry dependent electronic band structures and visually representing those relations in a 3D modelling and simulation software. Paper describes equations to derive the coordinates of the atoms, algorithms to derive the orientation of the bonds of the molecular structures, implementation of those algorithms in java3D and derivation of electronic band structures. Successful results were obtained after the implementation of the algorithms which are illustrated with few screenshots at the end. Show less

Thanks to the NoSQL movement, data storage solutions are no longer a solved problem. Many are working hard to build new storage solutions, and even more are willing to use them. On the flip side, if you are a programmer or a solution architect who wants a data storage solution for your application, you have to face the daunting task of weighing and understanding the tradeoffs associated with the application and make a decision. This article explores the data needs of end user applications and… Show more

Thanks to the NoSQL movement, data storage solutions are no longer a solved problem. Many are working hard to build new storage solutions, and even more are willing to use them. On the flip side, if you are a programmer or a solution architect who wants a data storage solution for your application, you have to face the daunting task of weighing and understanding the tradeoffs associated with the application and make a decision. This article explores the data needs of end user applications and various tradeoffs. It provides guidelines on the criteria for selecting data storage choices enabling the architects and developers to make an informed decision.
Show less

Enterprise IT infrastructure incurs many
costs ranging from hardware costs and software
licenses/maintenance costs to the costs of monitoring,
managing, and maintaining IT infrastructure. The recent
advent of cloud computing offers some tangible prospects of
reducing some of those costs; however, abstractions provided
by cloud computing are often inadequate to provide major
cost savings across the IT infrastructure life-cycle.
Multi-tenancy, which allows a single… Show more

Enterprise IT infrastructure incurs many
costs ranging from hardware costs and software
licenses/maintenance costs to the costs of monitoring,
managing, and maintaining IT infrastructure. The recent
advent of cloud computing offers some tangible prospects of
reducing some of those costs; however, abstractions provided
by cloud computing are often inadequate to provide major
cost savings across the IT infrastructure life-cycle.
Multi-tenancy, which allows a single application to emulate multiple application instances, has been proposed as a
solution to this problem. By sharing one application across
many tenants, multi-tenancy attempts to replace many small
application instances with one or few large instances thus
bringing down the overall cost of IT infrastructure. In this
paper, we present an architecture for achieving multi-tenancy
at the SOA level, which enables users to run their services and
other SOA artifacts in a multi-tenant SOA framework as well
as provides an environment to build multi-tenant applications.
We discuss architecture, design decisions, and problems
encountered, together with potential solutions when applicable.
Primary contributions of this paper are motivating multitenancy, and the design and implementation of a multitenant SOA platform which allows users to run their current
applications in a multi-tenant environment with minimal or no
modifications Show less

Geronimo, the Apache Software Foundation's open source Java™ 2 Platform, Enterprise Edition (J2EE) server, integrates dozens of technologies and concepts. Most of them are projects in their own right, each with its own configuration and deployment model. In Geronimo's quest to bring these heterogeneous systems under one umbrella, it boils down the various approaches to configuration and deployment into a single, homogeneous model. Geronimo developer Srinath Perera explains how Geronimo achieves… Show more

Geronimo, the Apache Software Foundation's open source Java™ 2 Platform, Enterprise Edition (J2EE) server, integrates dozens of technologies and concepts. Most of them are projects in their own right, each with its own configuration and deployment model. In Geronimo's quest to bring these heterogeneous systems under one umbrella, it boils down the various approaches to configuration and deployment into a single, homogeneous model. Geronimo developer Srinath Perera explains how Geronimo achieves this objective and links what Geronimo users see to concepts that underlie the server's architecture. Show less