Why is Open Source fidelity important ?

IT vendors who maintain fidelity to open source and ensure their products are 100% compatible with the open source packages they rely on, can realize some tangible benefits.

Fidelity to open source makes it easier to integrate with other products designed to be compatible with a common set of open source packages. This approach has enabled IBM to amass 100s of partners, accelerating the adoption of products like IBM's Big Insights which builds on the open source Hadoop distribution. Clients and software vendors using open source libraries and interfaces for Hadoop can rest assured that their applications will run with Big Insights.

This approach of embracing and maintaining the integrity of open source projects enables vendors to adopt open source enhancements and innovate on top for competitive differentiation. There is an opportunity for software vendors to test and certify that each component works well together in a manner suitable for enterprise production deployment. Additionally, vendors can address backward compatibility issues prevalent in some open source projects by undergoing rigorous regression testing.

As with many other open source projects, the IBM Big Insights development team has committers to the Hadoop project and its related projects. These committers contribute to open source innovation and ensures that IBM plays a role in shaping Hadoop and is aligned to its future evolution. 

Part 3 : High Availability Enterprise Hadoop Clusters

Hadoop is primarily made up of 2 elements 1) The distributed Hadoop Filesystem (HDFS) and a programming paradigm, MapReduce.

Redundancy is built into Hadoop clusters. Data is redundantly stored in multiple places across the cluster, with portions of the functions running on various servers in the cluster. Hadoop is designed with the expectation that node failures will occur. Hadoop is fault tolerant. If a failure occurs, it automatically heals itself by nominating another node in the cluster to perform the work destined for the failed node.

That said, for enterprise deployment, a hot standby for the NameNode and the JobTracker server in the MapReduce processing layer is required. This was achieved in Hadoop 1.0 by having an active / passive failover solution, where data is replicated across two separate Hadoop clusters. An alternative solution is to have a dedicated backup for the master node, which includes the NameNode and can also include the JobTracker service. Should the NameNode fail, the Hadoop cluster can restart using the backup NameNode.

IBM committers have been working on Hadoop 2.0 with the Hadoop open source community to address the single point of failure challenges. In Hadoop 2.0, it is possible to designate a hot standby for the HDFS NameNode, and MapReduce2 has been enhanced to eliminate the potential single point of failure in the JobTracker by distributing its functionality across nodes in the cluster.

Part 2 : What is IBM's Enterprise Hadoop ?

In a previous blog post, I introduced Hadoop at a very high level. 

Hadoop is clearly a technology that can be used to overcome the volume and variety challenges of data at rest. Traditional analytics tools work very well on structured data, but today, only 20% of the data is structured. The need for an alternative solution is clear.

While many organisations have been experimenting with Hadoop to bridge this gap, few have been able to leverage it to gain insights that creates meaningful value. This is partly due to the deep skills required to work with Hadoop, and also the lack of enterprise capabilities that address information governance concerns, security and data lifecycle management.

IBM introduced Infosphere Big Insights, its Enterprise Hadoop platform to enable organisations to leverage Hadoop in a way that could have a transformational impact on their business and deliver significant competitive advantage. To achieve this, IBM introduced focussed on delivering 3 key capabilities. 

1. Analytics support to enhance consumability, enabling analysts with minimal programming skills to get value out of big Data. 
2. Enhancing the Hadoop platform so that it integrates with the rest of the enterprise, enabling Analysts to query Big Data from their existing Data Warehouse and vice versa.
3. Delivering additional tools and capabilities that address governance, security, administration and performance concerns inhibiting the enterprise adoption of Big Data 

For more information on IBM's Infosphere Big Insights, visit this link.

Part 1 : What is Hadoop ?

Imagine networking thousands of computers together. Each computer has its own processor and hard disk drive. All these computers are running software that makes the computers appear as a single powerful "super computer" with lots of processing capability and storage space. 

Now assume you are Amazon, and using this "super computer" to store click stream records from your amazon.com web site. You now want to understand if there are trends that lead to customers not completing transactions after having added items into their shopping carts. Recall that the click stream data is spread across the local disk drives on all of these thousands of computers.

To gain the insights required, a copy of your Analytics (application) logic is sent to each individual computer. Each computer  then runs the application logic against data stored locally. Instead of bring data to the application, the application (or function) is moved to the location where data is stored. Moving data across a network has a significant impact on performance, and by avoiding this, near linear scalability is achieved. Increasing data processing requirements can be accommodated simply by adding more computers.

Hadoop is the term used to describe this distributed filesystem(HDFS) and data processing engine that can be used to handle extremely high volumes of unstructured data at Internet scale. These group of computers make up a Hadoop cluster. Each computer in a Hadoop cluster is referred to as a node. The programming model used to bring the function (application logic) to the data is known as Map Reduce.

Technologies like Hadoop is what enables companies like Facebook, Google and Yahoo! to store millions of digital images and elements of our conversations, without having to design or understand up front, the format of the information or content they need to handle. This flexibility and ability to scale in a near linear fashion is one of the key attraction of Hadoop. Yahoo! reportedly has over 40,000 nodes spanning its Hadoop clusters which store over 40PB of data.