NoSQL or Know SQL ?

I have spent the last few months exploring the Big Data landscape, discussing some of the technologies that are underpin the promise of Big Data like Hadoop and Map Reduce.

NoSQL is a set of capabilities that have emerged over the last few years, motivated by the fact that relational databases technologies are limited by the need to model an application up front, define a schema and operate around the schema. Proponents of NoSQL believe that this presents a significant limitation for some emerging classes of applications, particularly where the schema definition can not be determined in advance. It is important to note that NoSQL is a movement, as opposed to a specific technology. What motivates proponents of this technology is the need for SQL schema flexibility. The actual implementation varies, with proponents adopting various approaches, e.g. key store, document store, in-memory or graph oriented databases etc. 

Most of the clients I have interacted with understand the value of relational databases and the merits of being able to model applications and design around a specific scheme. This is how relational technologies are able to deliver scale, robustness, and transaction guarantees also referred to as the ACID properties. The dilemma clients have is how to retain the goodness of relational databases, while adopting elements of the NoSQL promise that has the potential to improve their overall effectiveness. This is where the notion of "Know SQL" comes in. In my view, it is not about the irrelevance of relational SQL based database technologies, but rather about knowing when more schema flexibility is desirable, particularly in the Big Data world.

Recognising this trend, IBM has been investing its relational database technologies, adding XML support into DB2 to deliver more schema flexibility, and introducing NoSQL Graph Support, also known as DB2-RDF. As a result, clients are able to achieve schema flexibility, without having to give up the proven capabilities and robustness of relational database engines. More information on DB2 NoSQL support can be found here.

Part 3 : High Availability Enterprise Hadoop Clusters

Hadoop is primarily made up of 2 elements 1) The distributed Hadoop Filesystem (HDFS) and a programming paradigm, MapReduce.

Redundancy is built into Hadoop clusters. Data is redundantly stored in multiple places across the cluster, with portions of the functions running on various servers in the cluster. Hadoop is designed with the expectation that node failures will occur. Hadoop is fault tolerant. If a failure occurs, it automatically heals itself by nominating another node in the cluster to perform the work destined for the failed node.

That said, for enterprise deployment, a hot standby for the NameNode and the JobTracker server in the MapReduce processing layer is required. This was achieved in Hadoop 1.0 by having an active / passive failover solution, where data is replicated across two separate Hadoop clusters. An alternative solution is to have a dedicated backup for the master node, which includes the NameNode and can also include the JobTracker service. Should the NameNode fail, the Hadoop cluster can restart using the backup NameNode.

IBM committers have been working on Hadoop 2.0 with the Hadoop open source community to address the single point of failure challenges. In Hadoop 2.0, it is possible to designate a hot standby for the HDFS NameNode, and MapReduce2 has been enhanced to eliminate the potential single point of failure in the JobTracker by distributing its functionality across nodes in the cluster.

Part 2 : What is IBM's Enterprise Hadoop ?

In a previous blog post, I introduced Hadoop at a very high level. 

Hadoop is clearly a technology that can be used to overcome the volume and variety challenges of data at rest. Traditional analytics tools work very well on structured data, but today, only 20% of the data is structured. The need for an alternative solution is clear.

While many organisations have been experimenting with Hadoop to bridge this gap, few have been able to leverage it to gain insights that creates meaningful value. This is partly due to the deep skills required to work with Hadoop, and also the lack of enterprise capabilities that address information governance concerns, security and data lifecycle management.

IBM introduced Infosphere Big Insights, its Enterprise Hadoop platform to enable organisations to leverage Hadoop in a way that could have a transformational impact on their business and deliver significant competitive advantage. To achieve this, IBM introduced focussed on delivering 3 key capabilities. 

1. Analytics support to enhance consumability, enabling analysts with minimal programming skills to get value out of big Data. 
2. Enhancing the Hadoop platform so that it integrates with the rest of the enterprise, enabling Analysts to query Big Data from their existing Data Warehouse and vice versa.
3. Delivering additional tools and capabilities that address governance, security, administration and performance concerns inhibiting the enterprise adoption of Big Data 

For more information on IBM's Infosphere Big Insights, visit this link.

Part 1 : What is Hadoop ?

Imagine networking thousands of computers together. Each computer has its own processor and hard disk drive. All these computers are running software that makes the computers appear as a single powerful "super computer" with lots of processing capability and storage space. 

Now assume you are Amazon, and using this "super computer" to store click stream records from your amazon.com web site. You now want to understand if there are trends that lead to customers not completing transactions after having added items into their shopping carts. Recall that the click stream data is spread across the local disk drives on all of these thousands of computers.

To gain the insights required, a copy of your Analytics (application) logic is sent to each individual computer. Each computer  then runs the application logic against data stored locally. Instead of bring data to the application, the application (or function) is moved to the location where data is stored. Moving data across a network has a significant impact on performance, and by avoiding this, near linear scalability is achieved. Increasing data processing requirements can be accommodated simply by adding more computers.

Hadoop is the term used to describe this distributed filesystem(HDFS) and data processing engine that can be used to handle extremely high volumes of unstructured data at Internet scale. These group of computers make up a Hadoop cluster. Each computer in a Hadoop cluster is referred to as a node. The programming model used to bring the function (application logic) to the data is known as Map Reduce.

Technologies like Hadoop is what enables companies like Facebook, Google and Yahoo! to store millions of digital images and elements of our conversations, without having to design or understand up front, the format of the information or content they need to handle. This flexibility and ability to scale in a near linear fashion is one of the key attraction of Hadoop. Yahoo! reportedly has over 40,000 nodes spanning its Hadoop clusters which store over 40PB of data.

What's IDC's prediction for 2013?

IDC recently published their 2013 predictions, with Mobile, Social, Cloud and Big Data platforms expected to  drive 90% growth in the IT market between 2013 and 2020. I summarized the key 2013 trends are as follows:

• IT spending will exceed $2.1 trillion, driven by the adoption of smart mobile devices
• IT spending in the growth markets will represent 34% of worldwide spend, and 50% of all new growth in the IT marketplace
• Package applications providers like IBM, Microsoft and Oracle will become major Software as a Service (SaaS) providers
• There will be an explosion in Platform as a Service (PaaS) offerings tailored for specific industry applications
• Converged systems will transition from hype to market reality, as enterprise datacenter and cloud-provider use cases converge
• Line of Business executives will drive the increasing adoption of industry solutions, with 60% of all new IT spend influenced by LOB executives
• Enterprises will transition from Social Network experimentation to integration
• Big Data investments investments will continue to grow, with a shift of focus to analytics and discovery tools, and analytic applications

What's a Field Programmable Gateway Array?

While conducting a deep dive into IBM's PureData for Analytics system, it became obvious that Field Programmable Gateway Array played a key part in its performance. Little did I know that this technology is used in many every day electronic devices, e.g. DVDs make use of it to facilitate reads of high quality compressed digital data off spinning discs without jitters.

The device itself is a semi conductor chip equipped with a large number of internal programmable gates. When programmed, it acts as a specialized hardware for specialized tasks requiring high performance.

In PureData for Analytics, data is delivered from disk to memory as quickly as it can be streamed off disk, compressed and cached in memory by the FPGA using a smart algorithm which ensure that frequently used data is served out of memory. The embedded engines in the FPGA can be dynamically modified and extended programmatically, and act on streaming data at extremely high speed. In addition to compressing the data using semiconductor based technology, the FPGA filters out unnecessary columns and rows to boost performance.

My first "in depth" look at the PureData for Analytics System

Hosts

The primary interface to the PureData for Analytics system are high performance Linux hosts. External tools and applications, e.g. reporting, backup and recovery etc, interact with the host via standardized interfaces, e.g. JDBC etc. The host compiles SQL queries into executable code snippets, creates optimized query plans and distributes the snippets to massive parallel processing nodes for execution. The host is in an active-passive high availability cluster configuration, mirroring data to the standby hosts which monitors the primary host and takes over in case of a failure.

S-blades

The bulk of the analytics workload processing occurs on intelligent massively parallel processing nodes called S-blades. S-blades are optimzed for processing analytics workloads at massive scale. They contain multi-core CPU, multiengine Field-Programmable Gate Architectures, and gigabytes of RAM, all optimized to work together to deliver peak performance. Continuous availability is made possible my the systems management software, which monitors the s-blades (including memory), and automatically takes a failed S-blade out of service and moves the processing load to a spare one.

Disks

The S-blades are connected to disk enclosures via a high-speed interconnect that enables streaming of data to the S-blade memory at the fastest rate possible. The disk enclosures contain high density, high performance disks. Redundancy is built into the data path from each S-blade to the disks. Each drive is mirrored in a RAID 1 configuration, and should a disk fail, the storage subsystem simply redirects I/O processing to the mirror without interruption of service. Spare drives are included, allowing the system to replace failed drives and regenerate content for full redundancy.

Network
The communication in the MPP grid occurs on an optimized IP based network designed for high volume data warehousing traffic patterns. It allows maximum utilization of the network bandwidth without overloading it, thereby allowing predictable performance close to the data transmission speed of the network. There are 2 completely independent networks for redundancy. The data network is also completely separate from the management network. This enables the system to assess the health of its components even where there might be data network problems.

Optimizing Database Warehousing Operations

The biggest challenge or bottleneck across date warehousing operations is the speed at which the database engine can read from and write to disk. This is commonly known as disk I/O bottleneck.

Most efficient analytics platforms minimize data movement, and are able to process streaming data from disk to memory in parallel, and on a massive scale. In the case of IBM's Netezza, this is accomplished by using innovative hardware acceleration. It uses Field Programmable Gate Arrays (FPGA) to filter extraneous data as early in the stream and as fast as the data can be streamed off disk.

By eliminating data that is not required close to the data source, downstream components like CPU, memory and network do not have to deal superfluous data, significantly reducing I/O bottlenecks and improving system performance.

Key Elements of a Big Data Strategy

1) Visualisation and Discovery is an important capability. Organizations need to understand the scope and content of their data sources. Federated search, discovery and navigation tools enables access to information, irrespective of where it is located or format, restricting access to those with the authority to view the content. The ability to enrich the content, e.g. by adding comments, rating, tagging can be used to add a social dimension to how data is presented to the end user. The ability to quickly examine, explore and discover data relationships is can create a competitive advantage.

2) Hadoop enables organisations to reduce the cost of their data management infrastructure by offloading structured and unstructured data not suitable for traditional data warehouse for deep cost effective deep analytics.

3) What can you do when analyzing stored data is not good enough? Stream computing enables analysis of realtime streaming data of multiple formats; prefiltering (using complex calculations) and selective storing of high velocity data in realtime.

4) What can be done to accelerate the adoption of data warehousing and analytics capabilities that can offer a competitive advantage? As IBM has demonstrated with it's Netezza technology, there is a competitive advantage to be derived from purpose built systems for complex analytics workloads. These Expert Integrated Systems are designed with simplicity in mind, minimal administration required, and ability to perform complex analytics on large volumes of structured data at blistering speeds. Additionally, expertise can be built into this systems by packing pre-built analytics and visualizations applicable to specific industry applications. This library of pre-built functions, and tools that enable custom functions to be built, accelerate the time to value for analysis of data in native formats, where it lives.

5) Finally Security and Governance are key aspects of Big Data management. Sensitive data needs to be protected, retention policies need enforcing and data quality governed. Information lifecycle and master data management, along with data quality and governance services, are very important considerations in operating a Big Data platform.

How can organizations boost their Big Data IQ?

1. Adopt Big Data platforms that enable data discovery and exploration. A Big Data platform should enable users to understand the variety of data, their sources, along with the quality and relationships to other data elements, while data is in place. This process enables data scientist to create the right analytic model and computational strategy.
2. Run analytics against the whole population of data, irrespective of the variety, structured or unstructured. The key point here is that organizational process do not make a distinction between underlying data structures.
3. Gain competitive advantage by running analytics faster than others. To do this, it might be necessary to run analytics on the same platform as data processing.
4. The ability to run analytics in realtime, enables organizations to react faster to change and gain competitive advantage.
5. Taking point 1) further, it is useful to have access a rich library of analytical functions and tool set. This will enable organizations to analyze and publish results much faster.
6. Tools that enable organizations to establish trust in data are critical for success. These tools should be able to address policy, security, governance, integration and lifecycle management requirements.

Examples of Big Data Use Cases

IT for IT: e.g. using log analytics to gain better insights into how IT systems are running, when and how they might breakdown.

Social Media Analytics: e.g. using Big Data to figure out what customers are saying about your brand (or your competitor's brand) and why they are saying it.

Customer Sentiments: e.g. minimizing customer churn by detecting loyalty decay and suggest the next best action before the contact is made with the customer.

Fraud Detection: e.g. using Big Data to detect cyclical fraud patterns

Using Jigsaw Puzzles to explain Big Data

In Paul Zikopoulos et al's book titled Harness the Power of Big Data, puzzles are used to explain how Big Data is used to solve problems. I found it very effective and have summarized below, what I consider to be the key points.

Big Data is like a puzzle that has been taken out of its box. Analytics is required to complete the picture. 

As you work on the puzzle and frame its edges and the puzzle is taking shape, you have a lot more context and understanding. Although the data has not changed (same number of puzzles), there is now metadata (relationships and patterns) to manage. So, as the puzzle gets resolved, we have more data associated with it, but the problem is a lot easier to solve. Even if someone threw in puzzle pieces that did not belong to this set, it would be easier to identify the foreign pieces due to the knowledge we have gained about the pattern of the puzzle and relationship.

How can we solve this puzzle more effectively?

We could split the work among a number of people (processes), asking each to perform a complementary, but discrete task, e.g. finding all pieces that have an element of a face. This is how Big Data platforms are able to scale out work, instead of relying on a limited number of processors. Because we can't anticipate how many pieces will make up the Big Data puzzle, scale out, machine learning and massive parallel processing capabilities of analytics systems are needed to frame the edge, sort, group the pieces and discover the patterns.

How is this different from data in a warehouse ?

Assume for a moment that all boarder shapes came in a separate bag, all pieces of a certain color in their own bag etc. This is how data in a warehouse might look like, and it can be analyzed with traditional reporting techniques, which focusses on data that is organized and of a known structure. 

When does a Big Data solution make sense?

Below are some thoughts on when a Big Data solution might make sense

1) Current platform capabilities are insufficient to process amount of data needed

2) New data structure is inconstant with the schema of existing analytics datastore, resulting in a mix of information types

3) Data transmission rates are too high for analytics platform or there is a need to ingest data quickly without knowing its schema ahead of time

4) Need to move from analysis of short history to several years of data

Data Warehouse in a Big Data World. What is the use case?

I like the analogy in the IBM Big Data Platform book titled "Harness the Power of Big Data".

It tells the story of the days long gone, when miners could easily spot nuggets or veins of gold with the naked eye. This made investing easier, as its value could be seen and therefore the resources required to extract it considered against its perceived value. Using a Big Data analogy, we can consider this gold to be "high value-per-byte of data".

Assume for a moment that there is more gold nearby, but it is just no visible to the naked eye. Trying to find this gold is a bit more of a gamble and potentially more expensive. This would be "low value-per-byte of data" due to the challenges associated with finding gold not visible to the naked eye.

With the right equipment however, it might be possible to economically process lots of dirt and keep the flakes of gold found. This flakes can be taken for processing and combined to make flakes of gold.

Back to our Big Data analogy...

In this scenario, it would make sense to keep all the dirt we could find (in a Big Data System), so that as new, economical dirt processing techniques emerge (Big Data Analytics on commodity systems) we would have an opportunity to extract the flakes of gold (value / insights) and store it for processing into gold bars (in our Data Warehousing system).

Hadoop is a Big Data batch system that allows users to store all data in its native business object format and get value out of it through massive parallel processing on commodity components.

Data Warehouse is characterized by "speed-of-thought response times" requirements where sustainable data with  proven value stored and delivered interactively.

It is therefore clear to see that in a Big Data world, there is value and a place for both Hadoop (Big Data) and Data Warehouse systems.

IBM's Hadoop system is Infosphere Big Insights. For simplified Big Data Analytics, look no further than IBM PureData for Analytics powered by Netezza, and Infosphere Warehouse for your Data Warehousing needs.

TerraEchos..."the next generation big-data analytics company"

I came across TerraEchos recently, an IBM business partner with a set of capabilities that illustrates the potential Big Data Analytics provides.

TerraEchos describe themselves as a company that extract meaningful information from massive amounts of complex streaming data on the fly, and simultaneously deliver insights, decisions and actions on the fly - at the precise moment they are needed. This in my view is the promise of Big Data Analytics.

As you can see from the diagram to the right, their Streaming Analytics capability requires significantly less time to analyze data.

Some of the Big Data Analytics capabilities the  TerraEcho platform exhibits includes the ability to analyze data irrespective of the amount, speed, or source of digital data, including input from any kind of cyber or physical sensor, in both structured and unstructured form. It is being positioned as suitable for organization that requires the processing, analysis, and visualization of multiple or complex streaming data sources.

One use case that caught my attention was a sophisticated sound classification system that can be used for real-time perimeter security control.  Thousands of sensors buried underground can be used to collect and classify sounds. The system can differentiate between a whisper of the wind and a human voice, or the sound of a human footstep from a running deer. If can even identify or affirm sounds that are difficult for humans to pick up.

TerraEcho has partnered with IBM to deliver these capabilities. IBM's PureData platform has been designed to simplify systems for delivering data services, making the deployment and analysis of Big Data more accessible. 

What are some of the trends creating opportunities for Big Data enriched analytics?

As I seek to understand Big Data and what it means to me as an IT practitioner, and to my clients as a consultant, I have found it useful to identify some of the trends that have underpin this opportunity

1. The number of RFID tags used in supply chain, tracking conference attendees, tracking luggage at airports, monitoring temperature of food, structures etc has increased from about 1.3 billion in 2005 to over 30 billion by end of 2011. Prices are predicted to drop below 1US cent making it possible to instrument event more systems.
2. A flight from London to New York generates about 650 TB of data which could be proactively analyzed to gain new insights that could lead to improvements in safety and other efficiencies.
3. Capturing every user's online clickstream would generate TBs of data that can be used to analyze and optimize the shopping experience. 
4. Data generated from smart meters can be used to better understand customer behavour, align supply better to demand, and enable customers to make more informed decisions about their energy usage patterns.
5. Take Facebook. The ability to analyze the whole data population, taking into account intents and sentiments can offer tremendous value. Doing this is not without its challenges. Facebook for example, experiences over 2.5 billion likes and more than 300 million photo uploads each day.
6. ..and Twitter. Twitter's 140 character or less design allows users to provide precise commentaries on a variety of subjects. The value to be derived from analyzing this data for sentiments and intents is significant.
7. What about location based services (LBS)? Apparently, the average commuter in London has their photo taken about 150 times as they travel to work. Most of the mobile devices we carry with us have LBS enabled. This information can be used to further personalize interactions. 

So, there is a lot of data being generated, and this will increase over time. Most of the data is not analyzed at all. Imagine being able to not only analyze data at rest, but also data in motion, as it hits the enterprise. He in lies the tremendous opportunities of Big Data Analytics. The PureData System from IBM simplified today's data requirements and enables clients to develop capabilities that enable them to gain insights that create a competitive advantage.

Big Data...What is it ?

Without a doubt, Big Data is considered one of the most important IT trends of recent time, and many would say a business imperative. But what really is it?

In this article, I summarize insights from a text I have been reading, Harness the Power of Big Data.

1. Big Data has nothing to do with the size of data, but rather, the ability to perform analytics on a broader spectrum of data, and gain a competitive advantage from the insights gained
2. The ability to instrument and capture data, not only data that is stored (at rest) but also data in motion (data being generated in realtime), an perform analytics on the whole population of this data, is a key requirement. 
3. Big Data is typically defined using the 3 Vs; volume, variety and velocity. IBM added an additional V, veracity.
1. Volume; Data growth between 2009 and 2011 is estimated at 80%. Six years from now, data is estimated to be around 35 Zeta Bytes, equivalent to about 4 trillion 8GB iPods.
2. Variety; This relates to the need to capture all data that could be useful for the decision making process, structured or otherwise.
3. Velocity; This relates to the ability to analyze, process and gain insights from data as it becomes available.
4. Veracity; Big Data can contain a lot of inaccurate / untrustworthy data. Veracity relates the process of transforming Big Data into trustworthy data and discarding the noise.