Part 2: PureSystems Analyst Roundtable Discussion

A key discussion topic for this week's Analysts roundtable is the PureData System for Transactions. 

PureData System for Transactions has been designed and built for clients with the highest demands for performance and availability. To many of our clients, such systems are core to their business and any downtime can result in loss of revenue, customer churn, tarnished image, etc.). PureData System is a  "Tier 1" system, designed for such mission critical applications.  

At the heart of PureData System for Transactions is DB2 pureScale, which provides extreme levels of availability and scalability, with application transparency. PureData System for Transactions inherits these capabilities to deliver high levels of reliability, performance and scalability out of the box. Existing DB2 applications are supported with no changes, and Oracle Database applications are also supported with minimal, and in some cases, no changes are necessary.

One of the most striking features of the IBM PureSystems family is the consistent, single web user interface through which all hardware and software components are managed and monitored. This significantly reduces the additional knowledge required to operate any offering within the PureSystems family, once a user is familiar with one of the offerings. Think about the effort required to transition from an iPod to an iPhone. Once a system is delivered, the PureData System for Transaction adoption process is as simple as

1. Power-on
2. Integrate system to the network
3. Create and deploy database clusters
4. Configure access and allocate storage to databases
5. Upload objects and data into the created databases

What IT systems does a Smart City need?

A McKinsey Global Institute article "Urban world: Cities and the rise of the consuming class", reflects on the speed and unprecedented scale of city expansion. This is placing an increasing demand for new environmentally friendly infrastructure for water and energy, buildings, transportation and communication. Migration is motivated mostly by the search for a better life, and in additional to the infrastructure challenges previously outlined, this also places a strain on the cities to deliver citizen based services such as education, health, public safety, economic development and social programs.

Smarter city operations rely on the ability to capture data that can be used to anticipate and proactively resolve problems. Problem resolution can be achieved by coordinating processes and resources for more efficient operations. Take public safety as an example. IT systems can be deployed to predict, monitor and mitigate crisis situations. This can be achieved by automatically analyzing video streams for threats based on known criminal patterns. Similar capabilities can be applied to transportation for more effective traffic management, and for analyzing water use and consumption patterns, thereby enabling utilities to identify leakages and optimize repair jobs for improved service delivery.


The figure to the left is from  "Competing on Analytics, Davenport and Harris, 2007". 

To be competitive, Smart Cities need to deploy Analytics capabilities that enable the delivery of a high quality of services that meets and exceeds citizen expectations today, and  can accommodate future needs for real-time dynamic access to innovative new services. 

Additionally, these new services need to be resilient, secure, compliant with local requirements, and sufficiently agile to address new risks posed by an ever more connected and collaborative world. And all of this needs to be achieved cost effectively. IBM's PureData System for Analytics is designed specifically for this use case.

Imagine a situation where city operations can be collaboratively managed via Executive, City Operations and Agency dashboards that include domain key performance indicators for standard operating procedures. Such a system could support centralized planning, execution and monitoring for more efficient operations.  

IBM's Intelligent Operations Center delivered on a Cloud enabled PureSystems platform, enables cities to not only contain operational cost and complexity, but achieve breakthrough productivity gains through rapid time to value via virtualization, optimization, energy stewardship.

 

PureData for Analytics Value Proposition

The PureData for Analytics offering, a member of the PureSystems family of Expert Integrated Systems offers a simple-to-use approach for serious Analytics on structured data.

Analytics activities such as data exploration, discovery, transformation, model building and scoring can be performed where the data resides, in the warehouse.

This reduces the time it takes to build and deploy Analytics models, accelerating fact-based decision based on insightful Analytics.

Practitioners can also experiment iteratively with different models, operationalizing and making advanced Analytics more accessible.

What's a Field Programmable Gateway Array?

While conducting a deep dive into IBM's PureData for Analytics system, it became obvious that Field Programmable Gateway Array played a key part in its performance. Little did I know that this technology is used in many every day electronic devices, e.g. DVDs make use of it to facilitate reads of high quality compressed digital data off spinning discs without jitters.

The device itself is a semi conductor chip equipped with a large number of internal programmable gates. When programmed, it acts as a specialized hardware for specialized tasks requiring high performance.

In PureData for Analytics, data is delivered from disk to memory as quickly as it can be streamed off disk, compressed and cached in memory by the FPGA using a smart algorithm which ensure that frequently used data is served out of memory. The embedded engines in the FPGA can be dynamically modified and extended programmatically, and act on streaming data at extremely high speed. In addition to compressing the data using semiconductor based technology, the FPGA filters out unnecessary columns and rows to boost performance.

My first "in depth" look at the PureData for Analytics System

Hosts

The primary interface to the PureData for Analytics system are high performance Linux hosts. External tools and applications, e.g. reporting, backup and recovery etc, interact with the host via standardized interfaces, e.g. JDBC etc. The host compiles SQL queries into executable code snippets, creates optimized query plans and distributes the snippets to massive parallel processing nodes for execution. The host is in an active-passive high availability cluster configuration, mirroring data to the standby hosts which monitors the primary host and takes over in case of a failure.

S-blades

The bulk of the analytics workload processing occurs on intelligent massively parallel processing nodes called S-blades. S-blades are optimzed for processing analytics workloads at massive scale. They contain multi-core CPU, multiengine Field-Programmable Gate Architectures, and gigabytes of RAM, all optimized to work together to deliver peak performance. Continuous availability is made possible my the systems management software, which monitors the s-blades (including memory), and automatically takes a failed S-blade out of service and moves the processing load to a spare one.

Disks

The S-blades are connected to disk enclosures via a high-speed interconnect that enables streaming of data to the S-blade memory at the fastest rate possible. The disk enclosures contain high density, high performance disks. Redundancy is built into the data path from each S-blade to the disks. Each drive is mirrored in a RAID 1 configuration, and should a disk fail, the storage subsystem simply redirects I/O processing to the mirror without interruption of service. Spare drives are included, allowing the system to replace failed drives and regenerate content for full redundancy.

Network
The communication in the MPP grid occurs on an optimized IP based network designed for high volume data warehousing traffic patterns. It allows maximum utilization of the network bandwidth without overloading it, thereby allowing predictable performance close to the data transmission speed of the network. There are 2 completely independent networks for redundancy. The data network is also completely separate from the management network. This enables the system to assess the health of its components even where there might be data network problems.

Optimizing Database Warehousing Operations

The biggest challenge or bottleneck across date warehousing operations is the speed at which the database engine can read from and write to disk. This is commonly known as disk I/O bottleneck.

Most efficient analytics platforms minimize data movement, and are able to process streaming data from disk to memory in parallel, and on a massive scale. In the case of IBM's Netezza, this is accomplished by using innovative hardware acceleration. It uses Field Programmable Gate Arrays (FPGA) to filter extraneous data as early in the stream and as fast as the data can be streamed off disk.

By eliminating data that is not required close to the data source, downstream components like CPU, memory and network do not have to deal superfluous data, significantly reducing I/O bottlenecks and improving system performance.