What IT systems does a Smart City need?

A McKinsey Global Institute article "Urban world: Cities and the rise of the consuming class", reflects on the speed and unprecedented scale of city expansion. This is placing an increasing demand for new environmentally friendly infrastructure for water and energy, buildings, transportation and communication. Migration is motivated mostly by the search for a better life, and in additional to the infrastructure challenges previously outlined, this also places a strain on the cities to deliver citizen based services such as education, health, public safety, economic development and social programs.

Smarter city operations rely on the ability to capture data that can be used to anticipate and proactively resolve problems. Problem resolution can be achieved by coordinating processes and resources for more efficient operations. Take public safety as an example. IT systems can be deployed to predict, monitor and mitigate crisis situations. This can be achieved by automatically analyzing video streams for threats based on known criminal patterns. Similar capabilities can be applied to transportation for more effective traffic management, and for analyzing water use and consumption patterns, thereby enabling utilities to identify leakages and optimize repair jobs for improved service delivery.


The figure to the left is from  "Competing on Analytics, Davenport and Harris, 2007". 

To be competitive, Smart Cities need to deploy Analytics capabilities that enable the delivery of a high quality of services that meets and exceeds citizen expectations today, and  can accommodate future needs for real-time dynamic access to innovative new services. 

Additionally, these new services need to be resilient, secure, compliant with local requirements, and sufficiently agile to address new risks posed by an ever more connected and collaborative world. And all of this needs to be achieved cost effectively. IBM's PureData System for Analytics is designed specifically for this use case.

Imagine a situation where city operations can be collaboratively managed via Executive, City Operations and Agency dashboards that include domain key performance indicators for standard operating procedures. Such a system could support centralized planning, execution and monitoring for more efficient operations.  

IBM's Intelligent Operations Center delivered on a Cloud enabled PureSystems platform, enables cities to not only contain operational cost and complexity, but achieve breakthrough productivity gains through rapid time to value via virtualization, optimization, energy stewardship.

 

The Promise of Mobile Business

Until very recently, I was mostly oblivious to the Bring Your Own Device (BYOD) trend that appears to be getting more and more embedded in the fabric of enterprise IT. I now find myself owning a personal iPhone, Samsung Galaxy and a MacBook Air, all of which I use for work. In my case, the acquisition was largely driven by a desire to use what I believed would enabled me to be most productive. I had also become more and more frustrated with the length of time it took to boot up my laptop, let alone the weight. The MacBook Air was therefore a pretty compelling proposition. Email on my iPhone was more pleasing, and the Samsung Galaxy enabled me to create a mobile private Wifi hotspot using my personal 3G broadband sim card. All I need is an iPad Mini to complete the picture!

From a business point of view, these trends are having a  transformational impact, creating opportunities for new business models as user's expectations drive what IT needs to deliver. Think about your mobile experience for a minute. I find myself contemplating switching banks based on the user experience of the mobile banking application. And for enterprise IT, it is not simply a matter of getting the best application out of the door. Mobile applications have a much faster lifecycle, and are much more iterative than traditional applications. Enterprise IT therefore needs to adopt a much more agile application development and release lifecycle. And the challenges do not end there. They also have to cater for a fragmented landscape of devices and network providers.

That said, the opportunities that can be realized through mobile business are immense. In a recent tweet, I shared a link to an article on 19 Surprising Facts About Africa’s Mobile Market. The potential for leveraging the mobile platform for business is clear. What makes the mobile platform different and compelling is the flexibility it offers in terms of the multitude of applications available, the wide range of connectivity options, multimedia support, location awareness and other sensory capabilities.

While many recognize the opportunities available in the business to consumer space, the business-to-enterprise benefits are not very well understood. The mobile platform, when used effectively, can significantly enhance worker productivity, by enabling mobile workers to have access to the information and resources they need, wherever they are and when ever they need it. Throughout most of my IT career, I have been a mobile worker. The benefits to me personally and to my employer are obvious. I am able to work from home when family needs dictate without always having to take a day off. My colleagues are able to reach me even while I am traveling. The ability to work from home  makes me more productive. I do not have to incur the cost and time of traveling into the office every day. I reduce my carbon footprint, while my employer is able to reduce their personnel costs with schemes like Bring Your Own Device to work and by reducing office space to a bare minimum. In many instances, these benefits can be realized by simply extending existing applications to mobile workers and customers. I have personally witness many of my clients significantly improve customer satisfaction and increase revenue by leveraging the mobile platform to deepen their customer engagement. Developing a mobile strategy is clearly one of the ways organizations can differentiate themselves.

To many, context awareness is the new killer application. This relates to the ability to dynamically capture contextual information from mobile devices and use it to optimize and change communication and business flows. With context awareness, you can increase customer loyalty by delivering loyalty reward to a customer based on their unique needs at a given point in time. This concept can be extended to the enterprise with mobility-linked business processes. In this instance, business processes can be improved by capturing and automating process interactions where and when they occur.

Let's discuss the scenarios above using examples. In a customer loyalty use case, a retailer can offer an application for creating and managing shopping list. As a customer creates a list, their intended purchase could analyze against their historical shopping patterns, enabling the retailer to offer personalized shopping vouchers. The retailer's objective is to retain the shopper's business and capture a greater share of spend. The process of analyzing spending patterns and making an offer needs to be done at realtime. The PureData Systems for Analytics is designed to address petascale analytical workload requirements. In a business to business context, similar principles can be applied to deliver information to field workers just in time. In other words, consider an engineer working on an offshore oil rig. As they go about their job and input data into their mobile workpad, information can be analyzed in real time and the business process changed on the fly so  additional checks or maintenance is conducted. The same principles can be applied to other scenarios, e.g. optimizing supply chain and logistics planning, organizational learning to improving business outcomes etc.

Recognizing the opportunity mobile business presents, we have been working closely with some of our clients, leveraging the Worklight platform to enable them to create rich, yet cost-effective mobile apps in this fragmented technological landscape. A key piece of the capability we deliver is enterprise integration, enabling clients to connect to enterprise back-end services in a secure and scalable manner, while securely managing and controlling the growing portfolio of deployed enterprise applications.

Part 3 : High Availability Enterprise Hadoop Clusters

Hadoop is primarily made up of 2 elements 1) The distributed Hadoop Filesystem (HDFS) and a programming paradigm, MapReduce.

Redundancy is built into Hadoop clusters. Data is redundantly stored in multiple places across the cluster, with portions of the functions running on various servers in the cluster. Hadoop is designed with the expectation that node failures will occur. Hadoop is fault tolerant. If a failure occurs, it automatically heals itself by nominating another node in the cluster to perform the work destined for the failed node.

That said, for enterprise deployment, a hot standby for the NameNode and the JobTracker server in the MapReduce processing layer is required. This was achieved in Hadoop 1.0 by having an active / passive failover solution, where data is replicated across two separate Hadoop clusters. An alternative solution is to have a dedicated backup for the master node, which includes the NameNode and can also include the JobTracker service. Should the NameNode fail, the Hadoop cluster can restart using the backup NameNode.

IBM committers have been working on Hadoop 2.0 with the Hadoop open source community to address the single point of failure challenges. In Hadoop 2.0, it is possible to designate a hot standby for the HDFS NameNode, and MapReduce2 has been enhanced to eliminate the potential single point of failure in the JobTracker by distributing its functionality across nodes in the cluster.

Part 2 : What is IBM's Enterprise Hadoop ?

In a previous blog post, I introduced Hadoop at a very high level. 

Hadoop is clearly a technology that can be used to overcome the volume and variety challenges of data at rest. Traditional analytics tools work very well on structured data, but today, only 20% of the data is structured. The need for an alternative solution is clear.

While many organisations have been experimenting with Hadoop to bridge this gap, few have been able to leverage it to gain insights that creates meaningful value. This is partly due to the deep skills required to work with Hadoop, and also the lack of enterprise capabilities that address information governance concerns, security and data lifecycle management.

IBM introduced Infosphere Big Insights, its Enterprise Hadoop platform to enable organisations to leverage Hadoop in a way that could have a transformational impact on their business and deliver significant competitive advantage. To achieve this, IBM introduced focussed on delivering 3 key capabilities. 

1. Analytics support to enhance consumability, enabling analysts with minimal programming skills to get value out of big Data. 
2. Enhancing the Hadoop platform so that it integrates with the rest of the enterprise, enabling Analysts to query Big Data from their existing Data Warehouse and vice versa.
3. Delivering additional tools and capabilities that address governance, security, administration and performance concerns inhibiting the enterprise adoption of Big Data 

For more information on IBM's Infosphere Big Insights, visit this link.

Part 1 : What is Hadoop ?

Imagine networking thousands of computers together. Each computer has its own processor and hard disk drive. All these computers are running software that makes the computers appear as a single powerful "super computer" with lots of processing capability and storage space. 

Now assume you are Amazon, and using this "super computer" to store click stream records from your amazon.com web site. You now want to understand if there are trends that lead to customers not completing transactions after having added items into their shopping carts. Recall that the click stream data is spread across the local disk drives on all of these thousands of computers.

To gain the insights required, a copy of your Analytics (application) logic is sent to each individual computer. Each computer  then runs the application logic against data stored locally. Instead of bring data to the application, the application (or function) is moved to the location where data is stored. Moving data across a network has a significant impact on performance, and by avoiding this, near linear scalability is achieved. Increasing data processing requirements can be accommodated simply by adding more computers.

Hadoop is the term used to describe this distributed filesystem(HDFS) and data processing engine that can be used to handle extremely high volumes of unstructured data at Internet scale. These group of computers make up a Hadoop cluster. Each computer in a Hadoop cluster is referred to as a node. The programming model used to bring the function (application logic) to the data is known as Map Reduce.

Technologies like Hadoop is what enables companies like Facebook, Google and Yahoo! to store millions of digital images and elements of our conversations, without having to design or understand up front, the format of the information or content they need to handle. This flexibility and ability to scale in a near linear fashion is one of the key attraction of Hadoop. Yahoo! reportedly has over 40,000 nodes spanning its Hadoop clusters which store over 40PB of data.

PureData for Analytics Value Proposition

The PureData for Analytics offering, a member of the PureSystems family of Expert Integrated Systems offers a simple-to-use approach for serious Analytics on structured data.

Analytics activities such as data exploration, discovery, transformation, model building and scoring can be performed where the data resides, in the warehouse.

This reduces the time it takes to build and deploy Analytics models, accelerating fact-based decision based on insightful Analytics.

Practitioners can also experiment iteratively with different models, operationalizing and making advanced Analytics more accessible.

What's a Field Programmable Gateway Array?

While conducting a deep dive into IBM's PureData for Analytics system, it became obvious that Field Programmable Gateway Array played a key part in its performance. Little did I know that this technology is used in many every day electronic devices, e.g. DVDs make use of it to facilitate reads of high quality compressed digital data off spinning discs without jitters.

The device itself is a semi conductor chip equipped with a large number of internal programmable gates. When programmed, it acts as a specialized hardware for specialized tasks requiring high performance.

In PureData for Analytics, data is delivered from disk to memory as quickly as it can be streamed off disk, compressed and cached in memory by the FPGA using a smart algorithm which ensure that frequently used data is served out of memory. The embedded engines in the FPGA can be dynamically modified and extended programmatically, and act on streaming data at extremely high speed. In addition to compressing the data using semiconductor based technology, the FPGA filters out unnecessary columns and rows to boost performance.

Optimizing Database Warehousing Operations

The biggest challenge or bottleneck across date warehousing operations is the speed at which the database engine can read from and write to disk. This is commonly known as disk I/O bottleneck.

Most efficient analytics platforms minimize data movement, and are able to process streaming data from disk to memory in parallel, and on a massive scale. In the case of IBM's Netezza, this is accomplished by using innovative hardware acceleration. It uses Field Programmable Gate Arrays (FPGA) to filter extraneous data as early in the stream and as fast as the data can be streamed off disk.

By eliminating data that is not required close to the data source, downstream components like CPU, memory and network do not have to deal superfluous data, significantly reducing I/O bottlenecks and improving system performance.

Examples of Big Data Use Cases

IT for IT: e.g. using log analytics to gain better insights into how IT systems are running, when and how they might breakdown.

Social Media Analytics: e.g. using Big Data to figure out what customers are saying about your brand (or your competitor's brand) and why they are saying it.

Customer Sentiments: e.g. minimizing customer churn by detecting loyalty decay and suggest the next best action before the contact is made with the customer.

Fraud Detection: e.g. using Big Data to detect cyclical fraud patterns

Using Jigsaw Puzzles to explain Big Data

In Paul Zikopoulos et al's book titled Harness the Power of Big Data, puzzles are used to explain how Big Data is used to solve problems. I found it very effective and have summarized below, what I consider to be the key points.

Big Data is like a puzzle that has been taken out of its box. Analytics is required to complete the picture. 

As you work on the puzzle and frame its edges and the puzzle is taking shape, you have a lot more context and understanding. Although the data has not changed (same number of puzzles), there is now metadata (relationships and patterns) to manage. So, as the puzzle gets resolved, we have more data associated with it, but the problem is a lot easier to solve. Even if someone threw in puzzle pieces that did not belong to this set, it would be easier to identify the foreign pieces due to the knowledge we have gained about the pattern of the puzzle and relationship.

How can we solve this puzzle more effectively?

We could split the work among a number of people (processes), asking each to perform a complementary, but discrete task, e.g. finding all pieces that have an element of a face. This is how Big Data platforms are able to scale out work, instead of relying on a limited number of processors. Because we can't anticipate how many pieces will make up the Big Data puzzle, scale out, machine learning and massive parallel processing capabilities of analytics systems are needed to frame the edge, sort, group the pieces and discover the patterns.

How is this different from data in a warehouse ?

Assume for a moment that all boarder shapes came in a separate bag, all pieces of a certain color in their own bag etc. This is how data in a warehouse might look like, and it can be analyzed with traditional reporting techniques, which focusses on data that is organized and of a known structure. 

Data Warehouse in a Big Data World. What is the use case?

I like the analogy in the IBM Big Data Platform book titled "Harness the Power of Big Data".

It tells the story of the days long gone, when miners could easily spot nuggets or veins of gold with the naked eye. This made investing easier, as its value could be seen and therefore the resources required to extract it considered against its perceived value. Using a Big Data analogy, we can consider this gold to be "high value-per-byte of data".

Assume for a moment that there is more gold nearby, but it is just no visible to the naked eye. Trying to find this gold is a bit more of a gamble and potentially more expensive. This would be "low value-per-byte of data" due to the challenges associated with finding gold not visible to the naked eye.

With the right equipment however, it might be possible to economically process lots of dirt and keep the flakes of gold found. This flakes can be taken for processing and combined to make flakes of gold.

Back to our Big Data analogy...

In this scenario, it would make sense to keep all the dirt we could find (in a Big Data System), so that as new, economical dirt processing techniques emerge (Big Data Analytics on commodity systems) we would have an opportunity to extract the flakes of gold (value / insights) and store it for processing into gold bars (in our Data Warehousing system).

Hadoop is a Big Data batch system that allows users to store all data in its native business object format and get value out of it through massive parallel processing on commodity components.

Data Warehouse is characterized by "speed-of-thought response times" requirements where sustainable data with  proven value stored and delivered interactively.

It is therefore clear to see that in a Big Data world, there is value and a place for both Hadoop (Big Data) and Data Warehouse systems.

IBM's Hadoop system is Infosphere Big Insights. For simplified Big Data Analytics, look no further than IBM PureData for Analytics powered by Netezza, and Infosphere Warehouse for your Data Warehousing needs.

TerraEchos..."the next generation big-data analytics company"

I came across TerraEchos recently, an IBM business partner with a set of capabilities that illustrates the potential Big Data Analytics provides.

TerraEchos describe themselves as a company that extract meaningful information from massive amounts of complex streaming data on the fly, and simultaneously deliver insights, decisions and actions on the fly - at the precise moment they are needed. This in my view is the promise of Big Data Analytics.

As you can see from the diagram to the right, their Streaming Analytics capability requires significantly less time to analyze data.

Some of the Big Data Analytics capabilities the  TerraEcho platform exhibits includes the ability to analyze data irrespective of the amount, speed, or source of digital data, including input from any kind of cyber or physical sensor, in both structured and unstructured form. It is being positioned as suitable for organization that requires the processing, analysis, and visualization of multiple or complex streaming data sources.

One use case that caught my attention was a sophisticated sound classification system that can be used for real-time perimeter security control.  Thousands of sensors buried underground can be used to collect and classify sounds. The system can differentiate between a whisper of the wind and a human voice, or the sound of a human footstep from a running deer. If can even identify or affirm sounds that are difficult for humans to pick up.

TerraEcho has partnered with IBM to deliver these capabilities. IBM's PureData platform has been designed to simplify systems for delivering data services, making the deployment and analysis of Big Data more accessible. 

What are some of the trends creating opportunities for Big Data enriched analytics?

As I seek to understand Big Data and what it means to me as an IT practitioner, and to my clients as a consultant, I have found it useful to identify some of the trends that have underpin this opportunity

1. The number of RFID tags used in supply chain, tracking conference attendees, tracking luggage at airports, monitoring temperature of food, structures etc has increased from about 1.3 billion in 2005 to over 30 billion by end of 2011. Prices are predicted to drop below 1US cent making it possible to instrument event more systems.
2. A flight from London to New York generates about 650 TB of data which could be proactively analyzed to gain new insights that could lead to improvements in safety and other efficiencies.
3. Capturing every user's online clickstream would generate TBs of data that can be used to analyze and optimize the shopping experience. 
4. Data generated from smart meters can be used to better understand customer behavour, align supply better to demand, and enable customers to make more informed decisions about their energy usage patterns.
5. Take Facebook. The ability to analyze the whole data population, taking into account intents and sentiments can offer tremendous value. Doing this is not without its challenges. Facebook for example, experiences over 2.5 billion likes and more than 300 million photo uploads each day.
6. ..and Twitter. Twitter's 140 character or less design allows users to provide precise commentaries on a variety of subjects. The value to be derived from analyzing this data for sentiments and intents is significant.
7. What about location based services (LBS)? Apparently, the average commuter in London has their photo taken about 150 times as they travel to work. Most of the mobile devices we carry with us have LBS enabled. This information can be used to further personalize interactions. 

So, there is a lot of data being generated, and this will increase over time. Most of the data is not analyzed at all. Imagine being able to not only analyze data at rest, but also data in motion, as it hits the enterprise. He in lies the tremendous opportunities of Big Data Analytics. The PureData System from IBM simplified today's data requirements and enables clients to develop capabilities that enable them to gain insights that create a competitive advantage.