Part 2 : What is IBM's Enterprise Hadoop ?

In a previous blog post, I introduced Hadoop at a very high level. 

Hadoop is clearly a technology that can be used to overcome the volume and variety challenges of data at rest. Traditional analytics tools work very well on structured data, but today, only 20% of the data is structured. The need for an alternative solution is clear.

While many organisations have been experimenting with Hadoop to bridge this gap, few have been able to leverage it to gain insights that creates meaningful value. This is partly due to the deep skills required to work with Hadoop, and also the lack of enterprise capabilities that address information governance concerns, security and data lifecycle management.

IBM introduced Infosphere Big Insights, its Enterprise Hadoop platform to enable organisations to leverage Hadoop in a way that could have a transformational impact on their business and deliver significant competitive advantage. To achieve this, IBM introduced focussed on delivering 3 key capabilities. 

1. Analytics support to enhance consumability, enabling analysts with minimal programming skills to get value out of big Data. 
2. Enhancing the Hadoop platform so that it integrates with the rest of the enterprise, enabling Analysts to query Big Data from their existing Data Warehouse and vice versa.
3. Delivering additional tools and capabilities that address governance, security, administration and performance concerns inhibiting the enterprise adoption of Big Data 

For more information on IBM's Infosphere Big Insights, visit this link.

Part 1 : What is Hadoop ?

Imagine networking thousands of computers together. Each computer has its own processor and hard disk drive. All these computers are running software that makes the computers appear as a single powerful "super computer" with lots of processing capability and storage space. 

Now assume you are Amazon, and using this "super computer" to store click stream records from your amazon.com web site. You now want to understand if there are trends that lead to customers not completing transactions after having added items into their shopping carts. Recall that the click stream data is spread across the local disk drives on all of these thousands of computers.

To gain the insights required, a copy of your Analytics (application) logic is sent to each individual computer. Each computer  then runs the application logic against data stored locally. Instead of bring data to the application, the application (or function) is moved to the location where data is stored. Moving data across a network has a significant impact on performance, and by avoiding this, near linear scalability is achieved. Increasing data processing requirements can be accommodated simply by adding more computers.

Hadoop is the term used to describe this distributed filesystem(HDFS) and data processing engine that can be used to handle extremely high volumes of unstructured data at Internet scale. These group of computers make up a Hadoop cluster. Each computer in a Hadoop cluster is referred to as a node. The programming model used to bring the function (application logic) to the data is known as Map Reduce.

Technologies like Hadoop is what enables companies like Facebook, Google and Yahoo! to store millions of digital images and elements of our conversations, without having to design or understand up front, the format of the information or content they need to handle. This flexibility and ability to scale in a near linear fashion is one of the key attraction of Hadoop. Yahoo! reportedly has over 40,000 nodes spanning its Hadoop clusters which store over 40PB of data.

Part 3 Social Business : Building learning visions

In the face of an increasingly globally competitive business landscape, organisations will have to deal with huge uncertainties related to their viability and sustainability and many will face a range of demographic changes over the next few decades, including mergers and acquisition, ageing workforce, the transience and mobility of the workforce, as well as a whole range of other economic and societal developments that are likely to impinge significantly on the types of skills and competencies required in the future.

In order to respond to such uncertainty and serve the changing needs and diversity within the workforce, learning spaces cannot be rigid or exclusive. They need to build on, interconnect and integrate informal and formal provision that already exists. Designing new learning spaces requires companies to consider not only the purpose of acquiring knowledge now, but far more importantly, the changes necessary for a better, more holistic learning environment for their workforce in the future. They need to start, then, by asking not ‘what tools do we want?’ but instead ‘what sort of learning environment do we want to see in future?’ They need to ask not ‘what sorts of learning relationships do they want to foster? What competencies do they want learners to develop? What tools and resources are available to support learning?’

Several dynamics need to be taken into account when considering alternative models of learning. Immediate contextual dimensions, such as cross organisational partnerships, wider cultural influences, as well as establishing clarity about critical learning factors, such as the role of the learner, the organisation and pedagogy. Until organisations ask these questions, they will not be designing learning spaces for the future, but will simply be reproducing technologies of the past, albeit with more innovative capabilities.

For learning to be effective, a strategy defining the sort of learning interactions and practices is desired. Organisations need a ‘built andragogy strategy’, the collaboration and human resource management vision to underpin the design principles for the workplace of the future.

To learn more about becoming a creating a smarter workforce, click this link. 

Part 2 Social Business : What might new learning networks look like

In this second series, we take a look at what new learning networks might look like. What might the organisational landscape of the future look like? What types of trainers, spaces and places for learning will be available? Where, and with whom, should learning happen?

If organisations are interested in achieving a fully personalised learning environment designed around the needs, interests and aspirations of learners, they need to challenge a number of fundamental assumptions which have historically underpinned the learning landscape:

• First, they need to challenge the assumption that expertise and knowledge reside only within the Human Resources department, and to ask instead, what might be gained from tapping into the resources that exist in the wider employee community and within the networks that employees are already connected to.

• Second, they need to challenge the assumption that ‘learning’ and ‘training’ are different words for the same thing, and to ask instead what different approaches to and models of learning exists the workplace and the wider Internet community?

• Third, they need to challenge the assumption that a one-size-fits-all approach is most effective, and examine how the recognition of learners’ diverse voices and experiences can enhance inclusion, aspiration and achievement through the creation of personalised learning trajectories.

• Finally, as digital resources increasingly offer opportunities for networked, collaborative and distributed learning and interaction, they need to challenge the assumption that the easiest and most cost effective approach to organising learning is within the walls of their organisation.

Organisations need to move away from the institutionalised logic of the human resources department as factory, to the network logic of the learning community. Indeed, they need to move towards a notion of extending learning, whereby human resources department rethink the possibilities around what can be learnt, where learning can happen and who is involved in the learning process. Rather than continuing to build systems based upon centralised control of the human resources organisations and a predefined curriculum, organisations need to move to systems organised through more porous and flexible learning networks that link communities of interests and multiple spaces and places of learning. Social Business tools can empower your employees to form communities and share knowledge

Part 1 Social Business : Why learning networks?

In this 3 part series, I revisit an article I wrote in March 2008 outlining the significance of networks within the context of learning.

Social, technical and leisure life is increasingly organised around networks. The metaphor (and reality) of the network has come to be seen as epitomising the social, economic and technological changes of the last 30 years. The network is now the fundamental underpinning structure of social organisation – and that it is in and through networks – both real and virtual - that life is lived in the 21st century. Networks are one of the most important organisational form of our time. By harnessing the ‘network logic’, the ways we view the world and the tools we use for navigating and understanding it, will change significantly.

The ability to understand how to join and build these networks, the tools for doing so and the purpose, intention, rules and protocols that regulate use and communications, therefore, become increasingly important skills. This concept of the ‘network society’ calls into question what it means to be ‘learned’ today – what new skills, what new ways of working and learning, what new knowledge and skills will be required to operate in and through these networks? It requires us to ask whether learning systems, premised not upon networks but upon individualised acquisition of content and skills, is likely to support the development of the competencies needed to flourish in such environments. 

Explaining how IBM is making the cities work better...

I was at home last night watching CNBC (could have been Bloomberg) with my 10 year old daughter when an IBM advertisement campaign came on and got her excited. Needless to say I was left with the task of explaining what IBM does today. What's the IBM company of 2013? What capability would encapsulate what "Building a Smarter Planet" signifies?

As I pondered through this, I settled on the view that the best way of approaching the issues is to focus on a challenge most of us experience / encounter on a daily basis. The need to leverage technology to sustain economic growth and enhance the quality of life of citizens in a rapidly changing world appeared to be a good choice.

From an offerings point of view, I narrowed my focus on IBM's Intelligent Operations Center portfolio. Our city leaders are being held to account, and need to improve their decision making. IT can help by enabling them to derive insights from the information and data they have at their disposal. In this instance, the volume of data needs to be sufficiently comprehensive and readily available for analysis. Additionally, where historical data is not available, IT can also be used to anticipate problems, so the right people can get in front of events and problems before they become a crisis. And all of these activities require coordination; coordination of people, resources, systems etc.
 
The need to empower city leaders, from mayors to the police chiefs, heads of transportation, and beyond is clear. The Intelligent Operations Center Portfolio of services enables users to

    1.    Leverage available information to make better decisions
    2.    Analyze problems and resolving them proactively
    3.    Coordinate resources and processes to operate effectively

The initial 3 areas of focus was the intelligent management of operations, whatever they might be, traffic and water management.

Intelligent Operations Management is about planning, organizing, monitoring and sharing information between various stakeholders. We are in an era where governments are being held to account, so the ability for public sector executives to access information on dashboards that include domain specific key performance indicators is critical. 

With respect to traffic management, the need for realtime visibility is a obvious. The ability to analyze historical data and correlate to traffic incidents can enhance future planning. Water is one of the world's most precious resources, and in many parts of the world needs to be managed carefully. IBM has invested in developing capabilities that enable utility companies to instrument, collect, analyze and visualize data on water usage using Geographical information systems. IBM is also innovating on the delivery of these capabilities to its clients, offering both the ability to deploy this on premise, in which case the PureApplication System would be a attractive platform, or as a Software as a Service offering on IBM's Smart Cloud.

What is driving the increasing adoption of Cloud Computing ?

The challenges associated with growing IT complexity and its impact on business agility has been the focus of recent conversations on this blog. There is clearly a need to address increasing operating costs associated with the delivery of IT services. Studies conducted by IBM, IDC and others has shown that over the last few years, management and administration costs have increased significantly against flat or decreasing new system spend, and marginal increases in power and cooling costs.

Cloud computing is the new consumption and deliver model inspired by consumer Internet services.  Underpinning cloud computing are proven capabilities of virtualization, automation and standardization, which is driving IT simplification and increased efficiency. Line of business executives are attracted to the self-service characteristics of Cloud, the new sourcing options it presents, and its economy of scale. Cloud essentially represents the industrialization of delivery for IT supported services. Leading cloud vendors like IBM provide clients with the flexibility to adopt the cloud solution appropriate for their organization, be it a Private Cloud, Public Cloud or a Hybrid Cloud including elements of both. In a recent newsletter, IDC predicted the increasing emergence of workload specific Clouds in 2013.

I see organizations adopting Cloud in 3 steps

1) Consolidation
2) Virtualization
3) Automation

Consolidation reduces server sprawl and results in a reduction in infrastructure complexity. Reduced complexity translates to less effort to operate and manage, which should improve operational costs and reduced total cost of ownership

By adopting a shared, virtualized infrastructure, IT organizations are able to significantly increase hardware utilization, resulting not only in a reduction in new hardware spend, but also with regards to associated costs like cooling and power. A virtualized infrastructure can also simplify application deployment and ongoing operation by removing physical resource boundaries.

Add automation to a consolidated and virtualized infrastructure, and you can dramatically reduce deployment cycles. Standardized delivery of IT services provides an opportunity to introduce granular metering and billing. This enables the flexible delivery of new processes and services.

IBM's PureApplication System is designed to accelerate cloud adoption. It enables customers to deploy Cloud services in less than 4 hours, from power on and connection to the client's network. It is integrated in the factory with advanced virtualization that enables IT resources to be shared among many applications. This results in more efficient utilization of IT resources and reduced hardware costs through economies of scale.

Patterns of Expertise, the ability to embed application configuration, deployment and lifecycle management best practices significantly reduces IT cycle times and management costs by enabling applications to be deployed with minimal skill in the shortest time possible.

Workload optimization algorithms and tight integration of all components, including software and hardware, results in a platform that can scale up and down to optimize IT resource utilization, with the network and storage system optimized to the workload.