How do you work files securely in the cloud without decrypting them?

A group of IBM researchers has released a Github project that implements a homomorphic encryption system –a way to work on encrypted data in a file without first decrypting the whole file. 

Why would anyone want to do that? Partly because if you have to decrypt the file to work on it, it's going to exist as plaintext somewhere. IBM has other ideas about this as well: leaving the encrypted file encrypted would keep data protected in the cloud while still letting users work on it. 

More here 

What is Advanced Analytics?

A significant number of blog posts have been devoted to the subject of Big Data Analytics, so it is a good time to revert back to what Advanced Analytics delivers to the business.

I view Advanced Analysis as the use of data and models to provide insights to guide decisions. Advanced Analysis is therefore about data analysis, and for advanced analytics to be useful, quality data from trusted sources is critical. This is the key difference with Big Data analysis which is focused on large and varied formats of data, accumulating at high velocities.

Advanced Analytics requires structured data. Where there is structured data, Enterprise Content Management would help, but where there is no structure in the data, the analysis will be fundamentally flawed. The saying "garbage in, garbage out" holds true for Advanced Analytics.

Advanced Analytics begins with the analysis of data. Once analysed, it is presented in executive dashboards with advanced visualisation capabilities. There are 4 terms noteworthy terms in the definition of Advanced Analytics...data, models, insights and decisions.

With Advanced Analytics, mathematical algorithms are used to evaluate data a context relevant to the user. Complex mathematical models discover patterns in the data which would have previously been unknown.

Models are the complex mathematical formulae used to augment the data available to knowledge and insights. The purpose of the models is to uncover insights that drive better decision making.

I have been engaged in conversations with business leaders, demonstrating how PureApplication System can be used to accelerate the delivery of Advanced Analytics capabilities. Using patterns of expertise, we are able to significantly reduce time to value, enabling line of business executives to benefit from innovative IT capabilities in less than a couple of months, where it would have previously taken years to implement.

The Promise of Collaborative Development and Operations

I came across a  DeveloperWorks article by IBM Distinguished Engineers Ruth Willenborg and Steve Abrams where they discussed Collaborative Development and Operations.

In my view, this is about creating a collaborative development platform that aligns all stakeholders, e.g. line of business, development, testing teams, suppliers and users. The ultimate goal is to accelerate the delivery of services by the business to its customers in a way that is measurable and where continuous feedback can be provided for process improvement.

IBM PureApplication System is a platform developed with these principles in mind. On one end of the scale, you have the development organization working with business analysts to design, prototype and build software assets. These assets are integrated and consumed on PureApplication system which is optimized for  web application delivery and lifecycle management. The platform's cloud computing capabilities enable operators to interact with it in a self service manner, resulting in faster delivery of services to consumers and end users. 

In today's agile world of mobile application and services delivery, collaborative development and operations is about enabling continuous innovation by software development organizations  and continuous delivery of application services to end users and consumers. This requires a platform that supports collaborative development, continuos integration, continuos testing, continuous release management and continuous monitoring, all in an iterative lifecycle.

This article shows how applications can be on boarded onto PureApplication System using patterns of expertise. Patterns offer significant potential in enabling IT organizations to transform themselves and become better aligned to the business.

Part 2: PureSystems Analyst Roundtable Discussion

A key discussion topic for this week's Analysts roundtable is the PureData System for Transactions. 

PureData System for Transactions has been designed and built for clients with the highest demands for performance and availability. To many of our clients, such systems are core to their business and any downtime can result in loss of revenue, customer churn, tarnished image, etc.). PureData System is a  "Tier 1" system, designed for such mission critical applications.  

At the heart of PureData System for Transactions is DB2 pureScale, which provides extreme levels of availability and scalability, with application transparency. PureData System for Transactions inherits these capabilities to deliver high levels of reliability, performance and scalability out of the box. Existing DB2 applications are supported with no changes, and Oracle Database applications are also supported with minimal, and in some cases, no changes are necessary.

One of the most striking features of the IBM PureSystems family is the consistent, single web user interface through which all hardware and software components are managed and monitored. This significantly reduces the additional knowledge required to operate any offering within the PureSystems family, once a user is familiar with one of the offerings. Think about the effort required to transition from an iPod to an iPhone. Once a system is delivered, the PureData System for Transaction adoption process is as simple as

1. Power-on
2. Integrate system to the network
3. Create and deploy database clusters
4. Configure access and allocate storage to databases
5. Upload objects and data into the created databases

Part 1: PureSystems Analyst Roundtable Discussion

I have a date with a group of European Industry Analysts later in the week, and as I reflect on the key messages I would like to convey, I thought it might be useful to share my thoughts on this blog. I will start off in part 1 of the series with an introduction to the PureSystems family, the topic of interest to the Analysts.

You might recall that in April 2012, IBM introduced PureSystems with the first 2 family members, PureFlex System and PureApplication System. The PureData System joined the family in October 2012.  

The PureFlex System integrates computing, storage, networking resources and system management to simplify and accelerate the delivery of infrastructure services.  It is ideal for those clients who want to accelerate deployment and simplify management of infrastructure for building custom application platforms and application stacks. PureApplication System is an integrated application platform, delivering platform services that include pattern-based application stack deployment and management, web application serving and database management. PureData System is an expert integrated system optimized exclusively for delivering data services.

PureSystems address the challenges associated with an increasing complexity in the provisioning of IT infrastructures capabilities (storage, compute and networking) and associated web application middleware software; and the increasing volume, velocity and variety of data used today in all aspects of business.

Best practices and expertise are built into the system, reducing and in many cases, eliminating the need for operators to go through extensive documentation and best practice white papers to understand how to configure, tune, deploy and operate applications and data on the systems. 

To ensure clients are able to get value from the platform faster, system is designed, integrated and optimized for specific high performance applications in the factory. On delivery, the client simply focusses on integrating the system into their infrastructure, rather than spending time connecting the various pieces together and installing software. Standardization significantly increases time to value by eliminating the need for extensive installation and configuration. Additionally, because each PureSystem is design, configured and optimally tuned in the factory in a standard manner, troubleshooting is made easier. 

In today's highly competitive business environment, competitive advantage arises from being able to take advantage of capabilities faster that anyone else. While some might focus on delivering a vast array of features and functions, a key design principle of PureSystems is to simplified the user experience, exposing what matters most to system users and operators. This enables them to be most productive. With PureSystems, procurement is simplified with a single line of support and part number.  No assembly of components is required. Applications and data are ready to load application in hours. Open integration to 3rd party software is possible, with the whole systems managed through an integrated management console that greatly simplifies system upgrades and maintenance across the stack. 

PureSystems comes in different models that have been designed, integrated and optimized to deliver application and data services to today's demanding applications with simplicity, speed & lower cost. For more information can be found here.

NoSQL or Know SQL ?

I have spent the last few months exploring the Big Data landscape, discussing some of the technologies that are underpin the promise of Big Data like Hadoop and Map Reduce.

NoSQL is a set of capabilities that have emerged over the last few years, motivated by the fact that relational databases technologies are limited by the need to model an application up front, define a schema and operate around the schema. Proponents of NoSQL believe that this presents a significant limitation for some emerging classes of applications, particularly where the schema definition can not be determined in advance. It is important to note that NoSQL is a movement, as opposed to a specific technology. What motivates proponents of this technology is the need for SQL schema flexibility. The actual implementation varies, with proponents adopting various approaches, e.g. key store, document store, in-memory or graph oriented databases etc. 

Most of the clients I have interacted with understand the value of relational databases and the merits of being able to model applications and design around a specific scheme. This is how relational technologies are able to deliver scale, robustness, and transaction guarantees also referred to as the ACID properties. The dilemma clients have is how to retain the goodness of relational databases, while adopting elements of the NoSQL promise that has the potential to improve their overall effectiveness. This is where the notion of "Know SQL" comes in. In my view, it is not about the irrelevance of relational SQL based database technologies, but rather about knowing when more schema flexibility is desirable, particularly in the Big Data world.

Recognising this trend, IBM has been investing its relational database technologies, adding XML support into DB2 to deliver more schema flexibility, and introducing NoSQL Graph Support, also known as DB2-RDF. As a result, clients are able to achieve schema flexibility, without having to give up the proven capabilities and robustness of relational database engines. More information on DB2 NoSQL support can be found here.

What IT systems does a Smart City need?

A McKinsey Global Institute article "Urban world: Cities and the rise of the consuming class", reflects on the speed and unprecedented scale of city expansion. This is placing an increasing demand for new environmentally friendly infrastructure for water and energy, buildings, transportation and communication. Migration is motivated mostly by the search for a better life, and in additional to the infrastructure challenges previously outlined, this also places a strain on the cities to deliver citizen based services such as education, health, public safety, economic development and social programs.

Smarter city operations rely on the ability to capture data that can be used to anticipate and proactively resolve problems. Problem resolution can be achieved by coordinating processes and resources for more efficient operations. Take public safety as an example. IT systems can be deployed to predict, monitor and mitigate crisis situations. This can be achieved by automatically analyzing video streams for threats based on known criminal patterns. Similar capabilities can be applied to transportation for more effective traffic management, and for analyzing water use and consumption patterns, thereby enabling utilities to identify leakages and optimize repair jobs for improved service delivery.


The figure to the left is from  "Competing on Analytics, Davenport and Harris, 2007". 

To be competitive, Smart Cities need to deploy Analytics capabilities that enable the delivery of a high quality of services that meets and exceeds citizen expectations today, and  can accommodate future needs for real-time dynamic access to innovative new services. 

Additionally, these new services need to be resilient, secure, compliant with local requirements, and sufficiently agile to address new risks posed by an ever more connected and collaborative world. And all of this needs to be achieved cost effectively. IBM's PureData System for Analytics is designed specifically for this use case.

Imagine a situation where city operations can be collaboratively managed via Executive, City Operations and Agency dashboards that include domain key performance indicators for standard operating procedures. Such a system could support centralized planning, execution and monitoring for more efficient operations.  

IBM's Intelligent Operations Center delivered on a Cloud enabled PureSystems platform, enables cities to not only contain operational cost and complexity, but achieve breakthrough productivity gains through rapid time to value via virtualization, optimization, energy stewardship.

 

The Economics of Cloud Computing

To be competitive in today's tough economy climate, IT organizations need to find ways of delivering innovative business services while taking cost out of their operations. IT costs are normally categorized as operating expense and capital expense, and effective cost containment requires the right balance between the two.

To reduce capital expense, many IT organizations have turned to virtualization, the ability to pool and share IT resources. Pooling resources reduces capital expense of hardware, software and facilities. Less infrastructure, and correspondingly, less energy is required to deliver the same quality of service. 

Standardization on a common software stack, common operational policies, service delivery and management processes, is considered one of the most effective ways of reducing operating expense. With standardization and automation, less human resources and skill is required to deliver the same quality of service. Expertise is generally recognized as one of the fastest growing piece of the IT spend.  

Addressing standardization and virtualization is key to reducing infrastructure costs, while meeting the dynamic needs of the business. With increased infrastructure standardization, organisations are able to achieve greater economies of operating expense. Similarly, IT organizations that leverage virtualization within their infrastructure realize greater economies of capital expenditure.  

Leading IT vendors have been ceasing on this opportunity to support a closer alignment of IT to the business. IBM's Expert Integrated Systems were motivated by the increasing need for IT organisations to achieve cost optimization. 

They are built on industry standards and designed to enable organizations to accelerate cloud adoption. Cloud computing enables business agility, making it possible for organizations to improve service delivery by applying engineering discipline and economies of scale in an Internet inspired architecture. Expert Integrated Systems can be deployed in a customer controlled private network to drive efficiency, while retaining control and customization in a private cloud model.

The Promise of Mobile Business

Until very recently, I was mostly oblivious to the Bring Your Own Device (BYOD) trend that appears to be getting more and more embedded in the fabric of enterprise IT. I now find myself owning a personal iPhone, Samsung Galaxy and a MacBook Air, all of which I use for work. In my case, the acquisition was largely driven by a desire to use what I believed would enabled me to be most productive. I had also become more and more frustrated with the length of time it took to boot up my laptop, let alone the weight. The MacBook Air was therefore a pretty compelling proposition. Email on my iPhone was more pleasing, and the Samsung Galaxy enabled me to create a mobile private Wifi hotspot using my personal 3G broadband sim card. All I need is an iPad Mini to complete the picture!

From a business point of view, these trends are having a  transformational impact, creating opportunities for new business models as user's expectations drive what IT needs to deliver. Think about your mobile experience for a minute. I find myself contemplating switching banks based on the user experience of the mobile banking application. And for enterprise IT, it is not simply a matter of getting the best application out of the door. Mobile applications have a much faster lifecycle, and are much more iterative than traditional applications. Enterprise IT therefore needs to adopt a much more agile application development and release lifecycle. And the challenges do not end there. They also have to cater for a fragmented landscape of devices and network providers.

That said, the opportunities that can be realized through mobile business are immense. In a recent tweet, I shared a link to an article on 19 Surprising Facts About Africa’s Mobile Market. The potential for leveraging the mobile platform for business is clear. What makes the mobile platform different and compelling is the flexibility it offers in terms of the multitude of applications available, the wide range of connectivity options, multimedia support, location awareness and other sensory capabilities.

While many recognize the opportunities available in the business to consumer space, the business-to-enterprise benefits are not very well understood. The mobile platform, when used effectively, can significantly enhance worker productivity, by enabling mobile workers to have access to the information and resources they need, wherever they are and when ever they need it. Throughout most of my IT career, I have been a mobile worker. The benefits to me personally and to my employer are obvious. I am able to work from home when family needs dictate without always having to take a day off. My colleagues are able to reach me even while I am traveling. The ability to work from home  makes me more productive. I do not have to incur the cost and time of traveling into the office every day. I reduce my carbon footprint, while my employer is able to reduce their personnel costs with schemes like Bring Your Own Device to work and by reducing office space to a bare minimum. In many instances, these benefits can be realized by simply extending existing applications to mobile workers and customers. I have personally witness many of my clients significantly improve customer satisfaction and increase revenue by leveraging the mobile platform to deepen their customer engagement. Developing a mobile strategy is clearly one of the ways organizations can differentiate themselves.

To many, context awareness is the new killer application. This relates to the ability to dynamically capture contextual information from mobile devices and use it to optimize and change communication and business flows. With context awareness, you can increase customer loyalty by delivering loyalty reward to a customer based on their unique needs at a given point in time. This concept can be extended to the enterprise with mobility-linked business processes. In this instance, business processes can be improved by capturing and automating process interactions where and when they occur.

Let's discuss the scenarios above using examples. In a customer loyalty use case, a retailer can offer an application for creating and managing shopping list. As a customer creates a list, their intended purchase could analyze against their historical shopping patterns, enabling the retailer to offer personalized shopping vouchers. The retailer's objective is to retain the shopper's business and capture a greater share of spend. The process of analyzing spending patterns and making an offer needs to be done at realtime. The PureData Systems for Analytics is designed to address petascale analytical workload requirements. In a business to business context, similar principles can be applied to deliver information to field workers just in time. In other words, consider an engineer working on an offshore oil rig. As they go about their job and input data into their mobile workpad, information can be analyzed in real time and the business process changed on the fly so  additional checks or maintenance is conducted. The same principles can be applied to other scenarios, e.g. optimizing supply chain and logistics planning, organizational learning to improving business outcomes etc.

Recognizing the opportunity mobile business presents, we have been working closely with some of our clients, leveraging the Worklight platform to enable them to create rich, yet cost-effective mobile apps in this fragmented technological landscape. A key piece of the capability we deliver is enterprise integration, enabling clients to connect to enterprise back-end services in a secure and scalable manner, while securely managing and controlling the growing portfolio of deployed enterprise applications.

Redefining the Economics of Enterprise IT

Did you followed IBM closely in 2012? If you did, you would (should) have come across PureSystems, a new approach to computing that is set to radically redefine the economics of IT. 

PureSystems introduces a new approach to IT system design, procurement, lifecycle management and retirement. Systems are designed and optimized in the factory for a particular set of related tasks (workloads) and are not meant to be general purpose or highly customizable. Generally, they have a static, predefined stack of hardware and software that cannot be altered. Where the system design does not meet the needs of a particular use case, other custom solutions may be more appropriate.

So why design a system with such "limitations" ?

As it turns out, more is less in this instance. PureSystems are purpose-built systems that are intended to be easy to setup, configure and manage. Much of the burden of setting up and managing traditional IT system have been removed or radically simplified, making it possible for clients to adopt highly complex enterprise IT systems without having to understand all the complex internal details. When one looks deeper into the time spent on the phases of implementing IT projects, it becomes clear that there is significant opportunity for improvement to accelerate time-to-value. Forrester did some research for IBM which showed that the top causes for delays were related to the time spent integrating, configuration and tuning systems, as well as deploying applications. These are key areas address by PureSystems.

Think for a moment about what Apple did with the original Macs and subsequently the iPod and iPhone. All Apple devices were  integrated and optimized in the factory for a well defined set of tasks, the Mac for personal computing, the iPod for multimedia entertainment, and the iPhone for mobile telephony and related functions. Apple was able to optimized the software running on these devices as they understood and tightly controlled the hardware. This enabled them to create a much more pleasing user experience, and this observation is not simply related to the Software Ergonomics, but they way the device feels; its weight, its thickness, the way it looks. There is much more to an Apple device that the functionality it offers, and this in my view, was the one of the triggers for Apple's cult following.

Now, turn to Google's Android. Google's Android operating system is freely available for all to use, and Samsung's innovation has ensured the Andriod operating system remains dominate outside of the USA. While openness and ease of adoption is most certainly a competitive advantage for Android, the loose coupling between the Operating System and the hardware makes it more challenging for Android to achieve the same level of brand specificity as Apple. In the Android world, users have a choice of devices from many players with their very own unique device designs. This does not in any way guarantee Apple's dominance or vice versa. In my view, there are merits in both approaches.

Let's get back to PureSystems...

With PureSystems, IBM has made available, the benefits of both approaches. The "Apple" of PureSystems are the Expert Integrated Systems that ship from the factory pre-racked, with all the storage, compute, networking and middleware software components in place, and ready for use. Like the iPhone, once the system is out of its packaging, you plug it into the power supply, connect it to the network, and go through a mostly automated set up process to initialized the systems and begin installing applications from an internal application store. As with Apple's devices, PureSystems has various offerings designed to address different usage profiles. There is an application system for running transactional web applications, a database system for transactional database applications, and an analytics systems for analyzing large data sets. This is akin to the iPhone, iPad, Mac etc. And as with the iPad and iPad Minis, these also come in different sizes and configurations that are easy to understand and adopt based on the anticipated workload. PureSystems offers the best user experience, enabling IT organisations to adopt enterprise computing capabilities while realizing much faster time to value.

But IBM does not stop there. Recognizing that others might have already invested in hardware from IBM and other vendors, IBM introduced IBM Workload Deployer. A subset of the capability used for installing and life-cycle managing applications in the PureSystems application store is available in IBM Workload Deployer (IWD). With IWD, clients can automate the installation and lifecycle management of applications onto a member of the PureSystems family, PureFlex, a pre-integrated, optimized infrastructure system, as well as other 3rd party and IBM custom built hardware systems. This model is akin to the Android model, where various device manufacturers build and design their custom hardware for the Android Operating System. While the software user experience might be consistent, the physical experience, i.e. the look and feel of the device varies across device manufacturers. So while the Samsung Galaxy S3 might run the Android Operating System as well as the Samsung Galaxy Note or LG, Motorola or any other vendor's device, the total experience is undeniably different.

In my view, investing in the PureSystems family is akin to investing in a range of Apple devices. The customer service experience, the look and feel, and experience of working with the offering is very consistent across the range of products. If you know how to use one, learning to use the other is relatively easy, when compared to learning something new. Investing in IBM Workload Deployer on the other hand has its advantages. You are able to deploy applications across a hybrid set of hardware infrastructures. This requires IBM, the client or a third-party to configure the hardware platform for use with IWD. Included in the PureSystems family is PureFlex, an offering well positioned for this use case. By acting as a middleware deployment and lifecycle management broker for a range of hardware platforms and operating systems, some of the optimizations possible from tight integration and control of the underlying hardware are not possible. Lifecycle management activities such as hardware firmware patch management would vary between platforms.

In conclusion, it is worth stating that Workload Deployer and PureSystems deliver a comprehensive set of cpabilities that address the need for agility and simplicity in IT services delivery to the business. Where an IT organization is focused on time to value, PureSystems has some distinct benefits, and where there is a need to leverage existing infrastructure investments, IWD is uniquely positioned. We see many of our most innovative clients adopting both systems effectively to transform their IT operations and differentiate themselves over their competitors. 

Why is Open Source fidelity important ?

IT vendors who maintain fidelity to open source and ensure their products are 100% compatible with the open source packages they rely on, can realize some tangible benefits.

Fidelity to open source makes it easier to integrate with other products designed to be compatible with a common set of open source packages. This approach has enabled IBM to amass 100s of partners, accelerating the adoption of products like IBM's Big Insights which builds on the open source Hadoop distribution. Clients and software vendors using open source libraries and interfaces for Hadoop can rest assured that their applications will run with Big Insights.

This approach of embracing and maintaining the integrity of open source projects enables vendors to adopt open source enhancements and innovate on top for competitive differentiation. There is an opportunity for software vendors to test and certify that each component works well together in a manner suitable for enterprise production deployment. Additionally, vendors can address backward compatibility issues prevalent in some open source projects by undergoing rigorous regression testing.

As with many other open source projects, the IBM Big Insights development team has committers to the Hadoop project and its related projects. These committers contribute to open source innovation and ensures that IBM plays a role in shaping Hadoop and is aligned to its future evolution. 

Part 3 : High Availability Enterprise Hadoop Clusters

Hadoop is primarily made up of 2 elements 1) The distributed Hadoop Filesystem (HDFS) and a programming paradigm, MapReduce.

Redundancy is built into Hadoop clusters. Data is redundantly stored in multiple places across the cluster, with portions of the functions running on various servers in the cluster. Hadoop is designed with the expectation that node failures will occur. Hadoop is fault tolerant. If a failure occurs, it automatically heals itself by nominating another node in the cluster to perform the work destined for the failed node.

That said, for enterprise deployment, a hot standby for the NameNode and the JobTracker server in the MapReduce processing layer is required. This was achieved in Hadoop 1.0 by having an active / passive failover solution, where data is replicated across two separate Hadoop clusters. An alternative solution is to have a dedicated backup for the master node, which includes the NameNode and can also include the JobTracker service. Should the NameNode fail, the Hadoop cluster can restart using the backup NameNode.

IBM committers have been working on Hadoop 2.0 with the Hadoop open source community to address the single point of failure challenges. In Hadoop 2.0, it is possible to designate a hot standby for the HDFS NameNode, and MapReduce2 has been enhanced to eliminate the potential single point of failure in the JobTracker by distributing its functionality across nodes in the cluster.

Part 2 : What is IBM's Enterprise Hadoop ?

In a previous blog post, I introduced Hadoop at a very high level. 

Hadoop is clearly a technology that can be used to overcome the volume and variety challenges of data at rest. Traditional analytics tools work very well on structured data, but today, only 20% of the data is structured. The need for an alternative solution is clear.

While many organisations have been experimenting with Hadoop to bridge this gap, few have been able to leverage it to gain insights that creates meaningful value. This is partly due to the deep skills required to work with Hadoop, and also the lack of enterprise capabilities that address information governance concerns, security and data lifecycle management.

IBM introduced Infosphere Big Insights, its Enterprise Hadoop platform to enable organisations to leverage Hadoop in a way that could have a transformational impact on their business and deliver significant competitive advantage. To achieve this, IBM introduced focussed on delivering 3 key capabilities. 

1. Analytics support to enhance consumability, enabling analysts with minimal programming skills to get value out of big Data. 
2. Enhancing the Hadoop platform so that it integrates with the rest of the enterprise, enabling Analysts to query Big Data from their existing Data Warehouse and vice versa.
3. Delivering additional tools and capabilities that address governance, security, administration and performance concerns inhibiting the enterprise adoption of Big Data 

For more information on IBM's Infosphere Big Insights, visit this link.

Part 1 : What is Hadoop ?

Imagine networking thousands of computers together. Each computer has its own processor and hard disk drive. All these computers are running software that makes the computers appear as a single powerful "super computer" with lots of processing capability and storage space. 

Now assume you are Amazon, and using this "super computer" to store click stream records from your amazon.com web site. You now want to understand if there are trends that lead to customers not completing transactions after having added items into their shopping carts. Recall that the click stream data is spread across the local disk drives on all of these thousands of computers.

To gain the insights required, a copy of your Analytics (application) logic is sent to each individual computer. Each computer  then runs the application logic against data stored locally. Instead of bring data to the application, the application (or function) is moved to the location where data is stored. Moving data across a network has a significant impact on performance, and by avoiding this, near linear scalability is achieved. Increasing data processing requirements can be accommodated simply by adding more computers.

Hadoop is the term used to describe this distributed filesystem(HDFS) and data processing engine that can be used to handle extremely high volumes of unstructured data at Internet scale. These group of computers make up a Hadoop cluster. Each computer in a Hadoop cluster is referred to as a node. The programming model used to bring the function (application logic) to the data is known as Map Reduce.

Technologies like Hadoop is what enables companies like Facebook, Google and Yahoo! to store millions of digital images and elements of our conversations, without having to design or understand up front, the format of the information or content they need to handle. This flexibility and ability to scale in a near linear fashion is one of the key attraction of Hadoop. Yahoo! reportedly has over 40,000 nodes spanning its Hadoop clusters which store over 40PB of data.

Part 3 Social Business : Building learning visions

In the face of an increasingly globally competitive business landscape, organisations will have to deal with huge uncertainties related to their viability and sustainability and many will face a range of demographic changes over the next few decades, including mergers and acquisition, ageing workforce, the transience and mobility of the workforce, as well as a whole range of other economic and societal developments that are likely to impinge significantly on the types of skills and competencies required in the future.

In order to respond to such uncertainty and serve the changing needs and diversity within the workforce, learning spaces cannot be rigid or exclusive. They need to build on, interconnect and integrate informal and formal provision that already exists. Designing new learning spaces requires companies to consider not only the purpose of acquiring knowledge now, but far more importantly, the changes necessary for a better, more holistic learning environment for their workforce in the future. They need to start, then, by asking not ‘what tools do we want?’ but instead ‘what sort of learning environment do we want to see in future?’ They need to ask not ‘what sorts of learning relationships do they want to foster? What competencies do they want learners to develop? What tools and resources are available to support learning?’

Several dynamics need to be taken into account when considering alternative models of learning. Immediate contextual dimensions, such as cross organisational partnerships, wider cultural influences, as well as establishing clarity about critical learning factors, such as the role of the learner, the organisation and pedagogy. Until organisations ask these questions, they will not be designing learning spaces for the future, but will simply be reproducing technologies of the past, albeit with more innovative capabilities.

For learning to be effective, a strategy defining the sort of learning interactions and practices is desired. Organisations need a ‘built andragogy strategy’, the collaboration and human resource management vision to underpin the design principles for the workplace of the future.

To learn more about becoming a creating a smarter workforce, click this link. 

Part 2 Social Business : What might new learning networks look like

In this second series, we take a look at what new learning networks might look like. What might the organisational landscape of the future look like? What types of trainers, spaces and places for learning will be available? Where, and with whom, should learning happen?

If organisations are interested in achieving a fully personalised learning environment designed around the needs, interests and aspirations of learners, they need to challenge a number of fundamental assumptions which have historically underpinned the learning landscape:

• First, they need to challenge the assumption that expertise and knowledge reside only within the Human Resources department, and to ask instead, what might be gained from tapping into the resources that exist in the wider employee community and within the networks that employees are already connected to.

• Second, they need to challenge the assumption that ‘learning’ and ‘training’ are different words for the same thing, and to ask instead what different approaches to and models of learning exists the workplace and the wider Internet community?

• Third, they need to challenge the assumption that a one-size-fits-all approach is most effective, and examine how the recognition of learners’ diverse voices and experiences can enhance inclusion, aspiration and achievement through the creation of personalised learning trajectories.

• Finally, as digital resources increasingly offer opportunities for networked, collaborative and distributed learning and interaction, they need to challenge the assumption that the easiest and most cost effective approach to organising learning is within the walls of their organisation.

Organisations need to move away from the institutionalised logic of the human resources department as factory, to the network logic of the learning community. Indeed, they need to move towards a notion of extending learning, whereby human resources department rethink the possibilities around what can be learnt, where learning can happen and who is involved in the learning process. Rather than continuing to build systems based upon centralised control of the human resources organisations and a predefined curriculum, organisations need to move to systems organised through more porous and flexible learning networks that link communities of interests and multiple spaces and places of learning. Social Business tools can empower your employees to form communities and share knowledge

Part 1 Social Business : Why learning networks?

In this 3 part series, I revisit an article I wrote in March 2008 outlining the significance of networks within the context of learning.

Social, technical and leisure life is increasingly organised around networks. The metaphor (and reality) of the network has come to be seen as epitomising the social, economic and technological changes of the last 30 years. The network is now the fundamental underpinning structure of social organisation – and that it is in and through networks – both real and virtual - that life is lived in the 21st century. Networks are one of the most important organisational form of our time. By harnessing the ‘network logic’, the ways we view the world and the tools we use for navigating and understanding it, will change significantly.

The ability to understand how to join and build these networks, the tools for doing so and the purpose, intention, rules and protocols that regulate use and communications, therefore, become increasingly important skills. This concept of the ‘network society’ calls into question what it means to be ‘learned’ today – what new skills, what new ways of working and learning, what new knowledge and skills will be required to operate in and through these networks? It requires us to ask whether learning systems, premised not upon networks but upon individualised acquisition of content and skills, is likely to support the development of the competencies needed to flourish in such environments. 

Explaining how IBM is making the cities work better...

I was at home last night watching CNBC (could have been Bloomberg) with my 10 year old daughter when an IBM advertisement campaign came on and got her excited. Needless to say I was left with the task of explaining what IBM does today. What's the IBM company of 2013? What capability would encapsulate what "Building a Smarter Planet" signifies?

As I pondered through this, I settled on the view that the best way of approaching the issues is to focus on a challenge most of us experience / encounter on a daily basis. The need to leverage technology to sustain economic growth and enhance the quality of life of citizens in a rapidly changing world appeared to be a good choice.

From an offerings point of view, I narrowed my focus on IBM's Intelligent Operations Center portfolio. Our city leaders are being held to account, and need to improve their decision making. IT can help by enabling them to derive insights from the information and data they have at their disposal. In this instance, the volume of data needs to be sufficiently comprehensive and readily available for analysis. Additionally, where historical data is not available, IT can also be used to anticipate problems, so the right people can get in front of events and problems before they become a crisis. And all of these activities require coordination; coordination of people, resources, systems etc.
 
The need to empower city leaders, from mayors to the police chiefs, heads of transportation, and beyond is clear. The Intelligent Operations Center Portfolio of services enables users to

    1.    Leverage available information to make better decisions
    2.    Analyze problems and resolving them proactively
    3.    Coordinate resources and processes to operate effectively

The initial 3 areas of focus was the intelligent management of operations, whatever they might be, traffic and water management.

Intelligent Operations Management is about planning, organizing, monitoring and sharing information between various stakeholders. We are in an era where governments are being held to account, so the ability for public sector executives to access information on dashboards that include domain specific key performance indicators is critical. 

With respect to traffic management, the need for realtime visibility is a obvious. The ability to analyze historical data and correlate to traffic incidents can enhance future planning. Water is one of the world's most precious resources, and in many parts of the world needs to be managed carefully. IBM has invested in developing capabilities that enable utility companies to instrument, collect, analyze and visualize data on water usage using Geographical information systems. IBM is also innovating on the delivery of these capabilities to its clients, offering both the ability to deploy this on premise, in which case the PureApplication System would be a attractive platform, or as a Software as a Service offering on IBM's Smart Cloud.

What is driving the increasing adoption of Cloud Computing ?

The challenges associated with growing IT complexity and its impact on business agility has been the focus of recent conversations on this blog. There is clearly a need to address increasing operating costs associated with the delivery of IT services. Studies conducted by IBM, IDC and others has shown that over the last few years, management and administration costs have increased significantly against flat or decreasing new system spend, and marginal increases in power and cooling costs.

Cloud computing is the new consumption and deliver model inspired by consumer Internet services.  Underpinning cloud computing are proven capabilities of virtualization, automation and standardization, which is driving IT simplification and increased efficiency. Line of business executives are attracted to the self-service characteristics of Cloud, the new sourcing options it presents, and its economy of scale. Cloud essentially represents the industrialization of delivery for IT supported services. Leading cloud vendors like IBM provide clients with the flexibility to adopt the cloud solution appropriate for their organization, be it a Private Cloud, Public Cloud or a Hybrid Cloud including elements of both. In a recent newsletter, IDC predicted the increasing emergence of workload specific Clouds in 2013.

I see organizations adopting Cloud in 3 steps

1) Consolidation
2) Virtualization
3) Automation

Consolidation reduces server sprawl and results in a reduction in infrastructure complexity. Reduced complexity translates to less effort to operate and manage, which should improve operational costs and reduced total cost of ownership

By adopting a shared, virtualized infrastructure, IT organizations are able to significantly increase hardware utilization, resulting not only in a reduction in new hardware spend, but also with regards to associated costs like cooling and power. A virtualized infrastructure can also simplify application deployment and ongoing operation by removing physical resource boundaries.

Add automation to a consolidated and virtualized infrastructure, and you can dramatically reduce deployment cycles. Standardized delivery of IT services provides an opportunity to introduce granular metering and billing. This enables the flexible delivery of new processes and services.

IBM's PureApplication System is designed to accelerate cloud adoption. It enables customers to deploy Cloud services in less than 4 hours, from power on and connection to the client's network. It is integrated in the factory with advanced virtualization that enables IT resources to be shared among many applications. This results in more efficient utilization of IT resources and reduced hardware costs through economies of scale.

Patterns of Expertise, the ability to embed application configuration, deployment and lifecycle management best practices significantly reduces IT cycle times and management costs by enabling applications to be deployed with minimal skill in the shortest time possible.

Workload optimization algorithms and tight integration of all components, including software and hardware, results in a platform that can scale up and down to optimize IT resource utilization, with the network and storage system optimized to the workload.

What's IDC's prediction for 2013?

IDC recently published their 2013 predictions, with Mobile, Social, Cloud and Big Data platforms expected to  drive 90% growth in the IT market between 2013 and 2020. I summarized the key 2013 trends are as follows:

• IT spending will exceed $2.1 trillion, driven by the adoption of smart mobile devices
• IT spending in the growth markets will represent 34% of worldwide spend, and 50% of all new growth in the IT marketplace
• Package applications providers like IBM, Microsoft and Oracle will become major Software as a Service (SaaS) providers
• There will be an explosion in Platform as a Service (PaaS) offerings tailored for specific industry applications
• Converged systems will transition from hype to market reality, as enterprise datacenter and cloud-provider use cases converge
• Line of Business executives will drive the increasing adoption of industry solutions, with 60% of all new IT spend influenced by LOB executives
• Enterprises will transition from Social Network experimentation to integration
• Big Data investments investments will continue to grow, with a shift of focus to analytics and discovery tools, and analytic applications