Digital Integration Platforms: What are they and why should I care?

RESEARCH REPORT // FREE
 
Digital Integration Platforms enable teams to work together to design, deploy, operate, monitor and change integration software services that connect business applications, data sources and other software platforms – wherever they may be hosted. Digital Integration Platforms are delivered as-a-service, licensed on a subscription basis and priced according to usage. Is a Digital Integration Platform a good fit for your organisation?

Top takeaways

Digital Integration Platforms use an established technology core…

Digital Integration Platforms enable teams to work together to design, deploy, operate, monitor and change integration software services that connect business applications, data sources and other software platforms – wherever they may be hosted.

The basic fundamentals of Digital Integrations Platforms’ core capabilities are not at all new: they borrow heavily from Enterprise Application Integration (EAI) tools, Enterprise Service Bus (ESB) products and Extract, Transform and Load (ETL) tools.

But they reflect new realities…

As endpoints proliferate and as business and technology change cycles continue to accelerate, a new kind of integration platform approach is vital. Integration of systems, applications, platforms and data sources needs to be far cheaper, easier and quicker than before. Yesterday’s high-cost, difficult-to-use platforms are naturally subject to tight control within enterprises. This is where Digital Integration Platforms come in.

… and they help organisations deal with today’s integration demands

Although the core functional capabilities of Digital Integration Platforms are similar to those of yesterday’s integration tools, they differ (to a greater or lesser extent, depending on the provider in question) from those older technologies in terms of four key characteristics: speed, simplicity, abstraction and openness.

Because of this, they offer a compelling way for business teams to solve integration and automation challenges that may otherwise remain unaddressed. Today’s emerging Digital Integration Platforms are the natural successors to yesterday’s more heavyweight, expensive integration tools.

What are Digital Integration Platforms?

Integration… the never-ending challenge

Ever since organisations have been able to freely invest in information technology, they’ve needed to transfer information between systems. For many years, batch file transfer was the dominant method of shifting digital data from one place to another; then, as datasets became larger and the effectiveness of ‘batch processing windows’ shrank, more real-time approaches to system co-ordination, based on messaging protocols, became increasingly popular. Increasing openness of system architectures in the 1980s and 1990s enabled distributed systems to call each other’s functions.

Today’s business environments and expectations are now fundamentally enabled by and intertwined with IT capabilities. Business is now ‘always-on’ to such an extent that it seems odd, or even absurd, to consider situations where there’s no need to rapidly co-ordinate the actions of sales, marketing, operations and customer service systems – to build integration bridges between CRM and ERP systems, for example.

Legacy systems have been with us for decades, and indeed the definition of what counts as ‘legacy’ seems to broaden every year. Few investments in yesterday’s technologies have in fact gone away, despite the best efforts of an industry constantly highlighting their deficiencies. Software systems continue to multiply. Partly, this is about business entropy; whether it’s because of mergers and acquisitions, or through the never-ending march of tactical business software applications to support new product lines, new promotional strategies or new communications channels. The supply of legacy systems for many large organisations in particular seems inexhaustible – and so integration continues to be front-and-centre as a technology capability with strategic importance.

New frontiers for integration

In the past five years, a new wave of platform and application endpoints has created a new wave of demand for technology integration – driven primarily by investments in mobile application platforms and software-as-a-service properties from the likes of Salesforce, Workday, NetSuite and more.

In particular, easy-to-acquire, easy-to-use SaaS applications (primarily to support sales and marketing activities, but not exclusively so) are creating new centres of gravity for technology spending, and in these centres, ‘standard’ IT concerns (relating to security, integration, data quality, compliance and so on) are easily, even if not deliberately, overlooked.

We need new capabilities to meet new integration demands

In pure functional complexity terms, the integration challenge created by this new wave of investment is less severe than the challenges created by the e-commerce investment wave of the early 2000s and the client-server investment wave of the mid-1990s. For one thing: new mobile application platforms and SaaS offerings typically ship with well-documented APIs that build on open standard communications protocols and easy-to-handle message formats. When integrating these assets, today’s integration platforms don’t typically need to bundle proprietary protocol adapters.

However at the same time, the broader requirement is more challenging.

Specifically: as endpoints proliferate and business and as technology change cycles continue to accelerate, a new kind of integration platform approach is vital. Integration of systems, applications, platforms and data sources needs to be far cheaper, easier and quicker than before. Yesterday’s high-cost, difficult-to-use platforms are naturally subject to tight control within enterprises. This is where Digital Integration Platforms come in.

Defining Digital Integration Platforms

Digital Integration Platforms enable teams to work together to design, deploy, operate, monitor and change integration software services that connect business applications, data sources and other software platforms – wherever they may be hosted. Digital Integration Platforms are delivered as-a-service, licensed on a subscription basis and priced according to usage.

We call Digital Integration Platforms “digital” because they exhibit qualities that are common to many kinds of modern digital products and services: they are licensed and delivered on an as-a-service basis and priced based on usage, and they’re designed to make them amenable to self-service adoption (where organisations can get started in delivering value with little formal training or experience).

Platforms” are places that collect, shape, and package resources to attract the creation and delivery of services. Digital Integration Platforms fit this definition because they provide digital ‘places’ that support the whole lifecycle of integration services – providing tools that assist in a cycle of activities, from integration design and development through operation to integration monitoring and change management.

Digital Integration Platforms are also often referred to as iPaaS (integration platforms-as-a-service) offerings.

Examples of vendors offering Digital Integration Platforms available to rent today include Dell (Boomi), Jitterbit (Harmony), SnapLogic (Enterprise Integration Cloud), Software AG (Software AG Integration Cloud), TIBCO (TIBCO Cloud Integration), Oracle (Oracle Integration Cloud Service), IBM (App Connect), Capgemini (Capgemini Enterprise iPaaS), Zapier and Mulesoft (AnyPoint).

The work that Digital Integration Platforms do

If you’re familiar with Enterprise Service Bus (ESB) technology and/or Extract, Transform & Load (ETL) tools then the core functional capabilities of Digital Integration Platforms will be familiar to you (see figure 1). Not surprisingly, different Digital Integration Platforms excel in supporting different kinds of use case.

Figure 1: Core functional capabilities of, and use cases for, Digital Integration Platforms

Source: MWD Advisors

As you can see from figure 1, Digital Integration Platforms typically offer two or more of four different classes of core integration capabilities:

  • Connection – enabling you to get data into and out of external systems. All Digital Integration Platforms provide this.
  • Simple transformation – enabling you to specify simple operations that change the structure of, or values within, data records – for example joining two fields together, searching and replacing strings, calculating date ranges, and so on. All Digital Integration Platforms provide this.
  • Complex transformation – enabling you to specify operations that work across groups of data records, and even groups of records of multiple types – for example sorting and filtering, aggregating data, summarising, averaging sets of values, splitting and recombining records, and so on.
  • Orchestration – enabling you to control complicated, multi-phase flows of automated integration and data transfer work across multiple applications and systems.

In addition, Digital Integration Platforms typically support the execution of integration logic according to one or more of three different triggering schemes:

  • Event-based – in which integration logic is triggered when some kind of external event occurs. All Digital Integration Platforms provide this.
  • Timed/manual – in which integration logic is triggered according to a scheme specified by a human operator.
  • Streamed – in which integration logic executes continuously on a stream of incoming data.

Key characteristics of Digital Integration Platforms

Although the core functional capabilities of Digital Integration Platforms are similar to those of yesterday’s ESBs and ETL tools, they differ (to a greater or lesser extent, depending on the provider in question) from those older technologies in terms of four key characteristics: speed, simplicity, abstraction and openness.

Speed and simplicity

The value propositions of most Digital Integration Platforms focus on speed and simplicity. They seek to enable people to address simple use cases with little-to-no technical design or development effort, and even when use cases require more technical work they seek to make that work as quick and simple as possible. They are frequently promoted as tools that ‘citizen integrators’ (people without technical integration training, but with some business analysis skills and domain understanding) can use without help from technical developers.

Many Digital Integration Platforms ship with pre-built components that can address simple use cases (for example, the sharing of prospect information between a popular digital marketing application and a popular SaaS property like Salesforce’s Sales Cloud) with minimal wizard-based configuration. They also often ship with tools that make it easy to discover the structure of application and platform interfaces, and automate the configuration of connections to those interfaces.

Abstraction

Beyond the use of pre-built components and discovery tools, Digital Integration Platforms deliver on their speed and simplicity capabilities by – wherever possible – abstracting away as much low-level technical detail as possible from the business of defining and changing integration services.

They typically do this by using graphical model-based approaches to specification; that is, you specify how integration services will work by using graphical drag-and-drop tools to visually link interfaces together, map data, and select and configure various integration functions (functions to manipulate data, handle errors, call out to other systems, control the overall flow of integration logic, and so on).

The best Digital Integration Platforms extend their use of abstract models beyond the business of defining integration services – using the same approach when you need to deploy, monitor, or change the configuration or functionality of integration services.

Openness

Digital Integration Platforms that are designed for enterprise use have to embrace openness, in more than one sense.

For one thing, these platforms need to enable you to extend or customise certain aspects of their functionality, to meet use cases that the core product isn’t built for. But more than that, they need to be open in the way they let you deploy and manage integration services.

For many organisations, it may be perfectly fine to run integration services on a central Digital Integration Platform instance hosted and managed by the vendor; but for many more, some integration services will be more appropriately deployed in their own data centres or on other public cloud services or hosting service provider installations.

The best Digital Integration Platforms for enterprises make it possible for you to choose where you want to deploy your integration services, while still making it straightforward to manage the lifecycles of those integration services (wherever they might be running).

Why are Digital Integration Platforms important?

Learning from the past

There is absolutely nothing wrong with the principles enshrined in the Service-Oriented Architecture (SOA) thinking that swept large organisations’ IT architecture and strategy functions between 2000-2010. Indeed, those principles continue to shape today’s API-centred platform and application service designs.

Where the industry collectively stumbled was in the nature of SOA implementations, and the constraints imposed by the enabling technology products available at the time. Infrastructure required to implement SOA tended to be expensive and difficult to work with, requiring significant technical training and making it difficult to deliver value quickly.

For most organisations, these challenges made SOA visions just too difficult to implement effectively. In no particular order:

  • IT organisations struggled to find projects that would deliver enough value to pay back the platform investment required. Many business integration demands went unmet.
  • Centres of Excellence (COEs) were vital to enable enterprises to deliver integration projects efficiently and consistently, but were embedded deep within IT organisations and focused mostly on controlling use of the platform rather than promoting its use.
  • Governance around the use of SOA platforms worked (often unintentionally) to discourage change.
  • Project backlogs grew to span many months, sometimes years.

Although Digital Integration Platforms at their core deliver many of the same functions as yesterday’s heavyweight SOA-era enterprise integration platforms, Digital Integration Platforms have the potential to enable organisations to deliver agile integration initiatives that deliver real business returns, because of their focus on simplicity, speed, abstraction and openness.

Helping create new contracts between business and technology teams

It’s all very well to talk about how digital technologies enable organisations to drive business-technology change in new ways, but we know that a great number of CIOs, and the IT organisations they manage, are under relentless pressure to meet their existing commitments and expectations. There’s little ‘headroom’ in the technology resources available to most organisations, and that makes any thought of real transformation very difficult indeed to prioritise.

In MWD Advisors’ Spring 2017 CIO survey, one of the most striking findings was that although 81% of CIOs expect to be taking more active roles in digital transformation and innovation initiatives over the coming 24 months, 69% also say that their own teams struggle to keep up with demand for new technology capabilities (and changes to those capabilities).

Figure 2: Relentless pressure on IT teams

Source: MWD Advisors

As we mentioned earlier in this report, this demand partly springs from a rise in net new business requirements (in large part for mobile business apps); but it’s also being driven by the implications of fast-evolving technology landscapes (see figure 2).

Enabling a positive, rather than a defensive, IT stance

A natural response to the challenge of managing demand in this new, uncertain, fast-moving world is for IT teams to resist change and isolate themselves – but this approach is futile.

Some CIOs and their teams attempt to use a mixture of persuasion, influence and force to dismantle what they see as unnatural centres of (SaaS-centred) technology procurement and use that are outside their control. Some, with board-level support, succeed: but the challenge is that ‘shadow IT’ is a symptom of the underlying ‘new normal’ in business technology – it’s not an isolated phenomenon. Because the underlying ‘new normal’ environment can’t be dismantled in any rational way (at least, not without sentencing the wider organisation to a severely handicapped future), any victory over it turns out to be temporary. And once the ‘new normal’ reasserts itself, the IT organisation’s reputation is even more tarnished.

A far more sustainable route to take revolves around acceptance and collaboration, though it requires maturity (for IT leaders to truly accept that they cannot control spending on, or use of, IT). Instead of fighting against ‘shadow IT’ centres, this route involves embracing all groups that need IT resources and capabilities, and working collaboratively with them to support them in their efforts, highlight potential risks and controls, and thereby demonstrate the value that a joined-up approach can bring.

An ‘acceptance and collaboration’ response is fundamentally about creating a new kind of contract between business and IT: a contract that revolves around joint working to serve the needs of customers and other stakeholders, rather than revolving around IT acting as a straightforward service provider to business teams.

However by itself, even the most open and positive approach to managing relationships with all sources of technology demand will not make the problem of managing demand go away, though it will make things easier.

What’s also needed are technology platforms that truly support collaboration between technologists and other businesspeople in meeting that demand, and that enables those groups to work together quickly to get things done.

Tackling the long tail of integration and automation projects

Most organisations of significant size struggle to meet demand for new technology, or technology change. Increasingly, business teams are taking responsibility for procuring ‘self-service’ platforms that can make them more productive, and then using those platforms to create their own solutions to their own problems. Although central IT provision can tackle the largest, most demanding problems with centrally-funded platforms, “long tail” problems may never get enough priority to be addressed.

Digital Integration Platforms are relatively simple (from a technology point of view) to implement and inexpensive, and offer a compelling way for business teams to solve integration and automation challenges that may otherwise remain unaddressed.

Because of this, today’s emerging Digital Integration Platforms are the natural successors to yesterday’s more heavyweight, expensive integration tools.

Download PDF version
RESEARCH REPORT // FREE