The IT world is moving forward fast. The digital transformation changes complete industries and peels away existing business models. Cloud services, mobile devices and the Internet of Things establish wild spaghetti architectures though different departments and lines of business. Several different concepts, technologies and deployment options are used. A single integration backbone is not sufficient anymore in this era of integration.
Different user roles need to leverage different tools to integrate applications, services and APIs for their specific need. A key for success is that all integration and business services work together across different platforms in a hybrid world with on premise and cloud deployments.
This article shows the different components available for a Hybrid Integration Architecture. The goal is not to discuss different vendor offerings but to explain different concepts and benefits of each component in general and how they relate to each other. Afterwards you can think about which components you might need for your scenario and do further research.
Hybrid Integration Platform
“Hybrid Integration Platform (HIP)” is a term coined by Gartner and other analysts. It describes different components of a modern integration architecture.
One of the keys for success in today’s more and more complex world is that different platforms work together seamlessly. Different components share interfaces and metadata, one single development environment and consolidated operations management. API Management is often used as a mediator between interfaces to encourage an agile and innovative enterprise culture – internally; as well as between partners; and for public external communication.
Leveraging a well-conceived hybrid integration architecture allows different stakeholders of an enterprise to react quickly to new requirements. Mission critical core business processes (also called “core services”) are still operated by the central IT department and change rather infrequently. On the other side, the line of business needs to try out new business processes quickly, or adapt existing ones, in an agile way without being delayed by frustrating rules and processes governed by the central IT. Innovation by a “fail-fast” strategy and creating so called “edge services” is getting more and more important to enhance or disrupt existing business models.
The following picture shows possible components of a Hybrid Integration Platform:
On Premise, Cloud Ready and Cloud Native Deployments
The following sections discuss these components in detail. As a preliminary point it is important to understand the differences between “on premise”, “cloud ready” and “cloud native” deployments as the cloud is changing integration architectures and processes significantly. New cloud platforms emerged to enable team development and operations of agile and flexible applications. The cloud platforms take over the responsibility for challenges, such as quick provisioning of new instances or re-assigning service calls in the case of errors.
On Premise: Classical deployment of applications in a private data centre on physical hardware, often leveraging virtualization. Additional hardware has to be bought and installed to scale up.
Cloud Ready: Deployment of applications via “Infrastructure as a Service” (IaaS) principle in the private cloud (e.g. using OpenStack) or public cloud (e.g. using Amazon Web Services, Microsoft Azure, Google Cloud Platform or others). The cloud provider offers as many hardware resources as needed, borrowed by the enterprise via a “pay per use” model. Existing applications are deployed like they were deployed on premise – without huge refactoring or architectural changes. The scalability and all its efforts such as service discovery, load balancing, elasticity (i.e. scale up and down), cluster management or fail-over have to be assured by the application team. “Just” the hardware resources can be gained quickly and easily by the cloud provider using scripts.
Cloud Native: Deployment of applications via “Platform as a Service (PaaS) principle in the private or public cloud (e.g. using Cloud Foundry or Red Hat’s OpenShift). In addition to taking care of the infrastructure, the PaaS solves challenges such as service discovery, load balancing, elasticity, cluster management or fail-over out-of-the-box. The application development needs to be adopted to cloud native concepts to allow agile development and quick innovation. In many cases, you prefer to build, develop, deploy and operate so-called microservices instead of realising larger applications, respectively monoliths, on top of a PaaS. Often, you leverage independent containers (e.g. Docker, CoreOS or CloudFoundry’s Warden) for building cloud native microservices or applications. Note that microservices and containers are two different things – they work very well together, but also without the other.
For a more detailed introduction to these key concepts of a hybrid integration architecture (and the benefits of microservices and containers in general) please take a look at the extensive article “A Cloud-Native Architecture for Middleware”. It explains these concepts in more detail and discusses several open source frameworks and products from different vendors.
Note that public cloud makes sense for many projects, however it is not the best choice for everything. As Massimo Pezzini from Gartner states on LinkedIn: “Over the next five years, the reality of most organisations will be hybrid cloud. Which will pose them the cloud-to-cloud-to-ground integration challenge. Which will push them towards the adoption of a hybrid integration platform strategy.”
Application Integration using an Integration Framework or Enterprise Service Bus
Application Integration existed for many years. Most enterprises have moved away from a batch-oriented integration architecture where they integrate applications using long running ETL (“Extract-Transform-Load”) processes taking minutes or even hours. Today’s integration usually leverages real-time integration using technologies such as web services or asynchronous messaging to always have up-to-date information in all relevant applications.
The tool of choice usually is either an open source Integration Framework such as Apache Camel or Spring Integration, or an Enterprise Service Bus (ESB) product such as MuleSoft Anypoint Platform, Talend ESB, WSO2 ESB, TIBCO ActiveMatrix BusinessWorks or Oracle Service Bus. These tools are usually used in bigger IT projects where mission-critical core business processes have to be integrated with a need for high availability, reliability and performance within the enterprise (“core services”). This affects many different technologies and applications to integrate standard software (e.g. CRM, ERP), legacy systems (e.g. mainframe), custom applications (e.g. Java, .NET) or external cloud services.
Most core services in industries such as finance, retail, airline or telecommunications run over an ESB cluster to leverage high performance, high availability, fault-tolerance and guaranteed delivery of transactions. A failure of the ESB cluster would disable bank transactions, air traffic or online shopping and create huge trouble for an enterprise. Most of the integration happens on premise in the private data centre. This makes sense and will probably not change in the next decades. The ESB cluster will stay the core integration solution.
Most ESB products became cloud ready in the meantime. However, many enterprises do not plan to move to the cloud beyond development and test environments with their core services. On premise deployments in the private data center are absolutely fine for some kind of projects – even in the “cloud era”.
At this point it should be highlighted that an ESB is not the complex, heavyweight beast (anymore) as many people think of it. This might have been true five to ten years ago – and the reason why many SOA projects failed at that time. Today most ESB products are much more lightweight and offer sophisticated tooling. If you still disagree (maybe because of your experiences from several years ago) please try out one of the modern products before saying “visual development does not work” or “everything is easier with writing source code”.
A modern ESB nowadays is used for the following tasks:
Integration with zero (or low) coding
Connectivity to all kinds of technologies, standard software and SaaS
Messaging infrastructure with high performance and high availability
Orchestration and choreography of services
Development of APIs and services
Scalable and lightweight platform
Independent deployment and operation of individual (micro)services
Automation leveraging continuous integration / continuous delivery
To conclude this section: Microservices do not spell the end of the ESB! Having said that, the trend in many new projects goes into the direction of more agile development and more flexible infrastructure. Microservices, container technologies such as Docker and cloud native architectures are getting more and more relevant in most enterprises. Therefore the “classical” ESB is not the right choice for every integration project. Let’s take a look at several other options in the next sections.
Application Integration on Top of a Cloud-Native PaaS Platform
Application Integration on top of a cloud-native PaaS platform is very similar to an on premise ESB and also used for implementing “core services”. Development is also done in the traditional IDE. However, the key difference of an on-premise ESB is that the solution is cloud-native. You use this kind of integration middleware to develop integration applications that are deployed natively onto a PaaS platform such as Cloud Foundry or OpenShift. It supports containers and microservices. Therefore, you can also leverage this platform to develop more agile or innovative “edge services” too. Another trend on the market is leveraging Container-as-a-Service (CaaS) instead of a self-managed PaaS platform. See for example Amazon EC2 Container Services (ECS) or Google Container Engine (GKE).
Some vendors offer a vendor-agnostic solution where you can deploy your integration applications anywhere without relying on a specific cloud platform or vendor. You can develop cloud-native services to be more agile and provide web scale:
Integration Apps and Services: Build consumable web APIs out of backend web services like ERP, CRM, order management using enterprise technologies like SOAP, SAP, Oracle, IBM MQ, etc.
Functional Microservices: Build apps focusing on business functionality without getting into code complexity
API Choreography Services: Visually choreograph APIs leveraging the integration tooling (e.g. process orchestration)
Many enterprises do not have a complete, long-term strategy yet. Hence it is important not to become dependent on a specific cloud platform but to develop cloud-platform-agnostic integration services.
TIBCO BusinessWorks Container Edition is an example for a cloud agnostic integration middleware supporting different PaaS and containers platforms such as Cloud Foundry, Docker, Kubernetes or AWS ECS. Another example is JBoss Middleware Services which allows the deployment of its middleware applications (including JBoss Fuse and A-MQ) onto OpenShift (which at it’s core is again Docker and Kubernetes). As you see many vendors are moving toward microservices and containers with their middleware.
Integration Platform as a Service (iPaaS)
An iPaaS Cloud Integration middleware is – contrary to a classic ESB or PaaS-based integration solution – a pure public cloud offering hosted by a specific vendor. The vendor takes care of cloud native features such as provisioning of infrastructure, elasticity or multi-tenancy. iPaaS focuses on integration of SaaS cloud services such as Salesforce or self-developed REST interfaces. Configuration of integration flows, life cycle management and service governance can be realised with a simple, intuitive web user interface.
Target audience for iPaaS is not necessarily the core integration developer. Instead it allows colleagues with less technical expertise (there sometimes called “ad-hoc integrator”) to define, deploy and monitor APIs and corresponding policies such as security requirements or throttling service level agreements. Independently of this, a developer can implement the service “in the background”. Using iPaaS allows the line of business to react quickly to new requirements or innovative ideas without struggling with the core IT team and its long release and quality management processes.
iPaaS can be used for both mission-critical core services and quickly changing respective innovative edge services. iPaaS has to work well together with other integration solutions in terms of out-of-the-box integration (e.g. import / export of integration services) and commercial support. This way you can easily develop an innovative iPaaS service and then deploy it on your on-premise ESB cluster later if the user rate increases significantly and the new service is promoted to an important core service.
The “Backends-for-Frontends (BOF) Pattern“ is another good use case for iPaaS. It exposes an existing service (i.e. backend) to various service consumers (i.e. frontends) in different “versions”, e.g. by offering different operations or parameters. The core service has just to be developed, operated and maintained once. However, various service consumers, such as different mobile devices or target audiences, perceive the service distinctly.
The iPaaS market is just emerging these days. Some examples are Dell Boomi, Informatica Cloud, MuleSoft Anypoint Platform, SnapLogic, Jitterbit or TIBCO Cloud Integration.
Integration Software as a Service (iSaaS)
In contrary to the above integration components iSaaS focuses on business users. They can create basic integration flows for personal or department interests in a very intuitive web user interface without any technical knowledge. This role is also called citizen integrator leveraging the do-it-yourself (DIY) principle.
Citizen Integrators build new integration flows by configuring them rather than developing and building them from scratch. There is no contact to technologies such as REST or JSON, you just leverage connectors to applications for daily business. For instance, a business user creates a daily automatic flow to synchronize data from a Google Sheet with Salesforce CRM. This removes the need to integrate these updates manually every day.
iSaaS integrations are complementary to on-premise, PaaS and iPaaS integrations. iSaaS integration flows should be viewed as “edge services” which are not strategic and mission-critical for the enterprise – but very relevant for the specific business user. Examples for iSaaS solutions are SnapLogic, TIBCO Simplr or IFTTT.
Integration Gateway for the Internet of Things
The previously discussed integration solutions are used for classical application integration. You integrate systems such as CRM, ERP, Mainframe or external cloud services so that all systems can communicate with each other without unmaintainable spaghetti architecture.
The Internet of Things (IoT) changes the role of integration. Here you need to integrate data directly on the devices as not all data should be sent to the private data center or public cloud. An IoT Integration Gateway interconnects all devices via various IoT Standards such as MQTT, CoaP, WebSockets, Bluetooth, ZigBee, NFC, RFID, USB, SPI, I2C or X-LINE
IoT raises several new challenges not relevant for classical application integration:
Devices are not connected to the cloud
Devices have low bandwidth to connect
Latency of connectivity is significant
Connectivity is not reliable
Connectivity is not cost-effective
The IoT Integration Gateway filters and aggregates data streams on site and sends only relevant information such as errors or alerts to the central integration platform or any service.
From a technical perspective you either use hardware from vendors such as Intel, ARM or Eurotech or you decide to implement a specific solution with an IoT gateway framework such as IBM’s NodeRED (implemented with node.js using JavaScript) or TIBCO’s Flogo (implemented with Google’s Go Programming Language). Both are available for free under open source license. The focus of these frameworks is a lightweight engine so that it can be deployed on very small devices with low hardware power. Development is done via intuitive web user interface and out-of-the-box connectivity to various IoT standards.
API Management for Core and Edge Services
The previously discussed integration alternatives allow the right choice for a specific integration scenario. Independently the trend is heading towards an “Open API Economy“ where services are exposed as API to other internal departments, partners or public developers.
API Management enables developers to reuse services to concentrate on new features, shorter time-to-market and innovation instead of recreating existing services again. Think about examples such as Paypal whose API is integrated into almost every online shop as payment option or Google Maps whose API is used in almost every website which includes a description of how to get somewhere.
API Management solutions include three components:
API Portal (on premise or public cloud) to expose internal services as product bundles for testing and consumption.
API Gateway (typically on premise, often deployed in the DMZ) to configure security (e.g. authentication, authorisation), transformations, routing or other service level agreements (e.g. throttling or caching).
API Analytics (on premise or public cloud) to analyse and optimise service calls – relevant for both service provider and service consumer.
Examples for API Management solutions are Apigee (Pure Player), Akana (Pure player, former SOA Software), Mashery (TIBCO) or 3scale (Red Hat). Sometimes a hardware API Gateway such as IBM DataPower Gateway is used as replacement or in addition to the software API Gateway.
Key for success in a hybrid integration architecture is a good cooperation between API Management and the integration solutions – no matter if it’s classical application integration, iPaaS or iSaaS. Out-of-the-box integration of these tools helps building a hybrid integration platform. For example, automatic deployment of a created service directly to the API Portal either via web user interface or scripting for Continuous Integration (CI) / Continuous Delivery support (CD) eases exposing APIs to other service consumers.
Integration of Data Streams using Streaming Analytics
Streaming Analytics (also called Stream Processing) is relevant for massive amounts of data streams and sensor data. You use continuous queries for sliding windows and correlations (e.g. for fraud detection or predictive maintenance) instead of using single transactions via messaging or request-response service calls (e.g. for a bank transaction or flight booking). Data streams are aggregated and correlated in real time while the data is in motion, i.e. before it is too late. See more details and real world use cases for streaming analytics in this InfoQ article: “Real-Time Stream Processing as Game Changer in a Big Data World with Hadoop and Data Warehouse”.
How is this related to a hybrid integration platform? Well, before analysing data streams you have to integrate all the different streaming interfaces such as messaging technologies, IoT standards, trading markets or social feeds. Therefore integration is a key topic and requires connectivity plus integration functionality such as transformations, filtering or sorting of events in real time.
From a technology perspective you often leverage big data frameworks such as Apache Hadoop or Spark in combination with streaming integration frameworks such as Apache Flume, Apache Nifi, Apache Storm, Apache Kafka or Concord. Middleware vendors offer powerful products for streaming analytics, for example IBM InfoSphere Streams, Software AG Apama oder TIBCO StreamBase. The latter also offers a free Accelerator for Apache Spark, which shows how an enterprise can leverage both commercial and open source frameworks together to create most business value. As an emerging market on top of streaming analytics, you can leverage live user interfaces for proactive actions. You can either build your own application using modern web frameworks such as angular.js or choose a tool such as Zoomdata, Striim, or TIBCO Live Datamart.
Process Integration with Business Process Management
Until now we just discussed automated integration. Beyond that, human interaction is relevant in almost every enterprise for exception handling and customer communication. Therefore Business Process Management (BPM) needs to be part of a complete Hybrid Integration Platform and work together seamlessly with other integration solutions.
Today BPM is more than just long-running BPMN processes with human interaction and calling SOAP or REST Web Services. More demanding scenarios have to be realized with a BPM component:
Case Management: Instead of pre-defined business processes (including branches and conditions) there is a need for more complex and flexible processes. Examples: Customer interaction in a call center; event of damage or loss in the insurance industry.
Intelligent BPM: The application of big data analytics and machine learning to implement intelligent business processes. Computers help humans to make the (probably) best decision in every single process instance. Example: Cross-selling of additional products or fraud detection.
Rapid Application Development Platform: Instead of building just a bunch of independent business processes a BPM platform is used to create complete (but manageable) solutions as web applications or mobile apps which allows making processes accessible to third parties more easy. Example: One small and simple application including all process steps to request a new credit card – instead of several uncoordinated different process steps.
The Need for a Hybrid Integration Architecture
This article shows that a single integration platform is not sufficient anymore in the era of cloud, mobile, big data and IoT – no matter if you think about on premise, public cloud or hybrid infrastructure. Differentiation between core services (focus on mission-critical enterprise services) and edge services (focus on agile development and innovation) is a key step towards a Hybrid Integration Platform.
Another key aspect is the difference between on premise and cloud-native infrastructure and software development. Microservices and containers allow for developing and operating lightweight and scalable services. But it also introduces completely new concepts whose complexity and drawbacks should not be underestimated. You also have to think about the trade-offs and make the right decision for your project.
Independently of these topics there is a significant trend towards an Open API Economy to expose internal services to everybody due to motivation for focusing on business problems, creating added value and innovate thinking with “fail fast” methodology.
Streaming Analytics for analysing data streams and Process Integration for human interaction are also part of a modern Hybrid Integration Architecture to be able to realise all the different integration scenarios in the on-going digital transformation.
The post The Need for a Hybrid Integration Architecture appeared first on Voxxed.