The IT world is moving forward rapidly. The digital transformation changes complete industries and peels away existing business models. Cloud services, mobile devices, and the Internet of Things establish wild spaghetti architectures through different departments and lines of business. Several different concepts, technologies, and deployment options are used. A single integration backbone is not sufficient in this era anymore.
A hybrid integration platform for core and edge services
“Hybrid Integration Platform (HIP)” is a term coined by Gartner and other analysts. It describes different components of a modern integration architecture. A key for success in today’s complex world is different integration platforms working together seamlessly. Read Gartner’s report “How to Implement a Hybrid Integration Platform to Tackle Pervasive Integration” to understand the new challenges of most enterprises in more detail.
Leveraging a well-conceived hybrid integration architecture allows different stakeholders of an enterprise to react quickly to new requirements. Mission critical core business processes (“core services”) are still operated by the central IT department. However, even these services are being changed much more frequently than they were just a few years ago and the organization must continually push for these to be agile.
On the other side, the line of business needs to try out new or adapt existing business processes quickly in an agile way without the use of delaying—and often frustrating—rules and processes governed by the central IT. Innovation by a “fail-fast” strategy and creating so-called “edge services” are becoming important to enhance or disrupt existing business models.
The following picture shows possible components of a hybrid integration platform:
This blog explains the different components of a hybrid integration architecture, their deployment models, when to use them, and their target audience. Afterward, each section shows how TIBCO maps to these components.
Application integration with an enterprise service bus (ESB)
When to use it:
ESBs are typically used in larger IT projects where mission-critical core business processes have to be integrated with a need for high availability, reliability, and performance within the enterprise (“core services”). This affects many different technologies and applications to integrate standard software (e.g. CRM, ERP), legacy systems (e.g. mainframe), custom applications (e.g. Java, .NET), and external cloud services.
Deployment model and target audience:
Integration specialists with technical experience implement ESB clusters to leverage high performance, high availability, fault-tolerance and guaranteed delivery of transactions. Most of the integration happens on premise in the private data center. This makes sense and will probably not change in the coming decades. The ESB cluster will stay the core integration solution. Therefore, even “microservices do not spell the end of the ESB!”
You can deploy to the cloud, but this is more “cloud-washed”, but not cloud-native deployment and operations! However, this also makes sense sometimes, e.g. for DEV or TEST instances or to use the IaaS capabilities without necessarily looking for leveraging cloud-native features like automatic elasticity or service dynamic discovery.
TIBCO’s offering
TIBCO ActiveMatrix BusinessWorks (BW6) is a leading integration and service delivery platform. A key strength of BW6 is its distributed nature of BW and EMS deployments instead of using a central cluster with several nodes.
Cloud-native application integration
When to use it:
Integration in a cloud-native environment is quite different than your traditional approach to application integration. Usually, companies leverage platforms-as-a-service (PaaS) such as Cloud Foundry, Kubernetes, OpenShift, or Apache Mesos.
The applications created using these cloud-native application integration tools run natively within these PaaS environments and therefore benefit from the tools offered by these platforms. This means the PaaS takes care of provisioning infrastructure and solves challenges such as service discovery, load balancing, elasticity, cluster management, or fail-over out-of-the-box. It also allows more agile development to implement, adopt, and scale new features or innovations quickly and efficiently.
The application development and architecture needs to be adopted to cloud-native concepts to allow agile development and quick innovation. Typically, you prefer to build, develop, deploy, and operate so-called microservices instead of realizing larger applications respectively monoliths on top of a PaaS. This especially includes applying the concepts behind “The Twelve-Factor App” principles, which recommends best practices for cloud-native applications such as stateless services, explicitly declared and isolated dependencies, or environment-independent backing services. Often, you leverage independent containers (e.g. Docker, CoreOS or CloudFoundry’s Warden) for building cloud-native microservices or applications. Automation using DevOps and continuous integration/continuous delivery (CI/CD) is another key concept for successful cloud-native middleware projects.
Deployment model and target audience:
Core IT is adopting these PaaS platforms, but they are doing so to drive the agility for all developers, including integration specialists and ad-hoc integrators. As a result, both the traditional integration teams, as well as the developers within line of business, are able to benefit from these platforms and the capabilities offered by the integration technologies. In this respect, the organization is becoming a cloud provider themselves for the internal developers.
PaaS are widely adopted on-premise, in the public cloud, or hybrid deployments. In addition, several Container-as-a-Service offerings are emerging these days (e.g. Amazon EC2 Container Service). However, many enterprises do not have a complete, long-term strategy yet regarding cloud or hybrid deployment architectures. It is important not to become dependent on a specific cloud platform or container technology, but to develop cloud-platform-agnostic integration services, which can be moved from one to another platform without effort or redevelopment.
In addition, PaaS-based middleware should support and integrate mature open source frameworks for cloud-native environments such as Spring Cloud Config, Consul, Eureka or Hystrix instead of re-inventing the wheel and introducing additional complexity. The article “A Cloud-Native Architecture for Middleware” explains the concepts behind cloud-native platforms and deployment options in much more detail.
TIBCO’s offering:
TIBCO BusinessWorks Container Edition (BWCE) allows cloud agnostic integration projects supporting different PaaS and containers platforms such as Cloud Foundry, Docker, Kubernetes or AWS ECS. BWCE also supports available cloud-native frameworks and design patterns. For example, you can leverage service discovery frameworks such as Consul and Eureka or resiliency patters like circuit breaker with Hystrix (see BWCE 2.1 release notes).
Integration platform as a service (iPaaS)
When to use it:
An iPaaS cloud integration middleware is a pure public cloud offering hosted by a specific vendor—in contrary to a classical ESB or cloud-native PaaS-based integration middleware. Using iPaaS allows the line of business to react quickly to new requirements or innovative ideas without struggling with the core IT team and its long release and quality management processes. Sometimes, enterprises also replace their on-premise deployments completely with iPaaS to outsource the burden and complexity of operations management. Therefore, iPaaS can be used for both mission-critical core services and quickly changing respectively innovative edge services.
Deployment model and target audience:
The vendor takes care of cloud-native features such as provisioning of infrastructure, elasticity or multi-tenancy. Like application integration on top of a PaaS, this needs to be a real cloud-native enterprise-grade runtime and not just a “cloud washed” offering. Otherwise, it is not possible to scale out quickly and easily while still keeping high demands regarding stability and resiliency of integration services.
Target audience for iPaaS is not necessarily the integration specialist with extensive technical experience. Instead, it also allows colleagues with less technical expertise (sometimes called “ad-hoc integrator”) to define, deploy, and monitor services and APIs with its corresponding policies such as security requirements or throttling service level agreements. Ad-hoc integrators often do not use the more powerful IDE but an intuitive and simple-to-use web user interface.
TIBCO’s offering:
TIBCO Cloud Integration (TCI) is TIBCO’s iPaaS offering. iPaaS has to work well together with other integration solutions. This way you can easily develop an innovative iPaaS service and then deploy it on your on-premise ESB cluster later if the user rate increases significantly and the new service is promoted to an important core service.
A key benefit to TIBCO’s offering is we share a common design experience, including the ability to move projects between environments. Each runtime was designed for the specific environment and use case they are intended for.
You can use the same IDE and visual coding concepts for all three offerings and import services and projects from one offering respectively deployment option to another (with a few limitations due to some different concepts).
For example, you can develop an edge service with TCI for trying out a new innovative idea. Later, if the service gets successfully adopted and creates revenue, you can simply move the service to BWCE or BW6 in the cloud or on premise.
Integration software as a service (iSaaS)
When to use it:
iSaaS tools are designed for the end user and make it very easy to integrate data between various cloud services, even though the end user does not think of this as integration at all. They simply need to share information between systems and want to eliminate the redundant, time-consuming copying they use today.
iSaaS offerings are complementary to on-premise, PaaS, and iPaaS middleware and serve “edge services” which are not strategic and mission-critical for the enterprise—but very relevant for the specific business user or groups within a line of business. iSaaS focuses mostly on the integration of cloud services and therefore offer plenty of SaaS connectors. The simplicity does limit what is possible with these integration flows, so if high control (customized integration) is needed then an iPaaS may be required.
For instance, a business user can create a daily automatic flow to synchronize data from a Google Sheet with Salesforce CRM. This removes the need to integrate these updates manually every day and saves a lot of time to focus on other topics.
Deployment model and target audience:
iSaaS is hosted and operated by the specific vendor. In contrary to the above-discussed integration components, iSaaS focuses on business users (also called citizen integrators in this context). These employees can create basic integration flows for personal or departmental interests in a very intuitive web user interface without any technical knowledge or help from IT.
TIBCO’s offering:
TIBCO Simplr allows business users to connect their many cloud services in a very intuitive web user interface without coding skills. It also allows them to create forms in the same way to be able to interact with other humans and go beyond automatic integration scenarios. The Spotfire connector allows users to do data discovery from sources such as Simplr Forms, Marketo, JIRA, etc. to find insights without any technical knowledge.
IoT integration gateway
When to use it:
The Internet of Things (IoT) changes the role of integration. It raises several new challenges not relevant for classical application integration:
Devices are not connected to the cloud
Devices have low bandwidth to connect
Latency of connectivity is significant
Connectivity is not reliable
Connectivity is not cost-effective
Here you need to integrate data directly on the devices, as not all data should be sent to the private data center or public cloud. An IoT integration gateway interconnects all devices via various IoT standards such as MQTT, CoaP, WebSockets, Bluetooth, ZigBee, NFC, RFID, USB, SPI, I2C, or X-LINE. The runtime has to be very lightweight so that it can be deployed even on very small devices with very low resources and memory.
Deployment model and target audience:
Integration specialists and developers use the IoT integration gateway to filter and aggregate data streams “on site at the edge” and send only relevant information such as errors or alerts to the central integration platform or any service. Development can be done via intuitive web user interface and out-of-the-box connectivity to various IoT standards. Though, these offerings also allow (or sometimes require) some kind of coding.
TIBCO’s offering:
TIBCO’s Project Flogo is implemented with Google’s Go programming language and is fully open source. It offers a very lightweight runtime to build IoT integrations at the edge (e.g. filtering and aggregating data and sending only relevant information to the cloud or enterprise network).
As Project Flogo is very lightweight (in contrary to similar projects which have at least 20x more memory footprint), you can also leverage it to develop microservices with very low resource usage. This is an ideal fit for cloud-native platforms or even serverless architectures.
Project Flogo perfectly completes the integration portfolio together with BW6, BWCE, TCI, and Simplr.
API management
When to use it:
Until now, we have been talking about creating services that bring together two or more applications or sources of data. The trend is heading toward an “Open API Economy“, where services are exposed as API to other internal departments, partners, or public developers. API management provides the controls organizations need to expose or leverage APIs.
Think about examples such as Paypal whose API is integrated into almost every online shop as a payment option or Google Maps whose API is used on almost every website which includes a description of how to get somewhere.
Deployment model and target audience:
The target audience is the line of business, which leverage API portals to think about new digital products to increase revenue, make customers happy, or build new innovative mashups. API management portal and runtime are usually operated in the cloud to leverage elasticity and scalability for the unpredictable and often changing number of API calls. The API gateway is often deployed on premise to ensure security and other service level agreements within the firewall of an enterprise.
The key for success in a hybrid integration architecture is a good cooperation between API Management and the different integration components. This enables developers to reuse services to concentrate on new features, shorter time-to-market and innovation instead of recreating existing services again.
TIBCO’s offering:
TIBCO Mashery is the leading API management solution on the market. It offers out-of-the-box integration with all of TIBCO’s integration components (BW6, BWCE, TCI) via web user interfaces and command line / scripting tools for automation and DevOps. For example, you can develop a new service with TIBCO Cloud Integration and automatically offer it via your API portal to other users.
Complementary Add-Ons: Streaming Analytics and BPM
Streaming analytics and business process management are not part of application integration in the narrower sense, but are relevant for a hybrid integration architecture. Streaming analytics (sometimes called stream processing) is used for integrating massive amounts of data streams and sensor data while the data is still in motion. You use continuous queries for sliding windows and correlations (e.g. for fraud detection or predictive maintenance) instead of single transactions via messaging or request-response service calls (e.g. for a bank transaction or flight booking). See more details and real world use cases for streaming analytics in this InfoQ article: “Real-Time Stream Processing as Game Changer in a Big Data World with Hadoop and Data Warehouse”.
TIBCO StreamBase is a leading offering with enterprise scale, maturity, and ease-of-use. TIBCO Live Datamart offers real-time visual analytics for monitoring and making proactive actions by humans on top of the streaming engine.
Human interaction is relevant in almost every enterprise for exceptional handling and customer communication. Therefore, business process management (BPM) needs to be part of a complete hybrid integration platform and work seamlessly with other integration solutions. Today, BPM is more than just long-running BPM processes with human interaction and calling SOAP or REST web services. More demanding scenarios have to be realized with a BPM component, including case management, intelligent business processes, rapid application development platforms for web and mobile apps, and self-service BPM as SaaS.
TIBCO ActiveMatrix BPM is a leading BPM offering and highly integrated with TIBCO’s integration components. It includes sophisticated support for process modeling, resource planning, case management, rapid application development, and many other pretty cool features.
TIBCO Nimbus Maps can be used as self-service BPM SaaS offering for the business user. It enables teams to define, simplify, share, and change their processes in minutes—without the need to use a powerful BPM engine or extensive modeling standard like BPMN.
The need for a hybrid integration architecture
This blog post showed why a single integration platform is not sufficient anymore in the era of cloud, mobile, big data, and IoT. Differentiation between core services with a focus on mission-critical (but also more agile) enterprise services and edge services with a focus on innovation and very agile development is a key step towards a hybrid integration architecture.
TIBCO offers all key components for such a hybrid integration platform. Each component focuses on its specific use cases and deployment options—instead of just offering “cloud-washed” or “simply containerized” solutions. In addition, all of TIBCO’s components are highly integrated so they work together to reduce complexity and efforts for the whole team, including integration specialists, ad-hoc integrators, and citizen integrators (aka business users).
Are you just beginning your journey with a hybrid integration architecture? Feel free to contact me to discuss your architecture, challenges, and questions. If you want to discover some components on your own, then check our new and growing TIBCO Community Wiki. It already contains plenty of information about the discussed components. You can also ask questions in the answers section to get a response by a TIBCO expert or other community members.