Temporal

Temporal offers a software platform that enables applications and services, such as microservices orchestration, distributed transactions, infrastructure provisioning, monitoring and polling, data pipelines, long-running processes, DSL workflows, and scalable actor systems.

Temporal overview

The Temporal solution consists of a programming framework (Temporal SDK) and a back-end service (Temporal Server). Temporal is compatible with a variety of software development kits (SDKs), including Java, Go, PHP, and NodeJS.

The Temporal Server provides a virtual memory that is not linked to any specific process. It preserves the full application state (including function stacks with local variables) across all kinds of hosting and software related failures.

A Temporal SDK enables users to write application code using a programming language, while the Temporal Server handles the durability, availability, and scalability of the application.

In terms of the CAP theorem, Temporal clusters offer availability and consistency. Temporal allows for activities to be retried and resources to scale horizontally. Due to this, availability loss does not result in a fault, but in increased latency, since the network failures are prevented from reaching the application level. If persistence nodes are lost or unreachable, the user's workflows will not progress, but the data's consistency will be preserved. In addition, the optional 'Multi-cluster Replication' feature increases availability of the system.

Temporal SDKs enable users to build applications around a set of key concepts:

• Workflows—functions or object methods that are the entry point and base of applications

• Activities—functions or object methods that handle non-deterministic business logic

• Workers—processes running on physical or virtual machines that execute workflow and activity code

• Signals—write-only calls to workflows that can update the variable values and workflow state

• Queries—read-only calls to workflows that can retrieve function return values and workflow state.

• Task queues—a routing mechanism that enables load balancing

Temporal is used for orchestrating microservices. It aims to complete workflows (defined in general-purpose programming languages) of any size and complexity and to simplify defining workflow compensation logic with native Saga pattern support. It also offers built-in support for exponential activity retries, whereby the user can define retries, rollbacks, cleanup, and human intervention steps in the event of failure.

Temporal provides full visibility into end-to-end workflows that can span multiple services. It is designed to make microservices orchestration manageable by providing insight into each workflow's state.

Temporal provides a toolset for managing the unpredictability of external services via mechanisms such as built-in execution guarantees, exponential activity retries, timeouts, and more.

Temporal workflows can express complex decision trees using a general-purpose programming language. The platform offers integrated support for long-running operations, polling, responding to events, automatic retries, and more to ensure provisioning flow. This means that if a provisioning workflow were to fail in the midst of operation, the platform would correct the error by restarting the flow at the point of interruption. Temporal can also route activity execution to a specific process or host, which may be used in a variety of provisioning scenarios.

Checkr migrated their workflows from the Apache Kafka platform to Temporal. Because it was not practical to migrate all of Checkr's workflows at once, specific components and flows within the system were migrated one by one. Each migration constituted refactoring a single section of their data pipeline to use Temporal workflows instead of the existing Kafka based solution. This enabled Checkr to migrate at their own pace, circumventing interference with the live system.

In addition to transferring existing aspects of the pipeline, Checkr began encouraging engineers to build new features and flows with Temporal. Following an increase in internal adoption, Checkr developers voluntarily decided to implement all new data sources used during background checks with Temporal, which is not a requirement imposed by the platform.

As Box continued to scale, the company needed to support larger customers with greater storage demands. In order to achieve this, a critical part of Box's core architecture had to be reconstructed in a way that both enhanced the user experience and improved the efficiency of product teams.

Box opted to migrate orchestration and service level business logic. The team started with an in-memory simulation to validate that the API (application programming interface) and architecture would meet their needs. Once the baseline compatibility was proven, rollout began in a controlled and step-wise manner, tenant by tenant.

At first, deployments were only made to internal tenants, and results were verified against the original solution that was still running the majority of the workload. After Temporal had been validated against production workloads, more and more tenants within Box were migrated. Subsequently, Box's teams initiated adoption of the product at all levels of the infrastructure.

Coinbase used the Temporal solution in order to re-architect their core transaction workload. Due to the scale of Coinbase's operations, a migration strategy was needed that would enable them to progressively shift their critical transaction workload. Coinbase engineers began iteratively replacing each existing component with a Temporal workflow.

The engineers availed of Coinbase's open-source deployment system to build containers for each of the translated workflows and make them available via blue-green deployment. Each new workflow was put behind a feature flag for a test period before being fully integrated. In a relatively short span of time, the majority of existing workflows had been successfully migrated to Temporal. Coinbase was particularly satisfied with the visibility provided by Temporal, which had not been a feature of the company's previous system.