---
title: Continuous Deployment in Mission-Critical Enterprise SaaS Software - Optibus
description: Continuous Deployment in Mission-Critical Enterprise SaaS Software
image: https://blog.optibus.com/hubfs/Imported_Blog_Media/article2.jpg
---

[![optibus-logo-2019](https://blog.optibus.com/hs-fs/hubfs/optibus-logo-2019.png?width=180&height=49&name=optibus-logo-2019.png "optibus-logo-2019")](https://www.optibus.com/)

[![optibus-dark-logo](https://blog.optibus.com/hubfs/optibus-dark-logo.svg "optibus-dark-logo")](https://www.optibus.com/)

- Product 
    - [Platform](https://www.optibus.com/product/platform/)
    - [Passenger Information](https://optibus.com/product/passenger-information/)
    - [Planning](https://www.optibus.com/product/planning/)
    - [Control & Monitoring](https://optibus.com/product/control/)
    - [MagicEye - Driver Safety](https://optibus.com/product/magic-eye/)
    - [Operations](https://www.optibus.com/product/operations/)
    - [Scheduling](https://www.optibus.com/product/scheduling/)
    - [Driver App](https://www.optibus.com/product/driver-app/)
    - [Rostering](https://www.optibus.com/product/rostering/)
    - [Optibus AI](https://optibus.com/product/ai/)
- Solutions 
    - [Agencies](https://optibus.com/solutions/optibus-for-agencies/)
    - [Win Tenders](https://optibus.com/solutions/optibus-for-tenders/)
    - [Operators](https://optibus.com/solutions/optibus-for-operators/)
    - [Electrify Fleet](https://optibus.com/solutions/optibus-for-electrical-vehicles/)
    - [Consultancies](https://optibus.com/solutions/optibus-for-consultancies/)
    - [Enhance Passenger Experience](https://optibus.com/enhance-passenger-experience/)
    - [Reduce Costs](https://optibus.com/solutions/reduce-operating-costs/)
- Resources 
    - [Blog](https://blog.optibus.com/)
    - [eBooks](https://www.optibus.com/resources/ebooks/)
    - [Webinars](https://www.optibus.com/resources/webinars/)
    - [Case Studies](https://www.optibus.com/resources/case-studies/)
    - [Videos](https://www.optibus.com/resources/videos/)
- Company 
    - [About](https://www.optibus.com/about-the-company/)
    - [Careers](https://www.optibus.com/careers/)
    - [Contact Us](https://www.optibus.com/about/contacts/)
    - [Events](https://www.optibus.com/events/)
    - [Media](https://www.optibus.com/news-events/media-coverage/)
- EN 
    - [English](https://blog.optibus.com/)
    - [Español América Latina](https://blog.optibus.com/es/)
    - [Español Europa](https://blog.optibus.com/es-es/)
    - [Deutsch](https://blog.optibus.com/de/)
    - [Français](https://blog.optibus.com/fr/)
    - [Português Brasil](https://blog.optibus.com/pt-br/)
    - [Português Portugal](https://blog.optibus.com/pt-pt/)
    - [日本語](https://blog.optibus.com/ja/)
    - [Italiano](https://blog.optibus.com/it/)
    - [Русский](https://blog.optibus.com/ru/)
    - [Türkçe](https://blog.optibus.com/tr/)
    - [български](https://blog.optibus.com/bg/)
    - <https://blog.optibus.com/engineering/continuous-deployment-in-mission-critical-enterprise-saas-software#empty>

[Request a demo](https://optibus.com/request-a-demo/)

# Continuous Deployment in Mission-Critical Enterprise SaaS Software

![Eitan Yanovsky](https://blog.optibus.com/hs-fs/hubfs/0.jpg?width=100&height=100&name=0.jpg)

#### Eitan Yanovsky

March 31, 2021

[![Share on Facebook](https://blog.optibus.com/hubfs/fb_social_icon.svg)](http://www.facebook.com/share.php?u=https%3A%2F%2Fblog.optibus.com%2Fengineering%2Fcontinuous-deployment-in-mission-critical-enterprise-saas-software%3Futm_medium%3Dsocial%26utm_source%3Dfacebook) [![Share on LinkedIn](https://blog.optibus.com/hubfs/linkedin%20logo_icon.svg)](http://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fblog.optibus.com%2Fengineering%2Fcontinuous-deployment-in-mission-critical-enterprise-saas-software%3Futm_medium%3Dsocial%26utm_source%3Dlinkedin) [![Share on Twitter](https://blog.optibus.com/hubfs/twitter%20logo_icon.svg)](https://twitter.com/intent/tweet?original_referer=https%3A%2F%2Fblog.optibus.com%2Fengineering%2Fcontinuous-deployment-in-mission-critical-enterprise-saas-software%3Futm_medium%3Dsocial%26utm_source%3Dtwitter&url=https%3A%2F%2Fblog.optibus.com%2Fengineering%2Fcontinuous-deployment-in-mission-critical-enterprise-saas-software%3Futm_medium%3Dsocial%26utm_source%3Dtwitter&source=tweetbutton&text=Continuous%20Deployment%20in%20Mission-Critical%20Enterprise%20SaaS%20Software%20-%20Optibus) [![Share on Email](https://blog.optibus.com/hs-fs/hubfs/email_icon-1.png?width=24&name=email_icon-1.png)](mailto:?subject=Check%20out%20https%3A%2F%2Fblog.optibus.com%2Fengineering%2Fcontinuous-deployment-in-mission-critical-enterprise-saas-software%3Futm_medium%3Dsocial%26utm_source%3Demail%20&body=Check%20out%20https%3A%2F%2Fblog.optibus.com%2Fengineering%2Fcontinuous-deployment-in-mission-critical-enterprise-saas-software%3Futm_medium%3Dsocial%26utm_source%3Demail)

Continuous deployment in mission-critical enterprise SaaS software sounds scary, right? How do you make sure (as much as possible) that there are no regressions introduced when new code is being deployed in such critical pieces of software?

This is one of the challenges that we continuously face. The [Optibus platform](https://blog.optibus.com/?page_id=6650&hsLang=en) is used by the people in charge of planning and operating mass transit (public transportation), and they rely on our system to plan and execute the operation of moving millions of people on a daily basis. If we deploy a change in the code that introduces an issue that prevents them from working, the impact is huge. Moreover, what if the change causes a degradation of the quality of our algorithms in a specific scenario, a minor change that is only different by 0.5% from their previous efficiency levels? This difference may seem insignificant but can easily create excessive costs in the millions. Not to mention the environmental impact of inefficient operations emitting more CO2 into the atmosphere.

So how can you continue to ship at a high pace while de-risking the code you introduce?

Having good test coverage is not enough. Among other things, we are applying TDD (Test Driven Development) methodologies across our platform, but these are specifically engineered cases that are mainly used to build good code and have high coverage of lines of code. That doesn’t really cover all the endless combinations of real-data scenarios — and they never will since new combinations are born every minute.

One common approach to handle this is canary deployment, which works very well. However, it relies on a few things to make it work:

1. Having a significant number of users that reach the same flows (edge cases) so you can really divert traffic in gradual ways and make sure you cover these cases
2. The ability to detect problems very fast — mainly detect failures
3. Having some of your users experience a problem as part of the process

In our case, none of the above are true. Our users are the people in charge of operating mass transit, so the number of our concurrent users is not in the tens of thousands. As a result, we don’t have a large enough group to statistically measure and cover enough edge cases in a very short period.

Detecting a non-trivial regression, such as performance degradation, for operations that can take a few minutes to run is hard. First of all — it is not an error, and how can you predict the expected run-time for this operation? Moreover, detecting a behavior change that yields a successful result (but with a slightly different output) is not trivial. In canary deployment, you monitor for errors to detect a problem — you don’t have an expected baseline since this is live data.

Finally, it is a mission-critical system. A user experiencing an error may not be able to serve the public in time, so we have very little tolerance for user failures.

**Playback framework**

With this in mind, we developed a different approach, which is similar to the “testing in production” concept but not actually **in** production. We created the [playback](https://github.com/Optibus/playback) open-source framework, which is a decorator-based Python library that records all the traffic in production for all of our Python-based services, intercepting all inputs and outputs of each of our different services independent of how the service is being invoked (HTTP, event-based, producer-consumer, etc.). These recordings are later fully replayable on new code versions as a standalone operation, not relying on any external data state. This allows us to do a full comparison between the recorded output and the playback output to check for regressions, degradations, fixes, or improvements.

What is a regression? This is not always as simple as just checking for equality on the outputs because its service has its own format of outputs. For instance, if an endpoint didn’t return an error but a result that is slightly different, is it a regression, an improvement, or just an equivalent result with different IDs? This is also where the “out of the box” canary deployment doesn’t suit well.

This comparison is done in the playback framework by using dedicated plugins for the different operation types, which have knowledge on what is considered an equivalent output, a regression, or an improvement.

The developer can use the framework to run playback of specific changes before merging, get specific recording samples that have properties s/he wants to test the code, and so on.

In addition, this framework allows us to sample enough recorded data from production that is uniformly distributed across the different customers and use cases — making sure we cover enough edge cases and user flows and replay them as part of our CI/CD on the new code version before we merge/deploy it. This way we are automatically validating that there is no regression and conducting complicated comparisons on short and long running operations. The result is that we are shipping code at high frequency without compromising on quality.

We are essentially “testing in production” without being **in** production.

**Playback High Level Architecture**

This is an applicative framework that you need to integrate within your code base in order to annotate the service entry points and the inputs and outputs that need to be captured. Its main components are:

- Tape Recorder — Records and replays entire operations
- Tape Cassette — The driver that is used by the Tape Recorder to store and fetch recordings
- Equalizer — Runs comparisons between recorded operations and playback results on new/different code version
- Playback Studio — A utility that makes it easy to run suites of regression checks of playbacks

You can see the full basic [example](https://github.com/Optibus/playback/blob/main/examples/basic_service_operation.py) of how to use the framework to capture an operation and play it back. Or an end-to-end [example using flask](https://github.com/Optibus/playback/tree/main/examples/flask) as a web server.

This is a snippet that demonstrates how to plug the framework into the operation that you wish to record and playback:

![](https://blog.optibus.com/hs-fs/hubfs/Imported_Blog_Media/Eytan-Blog-Post-1.jpg?width=719&height=835&name=Eytan-Blog-Post-1.jpg)

The InMemoryTapeCassette is useful for testing, but in production you would typically use the S3TapeCassette which uses AWS S3 to store and fetch recordings.

This is how you can use the Equalizer to run a comparison of many recordings:

![](https://blog.optibus.com/hs-fs/hubfs/Imported_Blog_Media/Eytan-Blog-Post-2.jpg?width=751&height=648&name=Eytan-Blog-Post-2.jpg)

You can check out [https://github.com/Optibus/playback/tree/main/examples/flask](https://github.com/Optibus/playback/tree/main/examples/flask) for a more comprehensive example. Contributions are welcome.

 Topics: [Engineering](https://blog.optibus.com/topic/engineering?hsLang=en)

![](https://blog.optibus.com/hubfs/new%20design%20blog/illustration.svg)

### Get transportation industry updates

(But not too often, don't worry!)

Sign up for insights

## Recent Posts

- #### [Optibus Launches Digital Vacation Bidding to Answer Bus Driver Demands for Self-Service Time-Off Control](https://blog.optibus.com/optibus-launches-digital-vacation-bidding-to-answer-bus-driver-demands-for-self-service-time-off-control?hsLang=en)
  
  [Read More](https://blog.optibus.com/optibus-launches-digital-vacation-bidding-to-answer-bus-driver-demands-for-self-service-time-off-control?hsLang=en)
- #### [Optibus Battles Driver Turnover and Overtime Expenditure with New Intelligent Driver and Vehicle Allocation](https://blog.optibus.com/new-intelligent-driver-and-vehicle-allocation?hsLang=en)
  
  [Read More](https://blog.optibus.com/new-intelligent-driver-and-vehicle-allocation?hsLang=en)
- #### [Three Minutes Away, Forever: The Anatomy of a Ghost Bus](https://blog.optibus.com/anatomy-of-a-ghost-bus?hsLang=en)
  
  [Read More](https://blog.optibus.com/anatomy-of-a-ghost-bus?hsLang=en)

## Popular Content

[![Transvida Consortium, part of SPTrans, picks Optibus for smarter planning and scheduling in São Paulo](https://blog.optibus.com/hubfs/Transvida%20Consortium%20client%20announcement%20graphic%20-%20website%20(1399%20%C3%97%20839%20px)-2.png)](https://blog.optibus.com/transvida-consortium-part-of-sptrans-picks-optibus-for-smarter-planning-and-scheduling-in-s%C3%A3o-paulo?hsLang=en)

## [Transvida Consortium, part of SPTrans, picks Optibus for smarter planning and scheduling in São Paulo](https://blog.optibus.com/transvida-consortium-part-of-sptrans-picks-optibus-for-smarter-planning-and-scheduling-in-s%C3%A3o-paulo?hsLang=en)

[Read More](https://blog.optibus.com/transvida-consortium-part-of-sptrans-picks-optibus-for-smarter-planning-and-scheduling-in-s%C3%A3o-paulo?hsLang=en)

[![Optibus appoints Simon Pearce as Chief Revenue Officer to drive sustainable commercial growth](https://blog.optibus.com/hubfs/Simon%20Pearce%20Appointment%20graphic%20blog%20%281500%20x%20875%20px%29%20%287%29.png)](https://blog.optibus.com/optibus-appoints-simon-pearce-as-chief-revenue-officer-to-drive-sustainable-commercial-growth?hsLang=en)

## [Optibus appoints Simon Pearce as Chief Revenue Officer to drive sustainable commercial growth](https://blog.optibus.com/optibus-appoints-simon-pearce-as-chief-revenue-officer-to-drive-sustainable-commercial-growth?hsLang=en)

[Read More](https://blog.optibus.com/optibus-appoints-simon-pearce-as-chief-revenue-officer-to-drive-sustainable-commercial-growth?hsLang=en)

[![The State Of Digital Transformation In Public Transportation – 5 Key Takeaways For 2024 And Beyond](https://blog.optibus.com/hubfs/ebook-The-State-of-Digital-Transformation-in-Public-Transportation-2024-1024x597-4.png)](https://optibus.com/ebook/2024-digital-transformation-public-transportation/)

## [The State Of Digital Transformation In Public Transportation – 5 Key Takeaways For 2024 And Beyond](https://optibus.com/ebook/2024-digital-transformation-public-transportation/)

[Read More](https://optibus.com/ebook/2024-digital-transformation-public-transportation/)

[![This Is How We Scale: Using a Centralized Logging Solution](https://blog.optibus.com/hubfs/pexels-scott-webb-27406.jpg)](https://blog.optibus.com/this-is-how-we-scale-using-a-centralized-logging-solution?hsLang=en)

## [This Is How We Scale: Using a Centralized Logging Solution](https://blog.optibus.com/this-is-how-we-scale-using-a-centralized-logging-solution?hsLang=en)

[Read More](https://blog.optibus.com/this-is-how-we-scale-using-a-centralized-logging-solution?hsLang=en)

[![nph chooses Strategic Planning for service redesign in Germany](https://blog.optibus.com/hubfs/nph%20client%20announcement%20graphic%20%281399%20x%20839%20px%29.png)](https://blog.optibus.com/nph-chooses-strategic-planning-for-service-redesign-in-germany?hsLang=en)

## [nph chooses Strategic Planning for service redesign in Germany](https://blog.optibus.com/nph-chooses-strategic-planning-for-service-redesign-in-germany?hsLang=en)

[Read More](https://blog.optibus.com/nph-chooses-strategic-planning-for-service-redesign-in-germany?hsLang=en)

[![Solving NP-Hard Problems To Optimize Large-Scale Systems: It Sounds Complicated, But It’s Not.](https://blog.optibus.com/hubfs/Imported_Blog_Media/iStock-527234840-1024x683-1.jpg)](https://blog.optibus.com/engineering/solving-np-hard-problems-to-optimize-large-scale-systems-it-sounds-complicated-but-its-not?hsLang=en)

## [Solving NP-Hard Problems To Optimize Large-Scale Systems: It Sounds Complicated, But It’s Not.](https://blog.optibus.com/engineering/solving-np-hard-problems-to-optimize-large-scale-systems-it-sounds-complicated-but-its-not?hsLang=en)

[Read More](https://blog.optibus.com/engineering/solving-np-hard-problems-to-optimize-large-scale-systems-it-sounds-complicated-but-its-not?hsLang=en)

[![Helping Transit Agencies Create A COVID-19 Exit Strategy At No Cost](https://blog.optibus.com/hubfs/Imported_Blog_Media/exit-strategy.jpg)](https://blog.optibus.com/announcements/transit-agencies-create-a-covid-19-exit-strategy-at-no-cost?hsLang=en)

## [Helping Transit Agencies Create A COVID-19 Exit Strategy At No Cost](https://blog.optibus.com/announcements/transit-agencies-create-a-covid-19-exit-strategy-at-no-cost?hsLang=en)

[Read More](https://blog.optibus.com/announcements/transit-agencies-create-a-covid-19-exit-strategy-at-no-cost?hsLang=en)

## See the power of Optibus for yourself

[Request a demo](https://www.optibus.com/request-a-demo/)

#### Get transportation industry updates

#### Product

- [Platform](https://www.optibus.com/product/platform/)
- [Planning](https://www.optibus.com/product/planning/)
- [Scheduling](https://www.optibus.com/product/scheduling/)
- [Rostering](https://www.optibus.com/product/rostering/)
- [Operations](https://www.optibus.com/product/operations/)
- [Driver App](https://www.optibus.com/product/driver-app/)
- [Passenger information](https://optibus.com/product/passenger-information/)
- [MagicEye – Driver Safety](https://optibus.com/product/magic-eye/)
- [Control & Monitoring](https://optibus.com/product/control/)
- [Optibus AI](https://optibus.com/product/control/)

#### Solutions

- [Agencies](https://www.optibus.com/solutions/optibus-for-agencies/)
- [Operators](https://www.optibus.com/solutions/optibus-for-operators/)
- [Consultancies](https://www.optibus.com/solutions/optibus-for-consultancies/)
- [Electrical Vehicles](https://www.optibus.com/solutions/optibus-for-electrical-vehicles/)
- [Tenders](https://www.optibus.com/solutions/optibus-for-tenders/)
- [Enhance Passenger Experience](https://optibus.com/enhance-passenger-experience/)
- [Reduce Costs](https://optibus.com/solutions/reduce-operating-costs/)

#### Resources

- [Blog](https://blog.optibus.com)
- [eBooks](https://www.optibus.com/resources/ebooks/)
- [Webinars](https://www.optibus.com/resources/webinars/)
- [Case Studies](https://www.optibus.com/resources/case-studies/)
- [Videos](https://www.optibus.com/resources/videos/)

#### Company

- [About](https://www.optibus.com/about-the-company/)
- [Careers](https://www.optibus.com/careers/)
- [Media](https://www.optibus.com/news-events/media-coverage/)
- [Events](https://www.optibus.com/events/)
- [Contact Us](https://www.optibus.com/about/contacts/)
- [Partnership Program](https://optibus.com/company/global-partnership-program/)
- [Academy Program](https://optibus.com/company/academy-program/)
- [Sustainability & ESG](https://optibus.com/esg/)
- [Security Whitepaper](https://optibus.com/company/security/)

#### Content

- Recentes
- Populares
- Categorias

### Lists by Topic

- [Transportation (174)](https://blog.optibus.com/tag/transportation)
- [Announcements (159)](https://blog.optibus.com/tag/announcements)
- [Public Transportation (59)](https://blog.optibus.com/tag/public-transportation)
- [Blog (55)](https://blog.optibus.com/tag/blog)
- [Corporate (49)](https://blog.optibus.com/tag/corporate)
- [Product (40)](https://blog.optibus.com/tag/product)
- [People (33)](https://blog.optibus.com/tag/people)
- [Technology (32)](https://blog.optibus.com/tag/technology)
- [Solutions (26)](https://blog.optibus.com/tag/solutions)
- [Engineering (24)](https://blog.optibus.com/tag/engineering)
- [Company (17)](https://blog.optibus.com/tag/company)
- [Latin America (15)](https://blog.optibus.com/tag/latin-america)
- [Women (15)](https://blog.optibus.com/tag/women)
- [North America (14)](https://blog.optibus.com/tag/north-america)
- [AI (12)](https://blog.optibus.com/tag/ai)
- [Passenger (10)](https://blog.optibus.com/tag/passenger)
- [case study (10)](https://blog.optibus.com/tag/case-study)
- [Drivers (9)](https://blog.optibus.com/tag/drivers)
- [Electric Buses (9)](https://blog.optibus.com/tag/electric-buses)
- [values (9)](https://blog.optibus.com/tag/values)
- [APAC (8)](https://blog.optibus.com/tag/apac)
- [Algorithms (8)](https://blog.optibus.com/tag/algorithms)
- [driver shortage (8)](https://blog.optibus.com/tag/driver-shortage)
- [Innovation (6)](https://blog.optibus.com/tag/innovation)
- [Operations (6)](https://blog.optibus.com/tag/operations)
- [Planning (6)](https://blog.optibus.com/tag/planning)
- [Driver retention (5)](https://blog.optibus.com/tag/driver-retention)
- [Partnerships (5)](https://blog.optibus.com/tag/partnerships)
- [impact (5)](https://blog.optibus.com/tag/impact)
- [Control (4)](https://blog.optibus.com/tag/control)
- [ebuses (4)](https://blog.optibus.com/tag/ebuses)
- [Automation (3)](https://blog.optibus.com/tag/automation)
- [Driver Portal (3)](https://blog.optibus.com/tag/driver-portal)
- [GTFS (3)](https://blog.optibus.com/tag/gtfs)
- [Security (3)](https://blog.optibus.com/tag/security)
- [covid-19 (3)](https://blog.optibus.com/tag/covid-19)
- [efficiency (3)](https://blog.optibus.com/tag/efficiency)
- [Billboards (2)](https://blog.optibus.com/tag/billboards)
- [Data (2)](https://blog.optibus.com/tag/data)
- [Digital Transformation (2)](https://blog.optibus.com/tag/digital-transformation)
- [cloud (2)](https://blog.optibus.com/tag/cloud)
- [digital bidding (2)](https://blog.optibus.com/tag/digital-bidding)
- [performance (2)](https://blog.optibus.com/tag/performance)
- [Africa (1)](https://blog.optibus.com/tag/africa)
- [Allocation Optimization (1)](https://blog.optibus.com/tag/allocation-optimization)
- [BRT (1)](https://blog.optibus.com/tag/brt)
- [Bidding (1)](https://blog.optibus.com/tag/bidding)
- [Cybersecurity (1)](https://blog.optibus.com/tag/cybersecurity)
- [Demand response (1)](https://blog.optibus.com/tag/demand-response)
- [Demand response transport (1)](https://blog.optibus.com/tag/demand-response-transport)
- [GTFS-Flex (1)](https://blog.optibus.com/tag/gtfs-flex)
- [India (1)](https://blog.optibus.com/tag/india)
- [Optibus Agent (1)](https://blog.optibus.com/tag/optibus-agent)
- [Passenger Information (1)](https://blog.optibus.com/tag/passenger-information)
- [Shuttles (1)](https://blog.optibus.com/tag/shuttles)
- [Trillium (1)](https://blog.optibus.com/tag/trillium)
- [design (1)](https://blog.optibus.com/tag/design)
- [punctuality (1)](https://blog.optibus.com/tag/punctuality)

see all

## Get transportation industry updates

(But not too often, don’t worry!)

[info@optibus.com](mailto:info@optibus.com) | [Privacy Policy](https://www.optibus.com/optibus-privacy-policy/) | [Cookies Policy](https://www.optibus.com/optibus-cookie-policy/) | [ISO 27001 certified](https://www.optibus.com/wp-content/uploads/2023/02/ISO-27001-Optibus-certification.pdf) | © Optibus 2026

<https://twitter.com/optibus> <https://www.facebook.com/Optibusltd/> <https://www.linkedin.com/company/optibus-ltd/>