Quantifying Chaos Engineering Effectiveness In Event-Driven Microservices

Abstract

The chaos engineering techniques used to analyze synchronous systems are not adequate when analyzing event-driven systems because of the underlying differences in the patterns of failure propagation. Controlled experimentation of containerized e-commerce microservices reveals severe observability differences between event-driven and REST-based designs, with a substantial "failure masking effect" in which resilience mechanisms unwittingly hide structural problems. By evaluating the major chaos engineering tools in a systematic manner and under varying failure conditions, one can identify a unique pattern of effectiveness in one or the other architectural pattern. Event-driven systems must employ longer chaos experiments, give priority to queue-based metrics rather than response times, and a mixed set of failure modes in order to obtain sufficient coverage. To improve resilience in event-driven systems, which fail according to patterns that are not uniform as commonly assumed by traditional testing methods, empirical guidelines determine the best testing times, strategy in metric selection, and specific pattern-based testing advice.