June 12, 2023

The "Integration and E2E tests only" trap

In the past three months, I have engaged in enlightening conversations with over seventy senior development managers (VPs, Directors, etc.), discussing the factors that impede the speed...

Gal Shahar

In the past three months, I have engaged in enlightening conversations with over seventy senior development managers (VPs, Directors, etc.), discussing the factors that impede the speed of development. Among the most prominent reasons that surfaced was software quality. How, you may ask? Let's delve into it.

Picture this: a standard developer introduces new code into the system. The code merges and spreads within the Dev environment. Another developer pushes code for a different feature, and yet another developer inserts code to fix a separate bug. This process continues with approximately 40 other developers.

Fortunately, the company's QA and automation teams have ensured an impressively comprehensive suite of thousands of End-to-End and Integration tests. These tests run once a day, at night, certifying everything is in order. One night, five tests fail. The early detection prevents a bug from reaching Production, thereby saving an important customer from lodging a complaint or even leaving. Sounds like a happy ending, right?

Not quite. The Director of Engineering arrives the following morning to find that five overnight tests (also known as "nightly") have failed. He declares a code freeze, picks up the phone to contact DevOps, who involves the QA, who tries to figure out which of yesterday's 43 developers caused the system to break. Team leads get involved, and after two days, the culprit of the problem is found. The bug becomes a ticket that needs to be resolved in the next sprint (if prioritized by the Product and R&D departments, of course), with the bug-causing code reverted from the system and attached to the next sprint's ticket for rectification.

This story mirrors the experiences of many VP R&Ds, Engineering & QA Directors I've spoken with. They've invested significant manpower into generating comprehensive tests for their system. Bugs that reach the customer are indeed a serious problem. However, they never anticipated that these tests would ironically become an issue, grounding the entire R&D team for two full days once a week.

So, what are the best practices you can adopt to rectify or prevent this issue before it arises? This is where the 'Shift Left' principle comes into play.

  1. Insist on running the tests in Continuous Integration (CI) before any code merges. This practice is an embodiment of 'Shift Left,' emphasizing catching and rectifying bugs as early as possible in the development lifecycle.
  2. Don't neglect lightweight tests like unit tests. Yes, they take time to write and maintain initially, but once they become an organizational practice and the infrastructure is in place, they naturally integrate into the development process. They empower developers to detect bugs as early as possible, even before merging and during development, locally.
  3. Tests on small code pieces offer you root cause analysis. When there's an issue, you'll discover its cause quickly, further bolstering the shift-left ideology.
  4. Testing small code pieces also allows you to examine and establish edge cases, not just happy flows.

The 'Shift Left' principle, when effectively adopted, can bridge the gap between software quality and development pace. It can transform the narrative from grappling with bug detection to proactive prevention, enhancing both software quality and development efficiency.

About Snowmate

Creating and maintaining a robust software testing infrastructure to find bugs & regressions early in the development cycle is time consuming and sometimes not possible. The good news? Snowmate can do it all for you, 100% automatically and effortlessly.

No AI, no magic tricks, pure deep tech.

Schedule a demo here.