Docker based WordPress dev environment

   Working in Agile means being flexible when it comes to the team tasks as well.  A good QA should be fully responsible not only for the testing activities, but for all of the rest when it comes to Product quality. If the team needs process improvement – initiate it. Better infrastructure – build it. The testing is not an SDLC phase from a very long time now, but rather - an integrated development activity.  

   Let’s look at our problem

•          shared and slow development/integration server
•           sluggish testing feedback loops
•           multiple OS based local development environment (Unix, Mac, Windows)
•           complex Frontend and Backend team integrations
•           need of shared and timely loading content between the team members

   Most of the above mentioned issues are caused by manually managed infrastructure. Going through the options with the team we’ve decided that a Docker based replacement should be built. Moving to a IaC is not an easy task even with a dedicated DevOps team at hand. But sometimes the only guy with the “Automation” in his job title is the QA engineer. So facing such challenge is a great learning opportunity (and IMHO, part of the day to day work). 

    First, we should get decent understanding how WordPress development works and how our team currently manages the process. Probably most of us have seen the following architecture

    

    However, this is not the case with Docker containers, as we can see from the Dockerfile. In this scenario, both the WordPress and the Apache will run inside the containers (on the developer machine). This leaves us with just the mySQL Server environment configs as shown on the hub. One more thing to note is that the wp-config.php comes with default values, so you need to either append your custom code or entirely replace the file. Example is the case when we need to read the localhost URL and not the integration server one. PHP sample code

and on our CLI run 

If we now go to our : (e.g. 127.0.0.1:8000) we should see the well-known White screen of death.  This could be caused by millions of things, but in our case we have a clean and connected environment. We’ve checked that the container is up and running, /wp-admin is loading as well.  After all WordPress acts as a CMS as well, so we need to consider the content. The same is located at wp-content/uploads. So if we check that directory inside the container with

we’ll see that it is empty.  Let’s get back to our last problem from the list – shared content between the team members. We should provide the team with the possibility to manage the work in progress and in the same time to keep their local copies clean. One such solution is the NFS. Yes, we’ve considered Swarm, data containers and volumes, but they are not supposed to do this task by design. The first option is for orchestrating containers, last two work only on one host and are pretty much equal in this case.  What we need is to spin up a VM box that will be our data host and configure it with  nfs-kernel-server.

   

  All of the above works well with Unix and Mac, but not with Docker Machine and Windows. We need a dedicated solution here, like SFTP and Eldos. Note that here our host is not the Windows OS, but the VM (Oracle VirtualBox)on which the Docker engine runs on. This could cause empty folders in your containers even if they do exist on your local file system.  Also replace the local path like this:  

    

Engineering Culture of stability: Flaky tests

 The scenario:

    We have an automation suites for our Apps that is set to run on every commit to master/deploy to Prod and for a long, long time (almost right after the beginning) we've been having issues trying to make it reliable enough.

    The tests are run in CI server (TeamCity) using Selenium WebDriver/Grid. We know the tests work because if we run them locally on our laptops (I and the team had tried it) they run perfectly every single time.

    But when they fail they don't always fail at the same spot. Sometimes it's a timeout while waiting for an Web element, sometimes the test ends up in an error page that shouldn't have reached in the first place and we have no idea how it got there... So yeah, it's frustrating.

    The team have tried a lot of different approaches to debug it. Re-writing the setup of each test to make sure everything is cleared up at the end of every single test so that the next one starts with a clean workspace/cache, making it so Selenium takes screenshots every time it fails to see what happened, tried different versions of chromedriver/chrome/selenium, added heavy logging of each action taken, put the tests to run several times in a row to see if there was any pattern...

The problem:

    Unfortunately, across our entire suites of tests, we see a continual rate of all test runs reporting a "flaky" result. We define a "flaky" test result as a test that exhibits both a passing and a failing result with the same code.  Root causes why you are getting flaky results are many: parallel execution, relying on non-deterministic or undefined behavior, flaky 3rd party code, infrastructure problems, etc. Some of the tests are flaky because of how test harnesses interact with the UI, sync timing issues, handshaking, and extraction of SUT state.

    Even if we have invested a lot of effort in removing flakiness from tests, overall the insertion rate is about the same as the fix rate. Meaning we are stuck with a certain rate of tests that provide value, but occasionally produce a flaky result.

Mitigation strategy:

    In my opinion even after tons of effort to reduce such problematic tests, flaky tests are inevitable when the test conditions reach a certain complexity level.  We will always have a core set of test problems only discoverable in an integrated End-to-end system. And those tests will be flaky. The main goal, then, is to appropriately manage those. I prefer to rely more on repetition, statistics and runs that do not block the CI pipeline.

    Just tagging tests as flaky is addressing the problem from the wrong direction, and it will lose potentially valuable information of the root causes. However, I think that there are some actions that can help us keep the flaky tests at their acceptable minimum. Consider introducing some of the below listed methods in your own context. They are split based on implementation difficulty, so you can plan your efforts accordingly:

[Easy] 

• re-run only failed tests. Failed build should keep those tests, mark them and trigger second build to execute them. 
• use combination of Exploratory testing and Automation runs. One of the basics for automation is to consider appropriate candidates (stable and are not changed too often).
• do NOT write many GUI System Tests - they should be rare, when needed. You need to build a pyramid. There are almost always possibilities to write tests at lower level.
• if you utilize parallel tests execution, consider moving some (few) tests into a single-threaded suite  

[Medium] 

• re-run tests automatically when they fail during test execution. You can read the test status in the TearDown and if failed, start new Process to execute the test again. Some open-source testing frameworks/tools also have annotations (e.g. Android has @FlakyTest, Jenkins has @RandomFail/ flaky-test-handler-plugin, Ruby ZenTest has Autotest  and Spring has @Repeat) to label flaky tests that require a few reruns upon failure.
• quarantine section (separate suite/build job)  that runs all new tests added in a loop for a certain amount of executions (Fitness function) to determine if there is any flakiness in them, in that time they are not yet part of the critical CI path. Execute reliability runs of all your CI tests per build to generate consistency rates. Using those numbers, push product teams to move all tests that fall below a certain consistency level out of the CI tests.
• consider advanced concepts like combination of xpath and Look&feel
• refactor for Hermetic pattern, avoid global/shared state or data and rely on random test run order
• proper Test Fixture strategy

[Advanced] 

• tool/process that monitors the flakiness rate of all tests and if the flakiness is too high, it automatically quarantines the test. Quarantining removes the test from the CI critical path and flags it for further investigation.
• tool/process that detects changes and works to identify the one that caused the test to change the level of flakiness 
• test that monitors itself for what it does. If it fails, look at root cause from the available log info. Then, depending on what failed (for example, an external dependency), do a smart retry. Is the failure reproduced? Then, fail the test.

Conclusion:

    I know all of the above is far from perfect or complete solution, but the truth is that you have to constantly invest in detecting, mitigating, tracking, and fixing test flakiness throughout your code base. 

Design patterns in QA Automation - PoC

    Following the example of Anton Angelov's "Design Patterns in Automation Testing" Series, I have  put together some of the PoC projects that I have prepared in the past. Some of the patterns were purely for the fun of learning, others did convince the team in their benefits and actually made it to the Big league projects. Many of the patterns in this list were introduced in a refactorings of existing Automation solutions, so we had to double check the value which they would bring to the Project in a real-life and everyday usage.
    Here is the list of the patterns I have managed to re-create again:

•  Blackboard

Building a software system for WebElements' image recognition.
Input is screenshot recorded as image and output is accessible WebElement.

• ChainOfResponsibility

Using this pattern we encapsulates the test steps inside a "pipeline" abstraction 
and have scripts "launch and leave" their requests at the entrance of the pipeline.

• Composite

Helps us to create Page Objects by forming a tree structure and ask each node in the tree structure to perform a task (loading, verifing itself). 

• Flyweight
• Interpreter

Can be used as rules engine to support business logic in tests or on creation of their fixtures.

• LazyInitialization

Provides delayed execution of certain tasks. Good example is a Shared fixture scenario or 
DB sandboxing.

• Mediator

Since it encapsulates how a set of objects interact, we can use it to share test execution data
and analysis between Reporting systems.

• Module

Extensibility modules let us consolidate our plug-ins into a centralized place. Good fit for this case
are the different Reporting systems we use.

• Multiton

Helps us to manage a map of named instances as key-value pairs. Also simplifies retrieval 
of shared objects (fixtures).

• ObjectPool

Uses a set of initialized objects kept ready to use, rather than allocating and destroying them on demand. 
Such expensive objects could be our test fixtures which we would like to re-use, but only one at a time 
between multiple parallel tests.

• Observer

Since it define a one-to-many dependency between objects so that when one (test) object changes state,
all its dependents are notified and updated automatically. Good fit are the Reporting systems 
that needs to be notified of test's status.

• PageLoader

Encapsilates navigation logic over site's pages via Bidirected and  Cyclic graph.

• RAII  

By holding a resource tied to object lifetime, we can destroy all of them at the end. Good examples are Transaction roll-back and DB cleaning.

• Servant  

Shared code for a group of classes, that appears in only one class without defining that functionality in each of them. Typical example is REST RequestSender that takes care of the multiple Contracts between the APIs.

• State

Allow an object (fixtures) to alter its behavior when its internal state changes, so our tests could make use of it.

    The implementations are mainly focused and do represent Automation testing perspective. A good example is the Observer  pattern used to show how we could notify different systems for the test run output results. 

    You can find the GitHub repo here...

Coder scroll

 

    Some consider programming to be a form of art (including myself). Simply put, there is no easy way of accurately measure artwork. Small paintings can be more meaningful than big landscapes, even just one unique perception of a woman's smile can put your framed canvas at the Louvre Museum. If we all aim at our masterpiece, how we know when we have achieved it? If we're constantly learning new techniques and tools, can we ever find our best work?
    During the past several years, I have collected some of the coding techniques that I consider worth remembering and using. But as the list grows I'm staring to realize that the art is combining them, not utilizing all. So probably this is the best place to put a

Disclaimer: Code following all the principles described in the list is not possible but we should try to follow most of them until there is a suitable reason for not following any.

    You can find the coder scroll here...

    If someone knows such principles, techniques, rules or whatever we find useful and it is not currently present in this collection, please leave a comment and I will update the scroll.

      

OF LOCATORS AND MEN

    If you have some experience in UI automation and testing, probably you already are familiar with the bitter taste of the ever-changing DOM. As most of your maintenance of the code is related to constant updating of WebElements (their locators) and minor scenario flow fixes. At first this may look like the Sisyphus task. I guess at some point we all ask ourselves - is there a better way? Let's leave aside all we have been taught in the numerous courses, books and articles about Selenium's location strategies. I will use web site automation and XPath in the examples below.
    One of the important principles of programming states

 Abstractions should not depend upon details. Details should depend upon abstractions.

So, why we bend our thinking in favour of bots? We desperately try to enforce the rules of low-level implementation instead of working with abstractions? If you use Selenium-like libraries to create your frameworks and/or crawlers, probably you locate your DOM elements like this:

    Nothing wrong here, it works well and tests are up to speed, again. For a very long time I did stop here, happy that I've fixed the tests. But in a while (usually too soon), this scenario happens again. I agree that my Selenium based tests require such location, but the approach is somehow messed up. Clearly, something is not as it should be. I used to blame development, processes and whatever comes to be on my way. But I am the owner of my code, so it's me to blame.
   
Focus on end-users.
    I guess you all have heard about Look&Feel in web design. It can be explained simply as aspects of the UI design, including elements such as colors, shapes, layout, and typefaces (the "look"), as well as the behavior of dynamic elements such as buttons, boxes, and menus (the "feel"). Without going further in UX design details, we could say that this is the brand and the users - two of things we care most. After all, the product (SUT) is made for humans, not for crawlers. If you work in any kind of Agile probably you have to start developing tests in parallel to the functionality. In a good company some Mocks can be found describing/displaying how the GUI will look like. Once agreed by the business people, those (almost) never change.
   
Stop thinking like a bot.
    We found our abstraction, insensitive to the HMTL implementation details. Humans will need text to understand the sites. This is the main pillar in our new and upgraded location strategy. Use labels, input value attributes and text to locate your elements. We could actually imply a pattern here - between text and web elements. After label with text "Username", most likely there always will be an input for your username.
    We can make good use of XPath's advanced functionality like

•  conditions
•  functions 
•  syntax

    Let's look again the Username input field example

    It might look a bit complex, but once you get to use it you will see that the stability of your tests is increased significantly. Making your XPath locators smarter will save you lots of maintenance.  They expect changes and handles them elegantly. I've made some metrics gathering and it turns out that for a build of 200 unique test runs and 54 minutes, the execution was slower with 44 milliseconds than before. A small price to pay since the browser's xpath engine actually performs the calculations, not your code.  

Blackboard Design Pattern - Layout Automation

  

    This software pattern is one of the few that still require serious Googling to be found. If it wasn't for a research PoC project I also would pass by it. But after a couple of days working with this concept I have to say, that I haven't been so interested in an idea for a long time ago. You will see it being classified as  behavioral or even architectural design pattern - depending from the usage. Most of the times its use case is in the fields of AI computing. Yes - it is such powerful concept. Even if my demo is just scratching the surface I also used such frameworks. In my case - image recognition. I already knew such library called AForge.Imaging, so putting it all together wasn't that hard.

   My vision that I had to prove was Stable web site automation based only on the layout. If you ever had your hand on web site testing you know the constant shifting sands of this domain. So the questions is - how to provide reliable UI tests? It is obvious that most DOM based solutions are highly dependant to the changes, no matter how good you are with XPath or CSS locators. But if we use the web element's layout we could decouple our tests from the HTML. Instead of keeping text based selectors, now we work only with images. And they don't change every day, once you have the designer's mockups you are done with the maintenance. This idea is not new and tools like Sikuli are gaining ground as we speak. But they do have certain disadvantages stopping them from dominating the domain. First one is that they require personal desktop - you can't run in parallel. My approach solves this with Selenium's TakeScreenshot  interface.  So you can keep the speed of the tests and only throw iron (computing power) when you need to scale it.

    Let's get back to the Blackboard pattern. Its main advantage is that works naturally with the development mindset. This pattern solves problems when there are no deterministic solutions known. We do this on a daily basis. It helps us coordinate separate, disparate systems that need to work together, or in sequence, continually prioritizing the actions it takes. It also allows our code to work in multiple threads to work closer together on separate processes, polling and reacting if needed.

    Example code implementation...

LoadableComponent is AJAX

    In my test automation I have used the LoadableComponent class aiming to make writing PageObjects less painful, like it was suggested by countless articles and their authors. They try to convince you that this will provide a standard way of ensuring that pages are loaded and you can use it to help reduce the amount of boilerplate code in your tests, which in turn make maintaining your tests less tiresome. 

    The truth is that we should consider the other side of the coin as well. This pattern violates at least a couple of principles. First is KISS, in 21st century HTML5 your pages should only keep content, not inline CSS or JS. In our test design this means - WebElement map only. Till now I was sure that  there are two basic ways of using my OOD Models -  smart tests or smart pages. All test flow logic had to be placed in one of them. The real question is - do we have to? 

    Next big thing that bothers me is the Single responsibility. It clearly states that a class should have only one reason to change. Is this the case with the examples (both simple and advanced) we are suggested to follow?


    In my humble opinion SOLID design has two kings, SRP and DIP.  Both OCP and ISP confirm the first one. There are a few other things we should consider when aiming at clean code. Overdesign is one of them. Do we need the complexity of having two (at least) classes instead of one. Keep in mind that PageActions class is also a good candidate for implementation. We need to separate the associated/needed actions per page as well. To simplify our lives, we can make the page validation a part of the Actions class. All this could lead to bulky test code. We need to draw the line here - loadable components are not page objects. 

    Using LoadableComponent in every website automation, without considering its pros and cons is not a good practice. After all they are just simple objects that encapsulates logic for page loading.  Do we need to have 20 lines classes? And do we need to couple those to the page objects? If your answer is yes, at least try to use Composition over inheritance and avoid the recommended samples like this one for your PageObject model

   public class ProjectPage extends LoadableComponent<ProjectPage> {
      ...

    Just avoid the overcomplicated hierarchy when you try to emulate real user experience in your E2E  "walking through pages" scenario features. 

    Having in mind all of the above said - when we should consider LoadableComponent? This solution works well in complex "deep" page hierarchy (like shopping journeys) and with no confident URLs (like SPA). The correct answer is AJAX-rich web sites. If your front end is build on top of any MVC, then maybe using partial URLs navigation is a better choice. Still, you have to keep some tests which will exercise the end-user journeys. One way to manage your page objects and their relations is a Graph. We do need a root role for some of the vertices, but this doesn't limit us to the Trees only.  I prefer to represent the graph with explicit edges at the cost of some additional memory. In fact the LoadableComponents are the Edges we need for our Vertex (page) objects. This way we could traverse our abstract data type representation without complex algorithms, only using node links. First we need to add abstraction to our Models


And Page objects containers don't need much of it. By implementing IPageVertex you can provide your own IWebElement map mechanics. Note that this usage is much like ISerializable in C#.


    Now our Edge implementations look like this