Thursday, June 8, 2017

HTTP API Beckend tests with Ruby and NodeJS

     No more outdated API documentation. This is the promise Dredd makes. If you ever had your hands on a regression test suite of a RESTful backend, you know how tiresome can be just to keep up with all the models changing. Furthermore, till this point there wasn't really a testing framework that sets some standards in this domain. In gereral Dredd is a language-agnostic command-line tool for validating API description document against backend implementation of the API. So it reads the description and step by step validates whether your API implementation replies with responses as they are described. Just enough to call it our core engine. Dredd supports two API Description Formats:

The later one is my favorite, since it works with yml files (as the modern CI servers). We can work outside the NodeJS tech stack, so let's try Ruby.  The hooks and the documentation can be found here. Going into Ruby's world we will try and keep the good practices and use chruby as the default Ruby version manager, Rake as our build utility and couple of linters to keeping us from potential (newbie) errors. Configuring TravisCI is straightforward for our Github repo.

    Dredd can work as a standalone tool, but as we know our Test harness should be layered. As any software most of the times the backend we are going to test is quite complex, so having a Ubiquitous language is a must. As battle tested solution, I will use Gherkin. There are two major options here - Cucumber and Robot frameworks. Any will do, but since I'm already on Ruby, I will go with the native BDD framework support (setting up Robot with Ruby is a bit tricky too).

    I call the DSL layer we need to implement - the domain model layer. Since this is unique to every  system, there is no way to provide working out-of-the-box samples, context is king as you know.  In the end all that is left for us is just some glue code that will assemble the user journeys (flows).
    Another big concern of ours is the Test environment and infrastructure we need to setup and maintain. Utilizing the IaC concept and the Docker containers, we can keep all in a single repository under version control.

    Pro tip:  Don't forget to set the container port on the host, instead of your local one when running on your CI server via Docker executor. In the same time we should keep the Ruby hooks-worker handler bind to the same container.

Monday, December 5, 2016

Zombie lands: Selenium recorders

    Sometimes even best intents and plans may end up being just that. I’m not going to emphasize how much of a bad idea is to rely on capture replay tools, codeless tests and code generators in your automation testing strategy. Especially in a product company with long lived platform and production environments. Enough said. But those companies are not the only ones that you may work as a QA engineer. Some of them provide software services, virtual teams or whatever marketing calls the outsourcing in order to be different. Not better. This has its own advantages, such as vast variety of technologies and teams to work with. Experience matters. Also, boring testing stuff is really a rare sight.

    I will try to stay on the business side for this post, even if this blog is technically oriented. So, what is the main purpose of having QAs on your project? Is it about testing developers code? Automating tests? Or infrastructure? Reporting? Someone to ask why we have bugs released and given to the client? None of this really matters when you realize that QAs don’t assure quality.

    Yes, you heard me saying it.

    QAs simply report your software’s health at the end of the day. That’s it. They don’t fix bugs or implement your new feature. Why I said it!? The quality should be build in, not bolted on. Like it or not, quality is responsibility of all of us. Yes Mr. PM – yours too. There is a slim chance that your team is truly cross-functional and every team member can finish any simple backlog task (regardless of its type). If this is your case – Congrats! You’ve made it to the Big league. What happens if you, as a QA, work in a company where the majority of projects are short-lived, no more than 3 months, with no real change requests or production environment to support? In the same time is quite possible that you are also responsible for a projects that are the exact opposite. This is normal – business wants to make good use of your skills. Not taking “small” projects seriously, is a poor decision. Just their budget is different. Automation done right is expensive luxury, few can afford. Being agile, even if your environment is not, should be a must. After all, our responsibilities for both types of projects are the same.

    As advocates of the end user, we should understand what the core business is all about. And wishfully make the best in our efforts to assure its delivered with acceptable quality. Having the golden hummer of automation, sometimes we desire to bend everything with it. Is it really profitable to have complex and powerful framework and setup, if no one else (besides you) is willing to learn, use and maintain it? Good chances are you will find such people in time, but the project’s deadline is there, now. It is important to deliver and in many cases (I’ve witnessed) quality is the first priority that is cut down. And this is real life story for many of us.

    What are the alternatives!? Hate to admit it, but my best course action will be a Selenium recorder. I do think there is a perfect storm, when this makes sense. I know how many of you will stop reading after this line, but bare with me. Consider this:

• only one Manual QA, no coding skills what so ever
• website with little or no 3rd party integrations
• fairly straight-forward business flow
• static web pages, as templates are prepared long before back-end coding kicks in
• reviews were made and layout was approved as final
• no QA environments, just single Staging one
• no meaningful CI setup
• no real time for full cross- browser and platform runs
• lack of mature font-end unit testing culture
• short testing frame (couple of weeks top, out of budget with months dedicated for development), this was already agreed with the client and he is happy

    If all of the above present, I would go with a Selenium recorder. Every automation test case should justify its price, at any given time. I prefer 70% of the functionality covered by capture-replay tests over 20% created via complex UI testing framework and setup. Code coverage is always a sensitive topic. But if you keep the test pyramid you should be fine. I know what some of you may say:

– If your framework was that good, a manual QA can create the tests on the BDD level. Simple as that.

    Fair enough, but if your framework is designed well, your tests shouldn’t rely on imperative BDD, right? Ok, I lost another five good QAs reading this post. What I am trying to convince you (and myself) is that sometimes mindless tests are not so evil. No one can deny that recorded tests are cheap. Easy to create and execute. Everyone can do it. No complex setup is required. Plug and play solutions have their place under the sun. I will use a good practice from the DevOps, which states that creating new environment should be easier than fixing the existing one. Do you see how this is applicable in our context? The recorded tests should not be fixed! This is not in their nature. They should be used only until fit. The moment you start adding configurations to them, you better start using your complex, layered test framework. It is better suited and designed for this anyway. Don’t fall into the trap of adding too much gun powder to your Selenium IDE setup. I know how tempting is to add sugary plugins, but they only hide the problem. If you need more complex tests, recorders are not for this project. Same goes for Selenium Builder. I don’t consider this as a retrograde way of thinking, just being pragmatic.

    As bottom line, I would like to go back to the roots of automation – it is designed to free time and assist your testing. Having recorded scripts will save you the boring and repetitive tests for the manual QA staff. And they will have a bit extra time for the more important exploratory testing.

Monday, July 25, 2016

Shift right: Monitoring made easy

    It is all about reliability and scalability. While one of our servers may go down, the application can’t, so testing has to be in real time. Mainly we are looking at two KPIs functionality and performance. Most importantly, we also have a real-time feedback to raise issues that might not be detected by the previous testing or tools. With TestOps raising, testing in production becomes an essential piece of our overall quality plan.
    Many companies are already giving a lot of attention to the Shift-left transformation, but I also think that making a Shift-right with proactive monitoring and alerting after our releases into the wild is just as important. A while ago I did a similar task, but using NewRelic Synthetics.
    There are enough tools and platforms into the Open source stack to actually do it by ourselves very quickly and easy. This particular implementation is more of a suggestion and I’m sure you can do it with your own toolbox. The architecture is really quite simple, but backed up by the powerful IaC concept.  

    I prefer to use three types of monitors:

  •          Ping: simply check to see if an application is online. It uses a HTTP client to make requests to our service.
  •           API: the HTTP client is not a full browser, and does not execute JavaScript. Used to monitor our API endpoints. This can ensure that our app server works, in addition to the website.
  •           Virtual browser: used for more sophisticated, customized monitoring. With these monitors, we can set up a monitor that navigates our website and ensures specific resources are present.

    The central part is dedicated to the engine, which is composed via Jenkins, Docker and GitLab. Deciding what (cloud) servers should be used is all on you. Integrating those three is really straightforward. Major benefits are powerful execution, clean environments and a central repository. Alerts are via email, thanks to the Jenkins build-in functionality.
    For the first two types of the monitors we need a REST client like Postman. With this tool we can easily create and organize our tests in collections. The execution and reporting are handled by Newman. We can output the results in html, JSON and xml. The last one is JUnit formatted, so this can be plugged into Jenkins dashboards.
   In order to get a virtual browser, we will need a separate container with nodeJS, xvfb and browsers on it. I use my own Dockerfiles to build the containers I need. Turns out sometimes it is better to have a custom solution. The orchestration of our containers is done via Docker Swarm

    That’s all. Thanks for reading.

Monday, July 11, 2016

Docker based WordPress dev environment

   Working in Agile means being flexible when it comes to the team tasks as well.  A good QA should be fully responsible not only for the testing activities, but for all of the rest when it comes to Product quality. If the team needs process improvement – initiate it. Better infrastructure – build it. The testing is not an SDLC phase from a very long time now, but rather - an integrated development activity.  
   Let’s look at our problem
  •          shared and slow development/integration server
  •           sluggish testing feedback loops
  •           multiple OS based local development environment (Unix, Mac, Windows)
  •           complex Frontend and Backend team integrations
  •           need of shared and timely loading content between the team members

   Most of the above mentioned issues are caused by manually managed infrastructure. Going through the options with the team we’ve decided that a Docker based replacement should be built. Moving to a IaC is not an easy task even with a dedicated DevOps team at hand. But sometimes the only guy with the “Automation” in his job title is the QA engineer. So facing such challenge is a great learning opportunity (and IMHO, part of the day to day work). 
    First, we should get decent understanding how WordPress development works and how our team currently manages the process. Probably most of us have seen the following architecture

    However, this is not the case with Docker containers, as we can see from the Dockerfile. In this scenario, both the WordPress and the Apache will run inside the containers (on the developer machine). This leaves us with just the mySQL Server environment configs as shown on the hub. One more thing to note is that the wp-config.php comes with default values, so you need to either append your custom code or entirely replace the file. Example is the case when we need to read the localhost URL and not the integration server one. PHP sample code

and on our CLI run 

If we now go to our : (e.g. we should see the well-known White screen of death.  This could be caused by millions of things, but in our case we have a clean and connected environment. We’ve checked that the container is up and running, /wp-admin is loading as well.  After all WordPress acts as a CMS as well, so we need to consider the content. The same is located at wp-content/uploads. So if we check that directory inside the container with

we’ll see that it is empty.  Let’s get back to our last problem from the list – shared content between the team members. We should provide the team with the possibility to manage the work in progress and in the same time to keep their local copies clean. One such solution is the NFS. Yes, we’ve considered Swarm, data containers and volumes, but they are not supposed to do this task by design. The first option is for orchestrating containers, last two work only on one host and are pretty much equal in this case.  What we need is to spin up a VM box that will be our data host and configure it with  nfs-kernel-server.

  All of the above works well with Unix and Mac, but not with Docker Machine and Windows. We need a dedicated solution here, like SFTP and Eldos. Note that here our host is not the Windows OS, but the VM (Oracle VirtualBox)on which the Docker engine runs on. This could cause empty folders in your containers even if they do exist on your local file system.  Also replace the local path like this:  


Friday, June 10, 2016

Engineering Culture of stability: Flaky tests

 The scenario:

    We have an automation suites for our Apps that is set to run on every commit to master/deploy to Prod and for a long, long time (almost right after the beginning) we've been having issues trying to make it reliable enough.

    The tests are run in CI server (TeamCity) using Selenium WebDriver/Grid. We know the tests work because if we run them locally on our laptops (I and the team had tried it) they run perfectly every single time.

    But when they fail they don't always fail at the same spot. Sometimes it's a timeout while waiting for an Web element, sometimes the test ends up in an error page that shouldn't have reached in the first place and we have no idea how it got there... So yeah, it's frustrating.

    The team have tried a lot of different approaches to debug it. Re-writing the setup of each test to make sure everything is cleared up at the end of every single test so that the next one starts with a clean workspace/cache, making it so Selenium takes screenshots every time it fails to see what happened, tried different versions of chromedriver/chrome/selenium, added heavy logging of each action taken, put the tests to run several times in a row to see if there was any pattern...

The problem:

    Unfortunately, across our entire suites of tests, we see a continual rate of all test runs reporting a "flaky" result. We define a "flaky" test result as a test that exhibits both a passing and a failing result with the same code.  Root causes why you are getting flaky results are many: parallel execution, relying on non-deterministic or undefined behavior, flaky 3rd party code, infrastructure problems, etc. Some of the tests are flaky because of how test harnesses interact with the UI, sync timing issues, handshaking, and extraction of SUT state.

    Even if we have invested a lot of effort in removing flakiness from tests, overall the insertion rate is about the same as the fix rate. Meaning we are stuck with a certain rate of tests that provide value, but occasionally produce a flaky result.

Mitigation strategy:

    In my opinion even after tons of effort to reduce such problematic tests, flaky tests are inevitable when the test conditions reach a certain complexity level.  We will always have a core set of test problems only discoverable in an integrated End-to-end system. And those tests will be flaky. The main goal, then, is to appropriately manage those. I prefer to rely more on repetition, statistics and runs that do not block the CI pipeline.

    Just tagging tests as flaky is addressing the problem from the wrong direction, and it will lose potentially valuable information of the root causes. However, I think that there are some actions that can help us keep the flaky tests at their acceptable minimum. Consider introducing some of the below listed methods in your own context. They are split based on implementation difficulty, so you can plan your efforts accordingly:

  • re-run only failed tests. Failed build should keep those tests, mark them and trigger second build to execute them. 
  • use combination of Exploratory testing and Automation runs. One of the basics for automation is to consider appropriate candidates (stable and are not changed too often).
  • do NOT write many GUI System Tests - they should be rare, when needed. You need to build a pyramid. There are almost always possibilities to write tests at lower level.
  • if you utilize parallel tests execution, consider moving some (few) tests into a single-threaded suite  
  • re-run tests automatically when they fail during test execution. You can read the test status in the TearDown and if failed, start new Process to execute the test again. Some open-source testing frameworks/tools also have annotations (e.g. Android has @FlakyTest, Jenkins has @RandomFail/ flaky-test-handler-plugin, Ruby ZenTest has Autotest  and Spring has @Repeatto label flaky tests that require a few reruns upon failure.
  • quarantine section (separate suite/build job)  that runs all new tests added in a loop for a certain amount of executions (Fitness function) to determine if there is any flakiness in them, in that time they are not yet part of the critical CI path. Execute reliability runs of all your CI tests per build to generate consistency rates. Using those numbers, push product teams to move all tests that fall below a certain consistency level out of the CI tests.
  • consider advanced concepts like combination of xpath and Look&feel
  • refactor for Hermetic pattern, avoid global/shared state or data and rely on random test run order
  • proper Test Fixture strategy
  • tool/process that monitors the flakiness rate of all tests and if the flakiness is too high, it automatically quarantines the test. Quarantining removes the test from the CI critical path and flags it for further investigation.
  • tool/process that detects changes and works to identify the one that caused the test to change the level of flakiness 
  • test that monitors itself for what it does. If it fails, look at root cause from the available log info. Then, depending on what failed (for example, an external dependency), do a smart retry. Is the failure reproduced? Then, fail the test.


    I know all of the above is far from perfect or complete solution, but the truth is that you have to constantly invest in detecting, mitigating, tracking, and fixing test flakiness throughout your code base. 

Friday, May 6, 2016

Design patterns in QA Automation - PoC

    Following the example of Anton Angelov's "Design Patterns in Automation Testing" Series, I have  put together some of the PoC projects that I have prepared in the past. Some of the patterns were purely for the fun of learning, others did convince the team in their benefits and actually made it to the Big league projects. Many of the patterns in this list were introduced in a refactorings of existing Automation solutions, so we had to double check the value which they would bring to the Project in a real-life and everyday usage.
    Here is the list of the patterns I have managed to re-create again:

  •  Blackboard
Building a software system for WebElements' image recognition.
Input is screenshot recorded as image and output is accessible WebElement.
  • ChainOfResponsibility
Using this pattern we encapsulates the test steps inside a "pipeline" abstraction 
and have scripts "launch and leave" their requests at the entrance of the pipeline.
  • Composite
Helps us to create Page Objects by forming a tree structure and ask each node in the tree structure to perform a task (loading, verifing itself). 
  • Flyweight
  • Interpreter
Can be used as rules engine to support business logic in tests or on creation of their fixtures.
  • LazyInitialization
Provides delayed execution of certain tasks. Good example is a Shared fixture scenario or 
DB sandboxing.
  • Mediator
Since it encapsulates how a set of objects interact, we can use it to share test execution data
and analysis between Reporting systems.
  • Module
Extensibility modules let us consolidate our plug-ins into a centralized place. Good fit for this case
are the different Reporting systems we use.
  • Multiton
Helps us to manage a map of named instances as key-value pairs. Also simplifies retrieval 
of shared objects (fixtures).
  • ObjectPool
Uses a set of initialized objects kept ready to use, rather than allocating and destroying them on demand. 
Such expensive objects could be our test fixtures which we would like to re-use, but only one at a time 
between multiple parallel tests.
  • Observer
Since it define a one-to-many dependency between objects so that when one (test) object changes state,
all its dependents are notified and updated automatically. Good fit are the Reporting systems 
that needs to be notified of test's status.
  • PageLoader
Encapsilates navigation logic over site's pages via Bidirected and  Cyclic graph.
  • RAII  
By holding a resource tied to object lifetime, we can destroy all of them at the end. Good examples are Transaction roll-back and DB cleaning.
  • Servant  
Shared code for a group of classes, that appears in only one class without defining that functionality in each of them. Typical example is REST RequestSender that takes care of the multiple Contracts between the APIs.
  • State
Allow an object (fixtures) to alter its behavior when its internal state changes, so our tests could make use of it.

    The implementations are mainly focused and do represent Automation testing perspective. A good example is the Observer  pattern used to show how we could notify different systems for the test run output results. 

    You can find the GitHub repo here...

Saturday, April 2, 2016

Coder scroll


    Some consider programming to be a form of art (including myself). Simply put, there is no easy way of accurately measure artwork. Small paintings can be more meaningful than big landscapes, even just one unique perception of a woman's smile can put your framed canvas at the Louvre Museum. If we all aim at our masterpiece, how we know when we have achieved it? If we're constantly learning new techniques and tools, can we ever find our best work?
    During the past several years, I have collected some of the coding techniques that I consider worth remembering and using. But as the list grows I'm staring to realize that the art is combining them, not utilizing all. So probably this is the best place to put a

Disclaimer: Code following all the principles described in the list is not possible but we should try to follow most of them until there is a suitable reason for not following any.

    You can find the coder scroll here...

    If someone knows such principles, techniques, rules or whatever we find useful and it is not currently present in this collection, please leave a comment and I will update the scroll.