Grapefruit series - Editor's choice

Going forward with the - to the Point, bite-sized content blog posts. Today, I want to talk about other professions, where Testing is put on a more rightful place. Just think about it! If you are a newspaper journalist, who reviews, corrects and agrees/approves what should be delivered to the customer? You are right - your editor! 

Imagine, you are a gourmet chef, you want your  restaurant in the top 'must visit' stops. Who helps customers understand the quality of the food, service and culture at your place? Someone who analyse all this and publish its findings? Food writer/critic does!

Police detectives are by nature - investigators. They often collect information to solve crime by talking to witnesses and informants, collecting physical evidence, or searching records in databases. As we all know, this is quite a respected and honourable role in the law enforcement!

Now a days, in the "young people's music", most of the songs feature rap. A rapper, is someone who  finds minimum rhymes to express tacit knowledge. You need deep thinking to shine!

I have been around, to see that professional software testers are not treated with at least the half of what we deserve. Is it our fault, to let others decide our importance!?

Grapefruit series - My EGO in software testing

It's time for the next Get to the Point, bite-sized post. Today I really want to put my (problems with) Ego in the spotlight. After all - is April Fools' Day, so what better time to do that!? 

Let's talk QA shamanism

“Look 'ma, no hands. ”

I can use my ass to sit on the keyboard - that's a test, right? I see people, who claim they are skillful and knowledgeable, so called gurus put a veil of mystery over what testing is. Maybe, hoping to start a cult and sell snake oil. Me and you, are clearly NOT in the same (QA) business!

No! Software testing is not a rocket science, it is natural. And should be fun! Testing is everything, but voodoo and we don't need witch doctors. The product or service we are testing are not caused by witchcraft. So, let's say I'm a Tech lead that knows only one, single technology (in his entire career). Then, why do I preach to others, how automation should be done in testing. Form and propose solutions, to all of other people's problems.

I have only 50 visits on my blog today. Should, I post content, that is just the eye candy aggregate of the information I found on the web. Will this offend the knowledge reader!?  On the other hand, as receiver of such great knowledge - how intelligent really am I? You want an example: great minds that have a huge impact on development and DevOps in the past decades, advocate Page object model as The solution. POM is great, when applied correctly, in a very narrow scope. Such case is pywinauto and desktop apps. Your selectors are an integral part of your GUI abstraction.

What's the point here?
We don't need fancy-schmancy terms, but simpler ones, so more and more people can be engaged in the software testing domain. All I'm saying is, that I shouldn't overengineer the testing - just because of my EGO needs.

Grapefruit series - Harness AI in your Testing mindset

Let's first look at the startups and their Dev-centric approach. Cold facts, show that beside the core development, there are little (dedicated) gravitating roles: PM, SysAdmin, QA, Support.

In such a fast paced and agile environments, one can see future trends. For instance, how AI affects our daily IT jobs. As automation testers, at some point I guess we have all asked ourselves: "Am I automating myself out of a job?". I know I did automate an automation test engineer in the past. And the scary part is - it took just a couple of days, image recognition library and Selenium. The solution was imperfect and slow, but good enough for the business. She had to go. More and more IT tasks will be done by lines of code automatically written by a machine.

For a very long time, we're at our first stop on the way to AI: automation. Ask a veteran sysadmin what their daily life was like 10 years ago, and you will realize just how much automation has already happened in the field. Some things you might not even think of, simply because we now take them for granted. Need an example? In the "old days", for a website with medium complexity you needed a team, now a single resource (freelancer) can do a job that took a team of maybe three or four to do just a decade ago. The technologies require less and less system administrators to set them up, configure, maintain and scale them. You no longer need to take extra care of monitoring your environments or scaling of the system. It is done automatically according to expected usage (predictive provisioning).

Technology has been steadily impacting the jobs of project managers for years now, too. Just a few quick examples: offload truly routine tasks to increase value, coordinating tasks to increase efficiency and collect updates from the team in order to produce reports or raise triggers. Budget is a big part of the project, AI tools can now chalk out the most optimum and financially viable schedule and budget for any kind of project based on projection modeling techniques. Automatic project trackers (like Timely) are showcasing more and more benefits.

Support divisions and workflows also changed rapidly in the past years. I have witnessed first levels of the service desk blown away by chatbots and FAQ articles automation. The digital transformation took only a few weeks and there was no significant gap in the operations. The users got faster answers, reduced research and predictive insights for just a fraction of the old cost. The company focused beyond reaping first-contact efficiencies and invested in spotting patterns that will help the team perform root cause analysis to prevent issues.  

There are countless ways that Machine Learning can benefit a software tester. The thing is, one could not simply take the algorithms of AI and apply them to another game as-is (even though they are more general-purpose than in the past). As engineers, we should embrace this power and put in our human touch in the new service we provide. Advances in the AI field are exponential.
New challenges are rising, and the trick is to figure out what skills those challenges require. Here is one - utilisation of chatbots in software testing! How we can use the domain knowledge incorporated in them!? How to harvest the user interactions (expectations, questions) and convert them into test cases!?

Grapefruit series - Challenge roles

      Get to the Point, bite-sized content is what this and next posts will be all about. After all, is blogging, not scientific research. Looking back at the last two years, honestly, I don't have the time to continue with detailed articles. The idea is to write more regularly (due to the narrow topic coverage), to share my raw thoughts and opinion.

     The very first thing I would like to address, is the roles that we all have in our teams. You have a single role in a single team - it will also work. Growing is part of our career. One way to do so, is moving to new roles. Taking ownership of new roles. Ones, that were not held by us (but exist, or not in the team). You need to deserve and keep it, making it official is just detail. So, if you are not in the place you wanna be - challenge the roles (NOTE: not positions).

     At some point, this should become a culture. Strongest roles, rule the team by leadership and competence. Strongest roles, have everything to prove, every day. Foster a way to learn and grow alongside your teammates. Never assume, that your title gives you comfort and you have the final say. The more important your role is - the more you have to prove to keep it. Look for the ones that challenge and level up your game.

HTTP API Beckend tests with Ruby and NodeJS

     No more outdated API documentation. This is the promise Dredd makes. If you ever had your hands on a regression test suite of a RESTful backend, you know how tiresome can be just to keep up with all the models changing. Furthermore, till this point there wasn't really a testing framework that sets some standards in this domain. In gereral Dredd is a language-agnostic command-line tool for validating API description document against backend implementation of the API. So it reads the description and step by step validates whether your API implementation replies with responses as they are described. Just enough to call it our core engine. Dredd supports two API Description Formats:

• Blueprint and
• Swagger

The later one is my favorite, since it works with yml files (as the modern CI servers). We can work outside the NodeJS tech stack, so let's try Ruby.  The hooks and the documentation can be found here. Going into Ruby's world we will try and keep the good practices and use chruby as the default Ruby version manager, Rake as our build utility and couple of linters to keeping us from potential (newbie) errors. Configuring TravisCI is straightforward for our Github repo.

    Dredd can work as a standalone tool, but as we know our Test harness should be layered. As any software most of the times the backend we are going to test is quite complex, so having a Ubiquitous language is a must. As battle tested solution, I will use Gherkin. There are two major options here - Cucumber and Robot frameworks. Any will do, but since I'm already on Ruby, I will go with the native BDD framework support (setting up Robot with Ruby is a bit tricky too).

    I call the DSL layer we need to implement - the domain model layer. Since this is unique to every  system, there is no way to provide working out-of-the-box samples, context is king as you know.  In the end all that is left for us is just some glue code that will assemble the user journeys (flows).
    Another big concern of ours is the Test environment and infrastructure we need to setup and maintain. Utilizing the IaC concept and the Docker containers, we can keep all in a single repository under version control.

    Pro tip:  Don't forget to set the container port on the host, instead of your local one when running on your CI server via Docker executor. In the same time we should keep the Ruby hooks-worker handler bind to the same container.

Zombie lands: Selenium recorders

    Sometimes even best intents and plans may end up being just that. I’m not going to emphasize how much of a bad idea is to rely on capture replay tools, codeless tests and code generators in your automation testing strategy. Especially in a product company with long lived platform and production environments. Enough said. But those companies are not the only ones that you may work as a QA engineer. Some of them provide software services, virtual teams or whatever marketing calls the outsourcing in order to be different. Not better. This has its own advantages, such as vast variety of technologies and teams to work with. Experience matters. Also, boring testing stuff is really a rare sight.

    I will try to stay on the business side for this post, even if this blog is technically oriented. So, what is the main purpose of having QAs on your project? Is it about testing developers code? Automating tests? Or infrastructure? Reporting? Someone to ask why we have bugs released and given to the client? None of this really matters when you realize that QAs don’t assure quality.

    Yes, you heard me saying it.

    QAs simply report your software’s health at the end of the day. That’s it. They don’t fix bugs or implement your new feature. Why I said it!? The quality should be build in, not bolted on. Like it or not, quality is responsibility of all of us. Yes Mr. PM – yours too. There is a slim chance that your team is truly cross-functional and every team member can finish any simple backlog task (regardless of its type). If this is your case – Congrats! You’ve made it to the Big league. What happens if you, as a QA, work in a company where the majority of projects are short-lived, no more than 3 months, with no real change requests or production environment to support? In the same time is quite possible that you are also responsible for a projects that are the exact opposite. This is normal – business wants to make good use of your skills. Not taking “small” projects seriously, is a poor decision. Just their budget is different. Automation done right is expensive luxury, few can afford. Being agile, even if your environment is not, should be a must. After all, our responsibilities for both types of projects are the same.

    As advocates of the end user, we should understand what the core business is all about. And wishfully make the best in our efforts to assure its delivered with acceptable quality. Having the golden hummer of automation, sometimes we desire to bend everything with it. Is it really profitable to have complex and powerful framework and setup, if no one else (besides you) is willing to learn, use and maintain it? Good chances are you will find such people in time, but the project’s deadline is there, now. It is important to deliver and in many cases (I’ve witnessed) quality is the first priority that is cut down. And this is real life story for many of us.

    What are the alternatives!? Hate to admit it, but my best course action will be a Selenium recorder. I do think there is a perfect storm, when this makes sense. I know how many of you will stop reading after this line, but bare with me. Consider this:

• only one Manual QA, no coding skills what so ever
• website with little or no 3rd party integrations
• fairly straight-forward business flow
• static web pages, as templates are prepared long before back-end coding kicks in
• reviews were made and layout was approved as final
• no QA environments, just single Staging one
• no meaningful CI setup
• no real time for full cross- browser and platform runs
• lack of mature font-end unit testing culture
• short testing frame (couple of weeks top, out of budget with months dedicated for development), this was already agreed with the client and he is happy

    If all of the above present, I would go with a Selenium recorder. Every automation test case should justify its price, at any given time. I prefer 70% of the functionality covered by capture-replay tests over 20% created via complex UI testing framework and setup. Code coverage is always a sensitive topic. But if you keep the test pyramid you should be fine. I know what some of you may say:

– If your framework was that good, a manual QA can create the tests on the BDD level. Simple as that.

    Fair enough, but if your framework is designed well, your tests shouldn’t rely on imperative BDD, right? Ok, I lost another five good QAs reading this post. What I am trying to convince you (and myself) is that sometimes mindless tests are not so evil. No one can deny that recorded tests are cheap. Easy to create and execute. Everyone can do it. No complex setup is required. Plug and play solutions have their place under the sun. I will use a good practice from the DevOps, which states that creating new environment should be easier than fixing the existing one. Do you see how this is applicable in our context? The recorded tests should not be fixed! This is not in their nature. They should be used only until fit. The moment you start adding configurations to them, you better start using your complex, layered test framework. It is better suited and designed for this anyway. Don’t fall into the trap of adding too much gun powder to your Selenium IDE setup. I know how tempting is to add sugary plugins, but they only hide the problem. If you need more complex tests, recorders are not for this project. Same goes for Selenium Builder. I don’t consider this as a retrograde way of thinking, just being pragmatic.

    As bottom line, I would like to go back to the roots of automation – it is designed to free time and assist your testing. Having recorded scripts will save you the boring and repetitive tests for the manual QA staff. And they will have a bit extra time for the more important exploratory testing.

Shift right: Monitoring made easy

    It is all about reliability and scalability. While one of our servers may go down, the application can’t, so testing has to be in real time. Mainly we are looking at two KPIs functionality and performance. Most importantly, we also have a real-time feedback to raise issues that might not be detected by the previous testing or tools. With TestOps raising, testing in production becomes an essential piece of our overall quality plan.

    Many companies are already giving a lot of attention to the Shift-left transformation, but I also think that making a Shift-right with proactive monitoring and alerting after our releases into the wild is just as important. A while ago I did a similar task, but using NewRelic Synthetics.

    There are enough tools and platforms into the Open source stack to actually do it by ourselves very quickly and easy. This particular implementation is more of a suggestion and I’m sure you can do it with your own toolbox. The architecture is really quite simple, but backed up by the powerful IaC concept.  

    I prefer to use three types of monitors:

•          Ping: simply check to see if an application is online. It uses a HTTP client to make requests to our service.
•           API: the HTTP client is not a full browser, and does not execute JavaScript. Used to monitor our API endpoints. This can ensure that our app server works, in addition to the website.
•           Virtual browser: used for more sophisticated, customized monitoring. With these monitors, we can set up a monitor that navigates our website and ensures specific resources are present.

    The central part is dedicated to the engine, which is composed via Jenkins, Docker and GitLab. Deciding what (cloud) servers should be used is all on you. Integrating those three is really straightforward. Major benefits are powerful execution, clean environments and a central repository. Alerts are via email, thanks to the Jenkins build-in functionality.

    For the first two types of the monitors we need a REST client like Postman. With this tool we can easily create and organize our tests in collections. The execution and reporting are handled by Newman. We can output the results in html, JSON and xml. The last one is JUnit formatted, so this can be plugged into Jenkins dashboards.

   In order to get a virtual browser, we will need a separate container with nodeJS, xvfb and browsers on it. I use my own Dockerfiles to build the containers I need. Turns out sometimes it is better to have a custom solution. The orchestration of our containers is done via Docker Swarm. 

    That’s all. Thanks for reading.

Docker based WordPress dev environment

   Working in Agile means being flexible when it comes to the team tasks as well.  A good QA should be fully responsible not only for the testing activities, but for all of the rest when it comes to Product quality. If the team needs process improvement – initiate it. Better infrastructure – build it. The testing is not an SDLC phase from a very long time now, but rather - an integrated development activity.  

   Let’s look at our problem

•          shared and slow development/integration server
•           sluggish testing feedback loops
•           multiple OS based local development environment (Unix, Mac, Windows)
•           complex Frontend and Backend team integrations
•           need of shared and timely loading content between the team members

   Most of the above mentioned issues are caused by manually managed infrastructure. Going through the options with the team we’ve decided that a Docker based replacement should be built. Moving to a IaC is not an easy task even with a dedicated DevOps team at hand. But sometimes the only guy with the “Automation” in his job title is the QA engineer. So facing such challenge is a great learning opportunity (and IMHO, part of the day to day work). 

    First, we should get decent understanding how WordPress development works and how our team currently manages the process. Probably most of us have seen the following architecture

    

    However, this is not the case with Docker containers, as we can see from the Dockerfile. In this scenario, both the WordPress and the Apache will run inside the containers (on the developer machine). This leaves us with just the mySQL Server environment configs as shown on the hub. One more thing to note is that the wp-config.php comes with default values, so you need to either append your custom code or entirely replace the file. Example is the case when we need to read the localhost URL and not the integration server one. PHP sample code

and on our CLI run 

If we now go to our : (e.g. 127.0.0.1:8000) we should see the well-known White screen of death.  This could be caused by millions of things, but in our case we have a clean and connected environment. We’ve checked that the container is up and running, /wp-admin is loading as well.  After all WordPress acts as a CMS as well, so we need to consider the content. The same is located at wp-content/uploads. So if we check that directory inside the container with

we’ll see that it is empty.  Let’s get back to our last problem from the list – shared content between the team members. We should provide the team with the possibility to manage the work in progress and in the same time to keep their local copies clean. One such solution is the NFS. Yes, we’ve considered Swarm, data containers and volumes, but they are not supposed to do this task by design. The first option is for orchestrating containers, last two work only on one host and are pretty much equal in this case.  What we need is to spin up a VM box that will be our data host and configure it with  nfs-kernel-server.

   

  All of the above works well with Unix and Mac, but not with Docker Machine and Windows. We need a dedicated solution here, like SFTP and Eldos. Note that here our host is not the Windows OS, but the VM (Oracle VirtualBox)on which the Docker engine runs on. This could cause empty folders in your containers even if they do exist on your local file system.  Also replace the local path like this:  

    

Engineering Culture of stability: Flaky tests

 The scenario:

    We have an automation suites for our Apps that is set to run on every commit to master/deploy to Prod and for a long, long time (almost right after the beginning) we've been having issues trying to make it reliable enough.

    The tests are run in CI server (TeamCity) using Selenium WebDriver/Grid. We know the tests work because if we run them locally on our laptops (I and the team had tried it) they run perfectly every single time.

    But when they fail they don't always fail at the same spot. Sometimes it's a timeout while waiting for an Web element, sometimes the test ends up in an error page that shouldn't have reached in the first place and we have no idea how it got there... So yeah, it's frustrating.

    The team have tried a lot of different approaches to debug it. Re-writing the setup of each test to make sure everything is cleared up at the end of every single test so that the next one starts with a clean workspace/cache, making it so Selenium takes screenshots every time it fails to see what happened, tried different versions of chromedriver/chrome/selenium, added heavy logging of each action taken, put the tests to run several times in a row to see if there was any pattern...

The problem:

    Unfortunately, across our entire suites of tests, we see a continual rate of all test runs reporting a "flaky" result. We define a "flaky" test result as a test that exhibits both a passing and a failing result with the same code.  Root causes why you are getting flaky results are many: parallel execution, relying on non-deterministic or undefined behavior, flaky 3rd party code, infrastructure problems, etc. Some of the tests are flaky because of how test harnesses interact with the UI, sync timing issues, handshaking, and extraction of SUT state.

    Even if we have invested a lot of effort in removing flakiness from tests, overall the insertion rate is about the same as the fix rate. Meaning we are stuck with a certain rate of tests that provide value, but occasionally produce a flaky result.

Mitigation strategy:

    In my opinion even after tons of effort to reduce such problematic tests, flaky tests are inevitable when the test conditions reach a certain complexity level.  We will always have a core set of test problems only discoverable in an integrated End-to-end system. And those tests will be flaky. The main goal, then, is to appropriately manage those. I prefer to rely more on repetition, statistics and runs that do not block the CI pipeline.

    Just tagging tests as flaky is addressing the problem from the wrong direction, and it will lose potentially valuable information of the root causes. However, I think that there are some actions that can help us keep the flaky tests at their acceptable minimum. Consider introducing some of the below listed methods in your own context. They are split based on implementation difficulty, so you can plan your efforts accordingly:

[Easy] 

• re-run only failed tests. Failed build should keep those tests, mark them and trigger second build to execute them. 
• use combination of Exploratory testing and Automation runs. One of the basics for automation is to consider appropriate candidates (stable and are not changed too often).
• do NOT write many GUI System Tests - they should be rare, when needed. You need to build a pyramid. There are almost always possibilities to write tests at lower level.
• if you utilize parallel tests execution, consider moving some (few) tests into a single-threaded suite  

[Medium] 

• re-run tests automatically when they fail during test execution. You can read the test status in the TearDown and if failed, start new Process to execute the test again. Some open-source testing frameworks/tools also have annotations (e.g. Android has @FlakyTest, Jenkins has @RandomFail/ flaky-test-handler-plugin, Ruby ZenTest has Autotest  and Spring has @Repeat) to label flaky tests that require a few reruns upon failure.
• quarantine section (separate suite/build job)  that runs all new tests added in a loop for a certain amount of executions (Fitness function) to determine if there is any flakiness in them, in that time they are not yet part of the critical CI path. Execute reliability runs of all your CI tests per build to generate consistency rates. Using those numbers, push product teams to move all tests that fall below a certain consistency level out of the CI tests.
• consider advanced concepts like combination of xpath and Look&feel
• refactor for Hermetic pattern, avoid global/shared state or data and rely on random test run order
• proper Test Fixture strategy

[Advanced] 

• tool/process that monitors the flakiness rate of all tests and if the flakiness is too high, it automatically quarantines the test. Quarantining removes the test from the CI critical path and flags it for further investigation.
• tool/process that detects changes and works to identify the one that caused the test to change the level of flakiness 
• test that monitors itself for what it does. If it fails, look at root cause from the available log info. Then, depending on what failed (for example, an external dependency), do a smart retry. Is the failure reproduced? Then, fail the test.

Conclusion:

    I know all of the above is far from perfect or complete solution, but the truth is that you have to constantly invest in detecting, mitigating, tracking, and fixing test flakiness throughout your code base. 

Design patterns in QA Automation - PoC

    Following the example of Anton Angelov's "Design Patterns in Automation Testing" Series, I have  put together some of the PoC projects that I have prepared in the past. Some of the patterns were purely for the fun of learning, others did convince the team in their benefits and actually made it to the Big league projects. Many of the patterns in this list were introduced in a refactorings of existing Automation solutions, so we had to double check the value which they would bring to the Project in a real-life and everyday usage.
    Here is the list of the patterns I have managed to re-create again:

•  Blackboard

Building a software system for WebElements' image recognition.
Input is screenshot recorded as image and output is accessible WebElement.

• ChainOfResponsibility

Using this pattern we encapsulates the test steps inside a "pipeline" abstraction 
and have scripts "launch and leave" their requests at the entrance of the pipeline.

• Composite

Helps us to create Page Objects by forming a tree structure and ask each node in the tree structure to perform a task (loading, verifing itself). 

• Flyweight
• Interpreter

Can be used as rules engine to support business logic in tests or on creation of their fixtures.

• LazyInitialization

Provides delayed execution of certain tasks. Good example is a Shared fixture scenario or 
DB sandboxing.

• Mediator

Since it encapsulates how a set of objects interact, we can use it to share test execution data
and analysis between Reporting systems.

• Module

Extensibility modules let us consolidate our plug-ins into a centralized place. Good fit for this case
are the different Reporting systems we use.

• Multiton

Helps us to manage a map of named instances as key-value pairs. Also simplifies retrieval 
of shared objects (fixtures).

• ObjectPool

Uses a set of initialized objects kept ready to use, rather than allocating and destroying them on demand. 
Such expensive objects could be our test fixtures which we would like to re-use, but only one at a time 
between multiple parallel tests.

• Observer

Since it define a one-to-many dependency between objects so that when one (test) object changes state,
all its dependents are notified and updated automatically. Good fit are the Reporting systems 
that needs to be notified of test's status.

• PageLoader

Encapsilates navigation logic over site's pages via Bidirected and  Cyclic graph.

• RAII  

By holding a resource tied to object lifetime, we can destroy all of them at the end. Good examples are Transaction roll-back and DB cleaning.

• Servant  

Shared code for a group of classes, that appears in only one class without defining that functionality in each of them. Typical example is REST RequestSender that takes care of the multiple Contracts between the APIs.

• State

Allow an object (fixtures) to alter its behavior when its internal state changes, so our tests could make use of it.

    The implementations are mainly focused and do represent Automation testing perspective. A good example is the Observer  pattern used to show how we could notify different systems for the test run output results. 

    You can find the GitHub repo here...

Coder scroll

 

    Some consider programming to be a form of art (including myself). Simply put, there is no easy way of accurately measure artwork. Small paintings can be more meaningful than big landscapes, even just one unique perception of a woman's smile can put your framed canvas at the Louvre Museum. If we all aim at our masterpiece, how we know when we have achieved it? If we're constantly learning new techniques and tools, can we ever find our best work?
    During the past several years, I have collected some of the coding techniques that I consider worth remembering and using. But as the list grows I'm staring to realize that the art is combining them, not utilizing all. So probably this is the best place to put a

Disclaimer: Code following all the principles described in the list is not possible but we should try to follow most of them until there is a suitable reason for not following any.

    You can find the coder scroll here...

    If someone knows such principles, techniques, rules or whatever we find useful and it is not currently present in this collection, please leave a comment and I will update the scroll.

      

OF LOCATORS AND MEN

    If you have some experience in UI automation and testing, probably you already are familiar with the bitter taste of the ever-changing DOM. As most of your maintenance of the code is related to constant updating of WebElements (their locators) and minor scenario flow fixes. At first this may look like the Sisyphus task. I guess at some point we all ask ourselves - is there a better way? Let's leave aside all we have been taught in the numerous courses, books and articles about Selenium's location strategies. I will use web site automation and XPath in the examples below.
    One of the important principles of programming states

 Abstractions should not depend upon details. Details should depend upon abstractions.

So, why we bend our thinking in favour of bots? We desperately try to enforce the rules of low-level implementation instead of working with abstractions? If you use Selenium-like libraries to create your frameworks and/or crawlers, probably you locate your DOM elements like this:

    Nothing wrong here, it works well and tests are up to speed, again. For a very long time I did stop here, happy that I've fixed the tests. But in a while (usually too soon), this scenario happens again. I agree that my Selenium based tests require such location, but the approach is somehow messed up. Clearly, something is not as it should be. I used to blame development, processes and whatever comes to be on my way. But I am the owner of my code, so it's me to blame.
   
Focus on end-users.
    I guess you all have heard about Look&Feel in web design. It can be explained simply as aspects of the UI design, including elements such as colors, shapes, layout, and typefaces (the "look"), as well as the behavior of dynamic elements such as buttons, boxes, and menus (the "feel"). Without going further in UX design details, we could say that this is the brand and the users - two of things we care most. After all, the product (SUT) is made for humans, not for crawlers. If you work in any kind of Agile probably you have to start developing tests in parallel to the functionality. In a good company some Mocks can be found describing/displaying how the GUI will look like. Once agreed by the business people, those (almost) never change.
   
Stop thinking like a bot.
    We found our abstraction, insensitive to the HMTL implementation details. Humans will need text to understand the sites. This is the main pillar in our new and upgraded location strategy. Use labels, input value attributes and text to locate your elements. We could actually imply a pattern here - between text and web elements. After label with text "Username", most likely there always will be an input for your username.
    We can make good use of XPath's advanced functionality like

•  conditions
•  functions 
•  syntax

    Let's look again the Username input field example

    It might look a bit complex, but once you get to use it you will see that the stability of your tests is increased significantly. Making your XPath locators smarter will save you lots of maintenance.  They expect changes and handles them elegantly. I've made some metrics gathering and it turns out that for a build of 200 unique test runs and 54 minutes, the execution was slower with 44 milliseconds than before. A small price to pay since the browser's xpath engine actually performs the calculations, not your code.  

Blackboard Design Pattern - Layout Automation

  

    This software pattern is one of the few that still require serious Googling to be found. If it wasn't for a research PoC project I also would pass by it. But after a couple of days working with this concept I have to say, that I haven't been so interested in an idea for a long time ago. You will see it being classified as  behavioral or even architectural design pattern - depending from the usage. Most of the times its use case is in the fields of AI computing. Yes - it is such powerful concept. Even if my demo is just scratching the surface I also used such frameworks. In my case - image recognition. I already knew such library called AForge.Imaging, so putting it all together wasn't that hard.

   My vision that I had to prove was Stable web site automation based only on the layout. If you ever had your hand on web site testing you know the constant shifting sands of this domain. So the questions is - how to provide reliable UI tests? It is obvious that most DOM based solutions are highly dependant to the changes, no matter how good you are with XPath or CSS locators. But if we use the web element's layout we could decouple our tests from the HTML. Instead of keeping text based selectors, now we work only with images. And they don't change every day, once you have the designer's mockups you are done with the maintenance. This idea is not new and tools like Sikuli are gaining ground as we speak. But they do have certain disadvantages stopping them from dominating the domain. First one is that they require personal desktop - you can't run in parallel. My approach solves this with Selenium's TakeScreenshot  interface.  So you can keep the speed of the tests and only throw iron (computing power) when you need to scale it.

    Let's get back to the Blackboard pattern. Its main advantage is that works naturally with the development mindset. This pattern solves problems when there are no deterministic solutions known. We do this on a daily basis. It helps us coordinate separate, disparate systems that need to work together, or in sequence, continually prioritizing the actions it takes. It also allows our code to work in multiple threads to work closer together on separate processes, polling and reacting if needed.

    Example code implementation...

LoadableComponent is AJAX

    In my test automation I have used the LoadableComponent class aiming to make writing PageObjects less painful, like it was suggested by countless articles and their authors. They try to convince you that this will provide a standard way of ensuring that pages are loaded and you can use it to help reduce the amount of boilerplate code in your tests, which in turn make maintaining your tests less tiresome. 

    The truth is that we should consider the other side of the coin as well. This pattern violates at least a couple of principles. First is KISS, in 21st century HTML5 your pages should only keep content, not inline CSS or JS. In our test design this means - WebElement map only. Till now I was sure that  there are two basic ways of using my OOD Models -  smart tests or smart pages. All test flow logic had to be placed in one of them. The real question is - do we have to? 

    Next big thing that bothers me is the Single responsibility. It clearly states that a class should have only one reason to change. Is this the case with the examples (both simple and advanced) we are suggested to follow?


    In my humble opinion SOLID design has two kings, SRP and DIP.  Both OCP and ISP confirm the first one. There are a few other things we should consider when aiming at clean code. Overdesign is one of them. Do we need the complexity of having two (at least) classes instead of one. Keep in mind that PageActions class is also a good candidate for implementation. We need to separate the associated/needed actions per page as well. To simplify our lives, we can make the page validation a part of the Actions class. All this could lead to bulky test code. We need to draw the line here - loadable components are not page objects. 

    Using LoadableComponent in every website automation, without considering its pros and cons is not a good practice. After all they are just simple objects that encapsulates logic for page loading.  Do we need to have 20 lines classes? And do we need to couple those to the page objects? If your answer is yes, at least try to use Composition over inheritance and avoid the recommended samples like this one for your PageObject model

   public class ProjectPage extends LoadableComponent<ProjectPage> {
      ...

    Just avoid the overcomplicated hierarchy when you try to emulate real user experience in your E2E  "walking through pages" scenario features. 

    Having in mind all of the above said - when we should consider LoadableComponent? This solution works well in complex "deep" page hierarchy (like shopping journeys) and with no confident URLs (like SPA). The correct answer is AJAX-rich web sites. If your front end is build on top of any MVC, then maybe using partial URLs navigation is a better choice. Still, you have to keep some tests which will exercise the end-user journeys. One way to manage your page objects and their relations is a Graph. We do need a root role for some of the vertices, but this doesn't limit us to the Trees only.  I prefer to represent the graph with explicit edges at the cost of some additional memory. In fact the LoadableComponents are the Edges we need for our Vertex (page) objects. This way we could traverse our abstract data type representation without complex algorithms, only using node links. First we need to add abstraction to our Models


And Page objects containers don't need much of it. By implementing IPageVertex you can provide your own IWebElement map mechanics. Note that this usage is much like ISerializable in C#.


    Now our Edge implementations look like this

Testing in production a.k.a monitoring

                              

    Web applications architecture and functionality becomes highly interactive, often executing complex client side logic in order to give a rich and engaging experience to the end-user. This inevitably  adds a certain complexity, meaning that it's no longer good enough simply to measure the effects users have on the back-end. You will need specific solution in place to take care of your business's most valuable features. It has become necessary not only to measure and analyse the client-side performance, but also to make sure the best possible 24/7 experience for your users.

    When I say “testing in production” you might recall a QA team that uses API heart-beat (CRUD) calls in hopes of keeping the application metrics up to date, but in reality this is no longer enough. End users may be were the ones who suffer, while your back-end continues to assure you that everything is OK. For many reasons, most businesses avoid testing in production altogether because it’s too risky for the end user. But the ones who knows how much a split second costs  - it's mandatory to monitor their user journeys and analyse the performance metrics gathered around the clock.  

    Solution is clear - proactively monitor your application. Simulate user behavior to catch problems before your customers do. One way to go is using some of the SaaS platforms that will take care of this critical activity for you. They will run automated tests around-the-clock and around-the-world for you. Complex software architecture? Multi-step user flows? Single page app? On-page interactions? They’ve got it all covered. You can easily customize and monitor using straightforward scripting.

You want your app to work correctly all the time. Troubleshooting and error identification are necessary parts of smoothly-running software. You'll get simulated essential user flows and interactions. Reproducing business-critical functionality looks quite easy. You'll also stay informed about the availability and reliability of your software and ensure an error-free product experience for your customers. Let's not forget the instant alerts (to your service desk or app support).

                                  

    However, you can develop a homegrown solution if the providers are not what you're looking for. In most cases for monitoring purposes a full scaled solution is an overhead, that can be avoided. You will get all the cloud benefits at a reasonable price. I've done some research and my weapon of choice is New Relic. Not that the competition is far behind, but it has some catching to do (especially with API headers in the requests). 

    A brief intro - setting up test monitors with New Relic Synthetics are easy. What you need is just to enter a URL and select a few configuration options. They will launch a Selenium-driven Chrome browser that sends relevant data immediately using turnkey test scripts. I'm more interested in building our own scripted monitors. New Relic Synthetics simulates your real customer experience starting in no time. 

  Read more about the project...