Selenium. Powered by: Computer vision

    I've already talked about the user agents, self-sufficient software and "no man in the loop". So now I'll turn to the rest of them that we need for implementing our own page scrapping  functionality we're going to use to take our Selenium framework to the next level.

    The big picture for the computer vision that we need to see includes the image recognition domain. These days there are a lot of frameworks that can are open source and can give you enough AI, to power your tests like never before. Even such small piece like analyzing images will take a lot more than single article explanation, so for the sake of consistency - I'll talk only about comparing images.    Page object model and Loadable component  will feed our new functionality. I don't hide that I don't like the concept of Loadable component (mainly because violates the Single responsibility principle, by putting actions logic in the presentation abstraction), but in this case it will do its job just fine. So our Page classes will be the only thing that the automation tester should create by himself. All the rest should be handled by the PageHarverster. If we want to take all this further - why not utilize a CookieCutter, which will create out template classes using the URL. For example:

     http:\\www.mysite.com/ will be HomePage class

     http:\\www.mysite.com/Account will be AccountPage class

     http:\\www.mysite.com/AddCard will be CardPage class

the WebElements can be created with some pattern that we found is being used by the frond-end devs (maybe they always put main elements in form), still we will need some adjusting since the XPath auto-generated values will be too long and fragile. This is the place where I should also mention another major concept - Web page is just a grouping of services. Some of them are more important, than others. But they all "live in containers". Those containers are exactly what we need for the Image recognition part. For example:

     Our Login page has two inputs (username and pass) and two buttons (join and login).

What is the point of validating them separately as WebElements? If even one of them is messed up - the whole service will suffer. That why we need the container and validate it as grouping of WebElements that has certain purpose. Here, the image from the Screenshot taken of this container will guarantee us that no regressions occur.

    Web harvesters by design are a perfect fit for our case - they  simulate human exploration of the WWWeb, they embed a fully-fledged web browser (IE, Mozilla Firefox) and they focus on the transformation of data on the web into structured one that can be stored and analyzed (in our case image files).

   

    If we combine all this together, we'll get autonomous test engine, that can recognize which site should visit and it's pages, will have enough knowledge which web elements (containers) to take and how to verify their content for us.

    In this Git repo you can find pseudo code for such WebHarvester based on Selenium's Firefox WebDriver. I agree that the code is kept at low level, but you find it easier to implement later. If I had to stick to high level descriptions only - you'll probably need pretty much the same time as me. In case you're wondering what is the 3th party framework I've used for image comparison - it's AForge. 

   A quick walkthrough - ResourceManager tracks down your Page classes in the test project's structure, via reflection initialize them as Loadable component and extracts collection of WebElement declared as Container. It's important to mention that every LoadablePage class should have knowledge how to load itself.  Once available on the loaded page WebElements are being extracted as image files (thanks to Selenium's full page screenshot functionality), later processed, analyzed and compared with the expected image of the same service container (thanks to the ImageTemplateComparator).  

   One key feature is that you as a tester should not work with any images. The Facade (PageCrawler) will handle them internally for you. If there are no expected images for the declared container, it'll take, store and use one from the current run. In order to keep the solution clean - the Git push (with new image files if any) is put as a separate CI server's build step. The connection and the objects are too expensive to be kept on the run.