A blog post has been well overdue for me- but hey better late than never.
In this post I want to bring together a mixture of current experience, some theory and a word of caution when tackling low-level automation and testing. An observation and point of note to anyone who finds themselves in a similar situation now or in the future. I recently found myself in a very interesting space on one of my projects which got me thinking about a potential point of caution when one is looking to automate or even test across the different levels of your system.
Recently there has been this great ‘Shift Left’ concept which was mostly spearheaded by the famous ‘Automation Pyramid/Triangle’– yeah that famous pyramid again. I am not putting that pic up as I think that pyramid is tired of making so many appearances 🙂 Instead I will use this pic of a real life pyramid for effect. If you need to check-out all about the automation pyramid, then I suggest you do some reading on it. Check out these 2 links for a start : https://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid and https://martinfowler.com/bliki/TestPyramid.html
Setting the scene
So where am I getting to? I must highlight 1 point before I go any further the following dilemma I mention below would not be evident on all types of systems that one faces but it would be the reality in most modern systems/applications including Low-code type systems and systems where there is a some sort of manipulation or translation from the UI to your middle layer or even API’s . Regardless the question begs to be asked on every project that you tackle just to reassure that the Automation and testing approach that you follow gives you a peaceful night’s sleep.
Well consider the following scenario on your project, you have a shiny new application with some cool front end tech coupled with some sleek back-end engineering. You wonderfully decide to follow this magical Automation pyramid, you think you are going to totally smash that Automation pyramid/triangle model by working real close with your devs to create superb unit tests, you want to add in a dash of component tests to further strengthen what you covered in your unit tests. You don’t stop there you decide want to throw in some fancy Integration tests. 1 last topping on this amazing triangle- you ponder a hint of E2E UI tests to top it all off. Not too much just the right amount. How awesome you think, this defence I have planned to build up is surely watertight!
What they don’t tell you!
Based on the famous triangle or pyramid this is usually how people want to tackle their Automation. But remember, it’s not magic! its not plug and play, you have to keep asking the questions. What am I testing? Am I automating just for the sake of it? am I automating or testing here just to fit in to the Automation pyramid?
When you get back to your desk to start on this cool testing task, you pause and think- hang on something is amiss! something is just not adding up here. Create unit, component and integration testing–but where? Well I found out the hard way and also found out where the major shortfall was on my project- that I will explain afterwards.
The simple answer is the Automation triangle needs to be expanded in practice. One needs to break it up even further when you tackle the actual effort on your project. This effort then explodes- as I said its not magic and one has to prepare for this personally and let it be known to the team and relevant interested parties- as this effort ultimately affects the sprint and in-turn estimation which then impacts actual time to delivery.
My attempt at breaking up the Automation triangle according to my experience is depicted below. I will then give a brief explanation below. The reality looks like this!
So now that I have broken the triangle up a bit more to put theory into practice by aiding actual day to day activity or project contribution. Lets unpack it a little more.
A testers full testing effort is usually not limited to just an API or one part of the UI. A testers effectiveness is evident when he/she has the full picture in his/her mind, testing the entire application as seen by the user and beyond! Front end, back-end, logging etc. With this in mind if the true sense of ‘Shift left’ is to materialise one needs to cover testing and automation at every possible level. UI and API (and whatever other backend magic that exists)
That then means that triangle now needs to be less vague and more specific to include both UI and API/Backend tests across the different levels for it to make practical sense to the actual testers who need to carry out the detailed tasks at hand.
UI Unit– usually done by the devs to cover unit level of a UI validation, field, text etc
UI Component –usually written by testers to see how that component/area of the UI behaves usually by mocking out its dependent calls or API’s
UI Integration (within the application itself)- usually written by testers to test how that new component behaves when integrated within the application at large. For example does a new field/element that is recently added to the application conflict with an already existing field or element- or just finding any weird general issue when this is merged to the greater individual application under test
UI Integration (to APIs/Backend)- usually written by testers to test one of the most important aspects-bringing the Front end and back-end together. Passing on variables and composing a payload or xml to be sent to the API or Backend/webservices etc.
API Unit-usually written by devs, where testers atleast have some visibility into coverage. Covering unit/s of an individual API composition
API Component-usually written by testers with dev assistance and input. Consisting of mocking of dependencies, run independent of environmental related data.
API Integration (to backend and/or other API’s)-usually written by testers. Consisting of actually calls to dependent APIs and/or downstream systems/dbs
E2E UI- usually written by testers- light coverage to bring all of the above together and just provide re-assurance that all is in order- Run in a fully integrated non-mocked working environment
Hope that makes sense 🙂
As mentioned earlier on the project I worked on I found this out the hard way. I actually had multiple tests and some great detailed coverage across all areas of the triangle accept for 1 area.
Any guesses where?
Well that area proved to be the UI integration to API. I had covered tons of different API scenarios and I figured I was safe when it came to hooking up the UI. But that wasn’t the case at all. The manipulation of variables and parameters from the UI as they passed through various different workflows that composed the payload, which then passed on to the actual API’s proved to be the big problem area. Individually everything worked great, but when integrated it didnt play nicely.
This was quickly identified and a solution suggested to hook into this UI workflow integration to API level and bolster coverage at that level to prevent any gaps in coverage.
So basically the lesson learnt from this is to understand your application and its inner workings when approaching your testing efforts. Following a model or blueprint blindly without asking the relevant questions can be catastrophic. The key is that a model is just a guide and should be used that way, for maximum effectiveness you need something that suits your situation. It can be tweaked to suit you unique situation or system.
Keep challenging your own thinking and the system will be a better place.
Finally you can sleep peacefully, knowing you have your bases covered!