Yesterday, we were giving a demo of our mobile test automation POC to a client for their android app in the financial
domain. It was a simple scenario with
login to the android app and then testing the market watch feed to the logged
in user.
The screen
as below:
The POC was
based on the existing manual test scenario with the suggested data from client.
Using the correct login id and password takes the user to the welcome screen. If the user id or the password is not correct
then the application would throw an error message and request you to try again
with valid credentials.
After the
demo of the successful login and further steps in the POC, the client representative
asked us to update the test to use invalid password which was very long. The data suggested was “anand long name in the password 8hj”, this string was 35 character
long. When we reran the test the login was
unsuccessful as the password was invalid, however the password field didn’t
accept the full string and truncated it to 12 characters “anand long n”.
So the conclusion was that the test passed as it didn’t allow to login
with invalid password with an observation that the test didn’t do enough to
catch the fact that the entire password string was not used to run the test and
the test data for the password was actually truncated.
Now should
the above test trap this condition and not allow any further characters post 12
characters to be typed into the password field?
Should it throw an error?
What do you
think?
Here is my $0.02…
I would look
at the behavior of the system in the requirements in the same scenario. What happens
when we manually run the test? Is it
conforming to the requirement? The manual
test and thus the test automation script needs to validate this behavior of the
system.
It would be
wrong for a test case to validate something which is not part of behavior of
the system and even worse trap and override the system to give warning and
errors.
In the above
test case when I manually enter the password, after 12 characters the system
does not let me type any more characters, but please note that it does not stops
me from entering any further characters into the field. If I was not looking at the screen then I
would simply type away all the characters and process with the next steps. A test automation tool is like a person who
does not look at the screen only interacts with the system under test.
This seems
innocuous at first but consider this, if the valid password is “123456789012”
(12 character string) and I used test data as “123456789012345” (15 character
string), the system would allow me to login with invalid test data as it would truncate
the long string into a valid password.
http://www.reactiongifs.com/wp-content/uploads/2012/11/typing.gif
Now it is up to the users to decide what should system do if I am happily typing away even though the password field allows max 12 characters?
1. The system should let user type more than 12 characters and not truncate (as it may pose a security risk to reveal the max length of the password)
2. The system should pop-up an error message that user has reached max characters allowed in the password
3. Or the existing way where the users can type as many characters but system will truncate it to max allowed characters without knowledge of the user.
Once we close on this system behavior from the
varied perspectives of functional as well as UX testing then we need to accordingly
validate this behavior using manual and automation test cases.
I would also
like to bring your notice that above scenario is also an example of how testing
can lead to functional as well as the user experience requirements when it is either done along with or prior to system requirement gathering. This will help us study requirements from a behavior
perspective which can then evolve and lead to better test cases and eventually high quality.
Happy testing!!
