Testing the system behavior which is not there

Yesterday, we were giving a demo of our mobile test automation POC to a client for their android app in the financial domain.  It was a simple scenario with login to the android app and then testing the market watch feed to the logged in user.

The screen as below:

The POC was based on the existing manual test scenario with the suggested data from client. Using the correct login id and password takes the user to the welcome screen.  If the user id or the password is not correct then the application would throw an error message and request you to try again with valid credentials.

After the demo of the successful login and further steps in the POC, the client representative asked us to update the test to use invalid password which was very long.  The data suggested was “anand long name in the password 8hj”, this string was 35 character long.  When we reran the test the login was unsuccessful as the password was invalid, however the password field didn’t accept the full string and truncated it to 12 characters “anand long n”.  So the conclusion was that the test passed as it didn’t allow to login with invalid password with an observation that the test didn’t do enough to catch the fact that the entire password string was not used to run the test and the test data for the password was actually truncated.

Now should the above test trap this condition and not allow any further characters post 12 characters to be typed into the password field?  Should it throw an error?

What do you think?

Here is my $0.02…

I would look at the behavior of the system in the requirements in the same scenario. What happens when we manually run the test?  Is it conforming to the requirement?  The manual test and thus the test automation script needs to validate this behavior of the system.

It would be wrong for a test case to validate something which is not part of behavior of the system and even worse trap and override the system to give warning and errors.

In the above test case when I manually enter the password, after 12 characters the system does not let me type any more characters, but please note that it does not stops me from entering any further characters into the field.  If I was not looking at the screen then I would simply type away all the characters and process with the next steps.  A test automation tool is like a person who does not look at the screen only interacts with the system under test.

This seems innocuous at first but consider this, if the valid password is “123456789012” (12 character string) and I used test data as “123456789012345” (15 character string), the system would allow me to login with invalid test data as it would truncate the long string into a valid password.

http://www.reactiongifs.com/wp-content/uploads/2012/11/typing.gif

Now it is up to the users to decide what should system do if I am happily typing away even though the password field allows max 12 characters?  

1. The system should let user type more than 12 characters and not truncate (as it may pose a security risk to reveal the max length of the password)

2. The system should pop-up an error message that user has reached max characters allowed in the password

3. Or the existing way where the users can type as many characters but system will truncate it to max allowed characters without knowledge of the user.

Once we close on this system behavior from the varied perspectives of functional as well as UX testing then we need to accordingly validate this behavior using manual and automation test cases.

I would also like to bring your notice that above scenario is also an example of how testing can lead to functional as well as the user experience requirements when it is either done along with or prior to system requirement gathering. This will help us study requirements from a behavior perspective which can then evolve and lead to better test cases and eventually high quality.

Happy testing!!