This section is the one I enjoyed writing the most and is the longest in this tutorial by far…so perhaps now is the time to get a cuppa and take a breather before you get started 🙂 !
I’m afraid that it has a lot of content, simply because it needs to, showing you powerful tools and techniques to help you design and test Systems that are actually running, allowing you to perform…Dynamic Testing!
Yes…the section is a little involved and will take some thinking about in order to understand things, but once digested the knowledge you will gain will be well worth the headache! Did I say that learning Software Testing would be easy? Perhaps not…and that is why as Test Analysts we are paid very good money!
Table of contents
- Tools and techniques to actually do some testing
- Requirements Traceability Matrix – a must have!
- Test Case Development
- Test Case Design Techniques
- Summary
- Test Yourself
- What Next?
1. Tools and techniques to actually do some testing 🙂 …
As we saw in the last section, the Verification Testing Process actually “tests” the deliverables of both the development and testing stages by reviewing documents and code. This section now brings us to the test design techniques used and the actual execution of testing.
The Validation Process (Dynamic Testing) is done by executing the code, using Dynamic Analysis Techniques to test! As you can see from the diagram above, we can break this process down into 2 main areas, namely Functional Testing and Non-Functional Testing (e.g. performance, scalability, usability testing etc). We will only be looking at the Functional side in this tutorial, with the Non-Functional side being explored in a future tutorial.
Functional Testing can be performed manually or by automation. We will only be looking at the manual side of things, with automation, again, being explored in a future tutorial.
In order for you to perform Functional Testing, we will be looking at Test-Case Design techniques to identify the tests that need to be made and how to execute them. The techniques you will be exploring will be explained separately, however, I must stress that you should always use a combination of most of them, if not all of them, in order to design strong enough tests.
As a Test Analyst, you will be predominantly using Black-Box testing, however, sometimes you will need to drill down further to test at a more detailed level…this is where White-Box testing comes in.
2. Requirements Traceability Matrix – a must have!
Taken from https://www.perforce.com/resources/alm/requirements-traceability-matrix:
“A requirements traceability matrix is a document that demonstrates the relationship between requirements and other artifacts. It’s used to prove that requirements have been fulfilled”
Before we get into the specific test tools and techniques that are used to actually test, let’s just take a quick look at a tool that I have always used throughout my IT career, which helps me organise my testing strategy, top down, and ensures that I don’t miss anything that needs testing.
If one has not already been produced by the SDLC Team, then I always produce a Requirements Traceability Matrix, as a spreadsheet, so as to identify all the requirements that need testing. I break each requirement down, in the spreadsheet, into the individual functionalities that have been coded to actually meet the requirement.
The purpose of a Traceability Matrix is to ensure that throughout the project, nothing is missed (not only by the Test Team, but by all teams throughout the SDLC, including design and coding teams) by ticking things off once completed.
The following diagram (which I have kept purposefully simple just to show you an example) shows a single Customer’s requirement and how it’s mapped through design, code and finally test, with a corresponding “Y” to show tasks that have been produced / completed:
Please Note: A real-life Matrix would definitely be more detailed. For example, a Functional Requirement would probably have many Code Components (Classes), which would be recorded in the Matrix. Also, one Functional Requirement could satisfy many Requirements
If we look at Requirement R.01.00, we can see what Functional Requirement (F.01.00) tasks have been completed:
- The Design has been specified (written) and reviewed
- The Code has been written, reviewed and Unit Tested
- The Integration Test Cases have been written, reviewed and tested
- The System Test Cases have been written, reviewed and tested
Once a Matrix, as above, has been produced, we can then trace each Requirement and ensure it has been satisfied (delivered) when all the steps have a “Y” in them.
3. Test Case Development
Test Case Development is ultimately all about producing Test Cases…they are the lowest level test component. We derive what functionality (Test Scenario) we are going to test from the Requirements and Design Specifications and then produce the Test Cases for these, with Test Steps showing us how to run the test case.
If you browse the web or read the multitude of available Software Testing Books, you will find quite a variety of different names for Test Scenario, Test Cases and Test Steps. You will see them called such things as Test Conditions, Test Procedures, Test Scripts etc. Well…I like these terms Test Scenario, Test Cases and Test Steps! Why? Because I think they explain what they are in the simplest of terms 🙂 !
As you will see from the diagram above, a Test Scenario can have many Test Cases, which in turn can have many Test Steps. I will explain each of these components, as follows, then I will use a real world simple Test Scenario to show you how they are used.
Test Scenario
A scenario is defined as any system’s functionality that can be tested, for example, scenarios such as:
- Check that login page functions correctly
- Check that personal details can be entered and saved correctly
- Check that the bank balance is updated correctly, when transactions are made
You will derive these scenarios from the User Requirement or Functional / Technical Design Specification documents. Note: I will show you a neat tool to aid you to be able to identify these scenarios in a future post (using DFDs (Data Flow Diagrams) to identify your tests!).
Test Case
Test Cases are a set of specific actions that are executed so as to validate the correctness of a Test Scenario. A Test Case contains the steps (needed to execute the test), test data, conditions for the test and a record of expected and actual results.
Test Steps
These describe the steps that need to be taken to execute and complete the Test Case.
3.1 “Home Finance” Example
So, let’s look at an example so as to explain how to design and write a simple Test Scenario and Test Case.
For this example, above, we have a web page where you are required to enter your personal details. You enter the page from the Settings Page and enter details into the 4 fields as per the requirements below. The fields are then validated when the SAVE button is clicked. If any of the fields are invalid, then they will be highlighted and an appropriate error will be displayed and you will stay on this page. If the fields are all valid, then you will be returned to the Settings Page and a message will be displayed confirming that the details have been saved.
Field Requirements:
- Surname
- It is mandatory
- Must be alphabetical, except for a hyphen being allowed
- This is not case sensitive
- Must be between 1 and 50 letters
- Forename
- It is mandatory
- Must be alphabetical, with only a hyphen also allowed
- This is not case sensitive
- Must be between 1 and 50 letters
- Adult (18+)
- Default = “Y” for people who are 18 or over
- Can be changed to “N” for under 18s
- Age
- It is mandatory for people over 18
- Must be only numerical
- Must be between 18 and 150
3.2 Now, let’s create Test Cases for the “Home Finance” Example:
The Test Scenario for this example would be something like:
Check that personal details can be entered and saved correctly
As you can see from this scenario, it is quite vague as to what needs testing, that is why we have the Test Cases…to add more specific detail to the tests.
Believe it or not…the number of tests for this very, very simple page can run into 20 or 30 tests alone! Not only are we checking for the functionality (where I would write 14 tests at least) , but we can also check for the font size, position of all the fields, the button and title…but let’s just stick with testing the functionality for now.
Test Case Examples:
The following 3 Test Cases show how tests could be recorded. (Note: Any of the Test Management Tools currently on the market will have similar ways for you to input your test results like this).
Test Case 1 has been Passed, because the details were SAVED correctly.
Test Case 2 and 3 have been marked as Fail, as the actual result shows that the Personal Details were SAVED, when there should have been a message displayed to say that some invalid data had been entered!
As said previously, the above diagram shows just a subset of the Test Cases that should be produced for the “Home Finance” Example. Looking at the Test Cases, hopefully, you will be able to see how we write Test Cases and specify a Test Scenario, Test Case and Test Steps and how they relate to each other.
4. Test Case Design Techniques
To help in the development of Test Cases, I would now like to introduce you to 3 main standard testing techniques:
- Black-Box (Specification-based)
- White-Box (Structure-based)
- Experienced-Based
4.1 Black-Box (Specification-based) Testing Techniques
In Black-Box Testing we identify all the Test Scenarios that need testing by looking at the Requirements and Functional Design specifications (Note: You should not have to use the Technical Design specification, as that would be used for White Box testing).
We are not testing the inner workings of a system, program or class, but testing that the interfaces between a system’s programs / components (the inputs and outputs) work and that they integrate as they should.
As an example, if we were to test the Google Browser, as a scenario, we could enter a phrase in the “Search Box” that we want information on and check that the search results return the correct data, we are not testing how the data is returned…how the internal workings of the Browser work, just the input and output!
So…the steps for the Test Analyst to perform Black-Box Testing would look something like:
- Examine the Requirements / Functional Design Specifications in order to identify the Test Scenarios that need testing
- Write the Test Scenarios
- Write the Test Cases and Test Steps, identifying the required data (to test all the appropriate test scenarios) and the Expected Outputs. The data will test both the positive and negative Test Scenarios (which will be explained below)
- Execute the Test Cases
- Compare Actual Outputs with Expected Outputs
- Raise any defects where needed and retest
4.1.1 Types of Black-Box Testing Techniques
To aid the writing of Test Cases, there are 4 main Black-Box techniques that we can use:
- Equivalence Partitioning
- Boundary Value Analysis
- Decision Table Testing
- Use Case Testing
These techniques allow us to explore the Requirements and Functional Design Specifications in a structured, systematic and thorough way, making the creation of Test Cases as efficient and complete as possible.
NOTE:
In both my roles as a Developer and Test Analyst over the years, I would use all these techniques! As a Developer, I would map out all the possible conditions in a Decision Tree to ensure that I had coded all the possible legs (rules) in a program, I would then use this to test that each leg worked as it should.
4.1.2 Equivalence Partitioning
A good test is one where we have a good chance of finding an error, but bear in mind, as stated in the earlier section 2.5 Principles of Software Testing – Exhaustive testing is impossible, an exhaustive input test of a program / component is impossible in most cases.
Using the Home Finance Example, above, there is a field where you can enter an age between 18 and 150. To test every permutation, we would have to enter each number between 18 and 150 (which equals 133 options), plus entering numbers outside this scope (e.g. < 18, > 150 and negative numbers), along with all the non numeric characters on a keyboard (to perform negative tests) to ensure the test is complete.
To make testing the above scenario manageable, we must reduce the number of tests needed by creating Equivalence Partitions. Equivalence Partitioning is a way of methodically reducing the huge, or sometimes impossible, number of tests needed into a much smaller effective set of tests.
Using the “age field” example again, we could, possibly, enter:
- 18
- 99
- 110
- 150
- 1
- 11
- 151
- 999
- -18
- -101
- H
- ?
If we look at the above set of numbers / characters, we can see different equivalence partitions:
Positive tests (testing that the system works as it should) entering:
- 18 and 99 – being 2 number values
- 110 and 150 – being 3 number values
Negative tests (trying to break the system) entering:
- 1, 11, 151 and 999 – being out of scope numbers
- -18 – being a 2 value invalid negative number
- -101 – being a 3 value invalid negative number
- H, ? – being non numeric characters
So…there we have it, we have reduced the number of tests (running into the hundreds) that should be made to perform a complete test to a small number (12) that we could comfortably do and still be confident that we have tested enough to ensure no bugs exist.
By following the Equivalence Partitioning thinking…we’re now thinking like a real Test Analyst 🙂 !
4.1.3 Boundary Value Analysis
It is well known that mistakes made by developers are around a system’s boundaries. This is where we can see clusters of bugs. What do we mean by this?
Again, if we use the Home Finance Example as an example and the Age field to explain…
The most likely problems will be found around the entry of the boundary values (these being 18 and 150). You may find that the entry of a number outside the range, i.e. 17 and 151 will cause problems. This is due to how programming is performed. When coding ranges, loops (iterations) are used and this is where a developer can (and will at some stage in their career) make mistakes.
Me…personally, I would use:
- values outside the range
- the actual range values
- values just inside the range
- i.e. for the Age field this would amount to: 17, 18, 19 and 149, 150, 151.
Top Tip: Testing when processing a file:
A really common example of boundary mistakes I come across time and time again, is when a developer has coded a file to be read. I always test for an empty file! And you’ll be amazed how many times I’ve had to record a defect because the program crashes!
In the case of the file, I would also test for a program processing:
- no records found in the file
- 1 record found
- > 1 record found
4.1.4 Decision Table Testing
So, we have seen how both Equivalence Partition and Boundary Analysis Testing can provide us with some sort of structure to our testing and, hence, in producing our test cases…which is great! For me though, this is where I actually start designing my tests…using Decision Table Testing. This allows me to identify my Test Cases from the Requirements and Functional Design Specifications.
I think the best way to explain this technique is by using the Home Finance Example again. A Requirements Specification would include a diagram of the Page Layout and the field requirements for the Page to be saved, as in section 3.1 “Home Finance” Example above. It should also provide the messages that will be displayed if any of the fields are invalid.
Here is the Personal Details Page again to remind us:
Using the details in the Requirements Specification, I can then create a Decision Table of all the conditions that can occur when performing the Test Scenario “Check that personal details can be entered and saved correctly”, by looking for all the combinations and their outcomes as follows.
Note:
If the condition is not true, then it will be marked with a corresponding “N” in the rule column and if it is true, then it’s a “Y”.
For each rule, there will be one or many actions that should be performed:
To explain the first rule:
If you enter:
- an invalid Surname, say with a special character of “&” in it
- a valid Forename
- an Adult (18+) of “N” in the dropdown
…then a message should be displayed saying “invalid Surname”.
I actually create all the negative testing rules first, building up to the rule for the positive testing (how the system should perform all being well)
NOTE:
In each of my roles as a Developer, Designer and Test Analyst, I would use these techniques! As a Designer, I take User Requirements and produce the pseudo code that Developers will code from
4.1.5 Use Case Testing
Use Cases can be used by the Test Team to produce Test Scenarios (performing Use Case Testing).
At the design stage of a project Use Case Designs are produced, usually by a Business Analyst or Software Architect. So…as a Test Analyst, you will simply be using a Use Case to create your Test Cases from, not producing one! A Use Case:
- shows how a system interacts with its environment (how it interacts with a User or another part of the system or even another system)
- shows the system interactions by defining the activities that are performed by Users (known as Actors in Use Case Design) to achieve certain functions
Let’s take a look at an example of a Use Case Design to better understand things:
Using the Home Finance Personal Details Entry Web Page, above, the Use Case Design would be something similar to the following, where I have shown the first two exceptions after the happy path:
As shown above, the Use Case can have different scenarios. A Use Case will have a “Happy Path”, one that represents the typical sequence of events and responses when a User uses the system. The same Use Case will also have “Alternative Paths”, ones where an exception occurs (e.g. in the above example, an invalid Surname or Forename).
So…when producing tests, we would be looking at producing a Test Case for a Happy Path and for each Alternate Path.
Please, please don’t get confused between a Use Case and a Test Case, the following will hopefully explain the difference:
- Use Cases list actions defining the interactions between an actor and a system to achieve a certain function. Test Cases are designed by referring to a Use Case, which have the different conditions to test the behaviour of software.
- Use Cases are used in the design of software. Test Cases are used for testing and recording the actual outcome
- Use Cases are prepared by a Business Analyst to capture the requirements of the client in detail. Test cases are designed by Software Test Analysts to provide the test conditions to test and record results
4.2 White-Box (Structure-based) Testing Techniques
What is White-Box Testing?
Also known as Glass-Box or Clear-Box testing, this type of testing, as the name suggests, tests the internal workings of either a program (component) or a low level design. To White-Box test, you will have to have access to the code or low level design (looking at something like pseudo code) and produce tests from these.
White-Box testing is done at either a Unit Test (therefore, being produced by a Developer) or Integration Test Level. As a Test Analyst you may not need to get involved with this low level testing.
If, Else and Iterations (Loops)
To look at computing in its most simplest form…for Developers, coding is simply about writing a series of “IF” or “ELSE” conditions and some code being REPEATED (iterated), that’s it…as simple as that! So, to test code, we would first look at what IF and ELSE conditions (decisions) are being made and what code is being iterated.
Developers should be unit testing all the legs of a program or a component, to do this they need to identify all the legs that they have coded. Me? As a developer, before I would start to code I would first use the pseudo code from the technical specification to produce a visual representation using a Flowchart of the program (PS. I actually produce a JSD (Jackson Structured Design), which is similar to a Flowchart but is more powerful, something I will explain in a later post).
Using the Flowchart, I would be able to identify every leg in a program and then complete the coding. I would then also be able to use the Flowchart to produce a Decision Tree so as to test and ensure that I had tested each leg.
Let’s use the Home Finance Personal Details Entry Page as an example again:
From section 3.1 “Home Finance” Example, in order to SAVE the page details, the field requirements were:
- Surname
- It is mandatory
- Must be alphabetical, except for a hyphen being allowed
- This is not case sensitive
- Must be between 1 and 50 letters
- Forename
- It is mandatory
- Must be alphabetical, with only a hyphen also allowed
- This is not case sensitive
- Must be between 1 and 50 letters
- Adult (18+)
- Default = “Y” for people who are 18 or over
- Can be changed to “N” for under 18s
- Age
- It is mandatory for people over 18
- Must be only numerical
- Must be between 18 and 150
The Pseudo Code in a Technical Specification would look something like:
From the Pseudo Code, I could then produce a flowchart to show all the paths (legs) that would need testing. Where the Rectangles represent a sequential statement and the diamonds represent the decisions being made:
From the Flowchart, I could then produce a Decision Tree, as was shown in the Decision Tree Testing section above:
4.3 Experienced-Based Testing Techniques
Test using experience
Experienced-Based testing is a technique where you are testing the system using your testing and system knowledge. There is no better way to become a really good Test Analyst than to gain as much experience, not only with the testing techniques explained here, but by learning how commercial systems work, be it in the Banking, Utilities, Manufacturing, Medical…or whatever world!
By “doing” testing, mistakes will be made, you will miss bugs, but this will only reinforce your experience…as you will learn doubly fast from your mistakes!
If you have followed the Black-Box and White-Box testing techniques to test a system, then you should have flushed out all the software bugs that would have existed…well, most of them! However, bugs can still be found even after such extensive testing, simply by using the system either as a (dumb) user who knows nothing about how it works or by one who has extensive experience of the system and knows where problems could lie.
Another reason to use Experienced-Based testing is because of a lack of documentation, from which Specification-based testing can be done.
Dumb User technique
In this day and age, I guess we should be using the term “User who is not knowledgeable of the system”, however, in the IT industry, the term Dumb User is well known to not be derogatory but to refer to somebody who will bash about the system attempting to perform a task…perhaps without a User Manual or guide!
This is a very useful technique, especially when testing online systems. The one test I like to use is to bash the keyboard and enter any character in the fields of a Web Page and see if you can crash the system! Other than bashing the keyboard, this technique can see the uninitiated user:
- Enter data that you haven’t thought of
- Clicking fields on a web page in different orders and clicking areas of the screen that should not respond with an action
- Navigate through a website in an unorthodox fashion
- And many more things…
The idea is to work against preconceived ideas that you will have built up by looking at the documentation of the system all the way through development and test! Set yourself free, think as a Dumb User would and see what happens! 🙂
Exploratory Testing
Whilst working at a large UK corporate retail company, I was asked to do some exploratory testing. I really didn’t understand why we were doing it the way that we were at first! The company was releasing live code every 3 days and the code releases were being backed up by running automation tests to catch any evident errors…so why exploratory tests? Today, it is more than obvious to me…as the systematic functional testing and automation tests could quite possibly miss obvious errors such as Web Page logos missing or being misaligned, fields not accepting data, not looking quite right or even holding the wrong default data.
Having since done extensive Automation Testing, I can tell you now that to do all the testing to show any aesthetic discrepancies, say of the retail Website I was testing, would have meant thousands upon thousands of automation test scripts being written and run!
Therefore, to complement the Automation Scripts, a team of exploratory testers were employed to run every possible test scenario that they could think of within a day to ensure that no obvious errors were found! The test scenarios, which had been formally produced, would replicate what customers would do to view products, add items to a basket, amend things and order items etc, which would entail every web page being visited.
Did the Exploratory Team find errors…oh yes! These were errors that would never have been found by automation tests! And there lies the secret of…Exploratory Testing!
5. Summary
Well, that was quite a lengthy section in this tutorial…in fact it was the longest section! I hope it all made sense, if not, then I can understand as it was quite involved, but don’t worry! Just slowly go over the section again, if some things are not quite clear, and take your time to digest things.
The tools and techniques that you will have learned here are all the ones that will be needed for you to test a System’s new functionality! In fact, if you were to go for an interview now, these skills are the ones that an interviewer would be focusing on to show whether or not you actually know how to do hands-on testing!
In this section, you will have learned:
- You can trace all the components that need testing by looking at a Project’s Requirements Traceability Matrix or if one doesn’t exist by creating your own
- For Test Case Development:
- A Test Scenario identifies a System’s functionality that is to be tested and is derived from a Project’s Requirements or Functional Design Specification documents
- A Test Scenario can have many Test Cases, where Test Cases are the tests that you will be actually executing in order to satisfy a Test Scenario
- Test Steps explain the steps that you will take to complete a Test Case
- Test Case Design Techniques can be broken into 3 main techniques:
- Black-Box (Specification-based) testing
- White-Box (Structure-based) testing
- Experienced-Based testing
- Black-Box testing tests that the interfaces between a system’s programs / components (the inputs and outputs) work and that they integrate as they should. It can be further broken down into 4 sub techniques:
- Equivalence Partitioning
- Boundary Value Analysis
- Decision Table Testing
- Use Case Testing
- White-Box testing tests the internal workings of either a program (component) or a low level design
- Experienced-Based testing is a technique where you are testing the system using your testing and system knowledge
6. Test Yourself
Important to do well…
So, this is your last test for this tutorial! It has 26 questions and as with the other tests, once complete, check your answers, if you do not score that well (let’s say not getting at least 18 correct), please either go back over the tutorial’s section again, perhaps making more notes as you go, or re-do the test to ensure you have a good understanding. It is always important that you understand the section that you have read / learned before you move on 🙂 !
Click here to Test Yourself.
7. What next?
So…you’ve made it! Congratulations! You’ve reached the end of this tutorial. But there’s more…
Download your very own Application to test against
As stated in an article https://www.psychotactics.com/art-retain-learning/ by Sean D’Souza, one of the best ways of retaining what we have learned is to practise our new found skills…by “doing it”!
Yes, hopefully, you will have learned most of what you need to know about Software (Manual) Testing from this tutorial, not only by reading, but by cementing your learning by making notes and doing the Test Yourself tests. However, the “icing on the cake” really is about looking at a real Software Application, writing your own Test Cases and testing it!
If not already done so, again, my suggestion would now be to visit the HomeFinance Application tab on this website, read the Overview and then download and install the application locally on your PC. With the application installed, you can then take a look at it and see how you would test it!
Please Note: Although the installation of this application can at times seem a little involved, it will give you a great deal of confidence in downloading, installing and running software. Skills that will give you added Brownie Points at an interview and stand you in good stead as a Test Analyst!
Also, the added bonus of installing this application will be that you can use it in my future tutorials, where you will be using it to learn more advanced testing skills.
Completing this tutorial…is just the beginning of your testing career!
As mentioned before, hopefully, by going through this tutorial it is just the start of your learning to be a Software Test Analyst! The journey has just begun…as they say!
Yes, you can get by with just learning Software Testing without having technical skills, but if you were to add these “in-demand” skills to your repertoire, then the job opportunities and rewards will be so much more!
With this in mind, please keep an eye out for my future tutorials and interview guides that will help you:
- SQL Basics for Software Testing Tutorial
- Interview Techniques and Questions
- API Testing for Beginners Tutorial
- Basic Java Programming for Testers Tutorial
- Basic Selenium Programming for Automation Testing Tutorial
- Advanced Selenium Programming Tutorial
To ensure that you don’t miss any future tutorials that I publish, please Subscribe 🙂 !
Keep on learning…
In order to learn new things myself, I would go through a tutorial like this first, making notes and testing myself along the way. I would also be looking to cement my learning by reading as much material as I could lay my hands on, perhaps books, magazines, forums or other internet sites connected to the topic.
I have been in the IT industry for over 20 years and there has not been a day where I have not learned something new…whether it be new skills, ideas or simply learning to do things better! My advice to you…
Keep learning and…keep smiling 🙂 ! Hopefully I’ll see you at my next tutorial!
Prev lesson ⇒ 4. Verification Test Process (Static Testing)