This section will introduce you to the underlying concepts of Software Testing, hopefully, giving you a good overview of testing and explain some of the terminology that you may not have come across before.
Table of contents
- What is Software Testing?
- Why test software?
- Who performs Software Testing?
- So, why have a separate team to do the testing?
- Principles of Software Testing
- The Software Test Process (Fundamental Test Process)
- Summary
- Test Yourself
1. What is Software Testing?
“Software Testing is the process of checking whether the results of an activity matches the expected results, and that the software system, being tested, is free of defects (bugs)”
Simply put, by testing we are looking to see if a software system, that has been requested by a customer or user (the person who is actually asking for a new system or changes to a system), is actually working as they want it to be, without any defects. We will be looking to check:
- The agreed Requirements of the customer have been met
- All the delivered functionality works correctly, being defect / bug free
- The system is usable and its performance (speed, response time, stability, reliability and scalability wise) is to the agreed standard
Through testing, Test Analysts can identify the following:
- Gaps in Requirements
- Missing elements in a project or system
- Errors (defects / bugs)
The benefits of testing include:
- Improving product quality
- Reducing development costs
- Improving performance
Modern testing can be done either manually or by using automation tools. This tutorial will be covering the manual testing side of things. The automation testing will be touched on, but will be covered in more detail in a separate future tutorial.
2. Why test software?
“It is humans who design and develop software systems. As humans ‘we will’ make mistakes, by this very nature software systems are, therefore, most vulnerable to defects and failures, hence, the need for Software Testing”
Today, the world is dominated by computer software systems, whether you are shopping, banking, climbing aboard a plane to fly somewhere or just enquiring about a bill you have, all these processes will almost certainly be controlled by a computer system. It is a fact, in the modern world, that there is no getting away from computers these days! The demand for people to not only develop computer systems, but also to ensure that the systems work correctly (i.e. Software Test Analysts) is greater than it’s ever been!
Why is there a high demand for Test Analysts? Simply put, the correct functioning of software systems is imperative for businesses (utility companies, banks, hospitals, the flight industry, to name a few) to not only survive economically, but also in certain circumstances to ensure the safety of human lives.
Software Testing determines how successful systems are, therefore, determining a business’ successful growth rate! If certain systems fail, then this could potentially mean disaster for companies or even life threatening situations for others. Is Software Testing important…oh yes!
3. Who performs Software Testing?
It is not only the Software Test Analysts who test. Testing (checking correctness) starts as soon as the project starts, involving all team members (technical and non technical), from Project Managers, Business Analysts, Designers (Architects) to Developers.
Once the Requirements have been gathered for a new system by a Business Analyst and the design has been created by an Architect, the documents that hold this information will need to be reviewed. These reviews would most likely be done by several people…Business Users, other Business Analysts, Architects, Managers and most certainly Test Analysts to see if there are any gaps in the Requirements or Design…and so, the testing starts! By reviewing the documents, we are testing to ensure that the documents that are being produced are in fact correct!
The same would go for Developers’ work (the computer code). Code, nowadays, is Unit Tested (tests built within the code) and reviewed by other Developers…again, another form of testing…closing the gap on those little bugs 🙂 !
Typical testing responsibilities at different stages of a software development life cycle:
- Business User – responsible for User Acceptance Testing (UAT), checking that the completed work / system will work how it should work in the real world
- Business Analysts / Architects (System Designers) – reviews and verifies each other’s work and also help the Test Manager to produce the Test Strategy and Plan
- Test Manager – Produces the Test Strategy, the Plan and manages the test process
- Test Analyst – as a Test Analyst, you could be involved in the manual, performance and automation test process, albeit, in larger companies, the automation and performance processes are usually done by dedicated teams. A Manual Test Analyst is ultimately responsible for executing the functional test plans, verifying that the new piece of code / system works as it should from a functional point of view
- Performance Test Analyst – in larger companies, the performance testing is a dedicated process done by the Performance Team. They will ensure the System performs as required (speed, response time, stability, reliability and scalability wise)
- Automation Test Analyst – responsible for the regression testing of systems by running scripts that they will have coded
4. So why have a separate Test Team?
In the early years of computing, the production of code was done by Developers (Computer Programmers), who would test their own work and put it live…scary! I was one of those 🙂 !
Through the back end of the 1990s, with the computer systems becoming larger, more integrated and more complex, the computing world shifted towards having other people, other than programmers, test the code being produced…these people became Test Analysts! It may seem obvious now to have a separate test team, but back then it was the norm for the Developers, System Analysts (Designers) and Business Analysts to perform testing.
So, what are the advantages of a separate test team:
- Separate budget
- A Test Team will have their own budget, not only for the testing of a project, but also for maintaining their ability to provide the highest quality of testing (E.g. for providing training to all their staff so keeping their skills at an optimum). With their own budget for a project, the team will not be coerced into meeting tight deadlines set by others and hence reduce their quality, they will be in control of how they want to test
- Dedicated purely to test
- A Test Team is dedicated to finding errors. Yes, it will be obvious that developers do not want to deliver something that has errors, but I know that most developers do not like testing! I have also experienced, first hand, that they really do miss the most obvious errors…which will not bring the system down, in most cases, but will damage the reputation of an IT department!
- Provides confidence
- Customers who request software will be more confident that they are receiving a product that they asked for, as Test Analysts will be more aligned to ensure the requirements of a system are being met and where Developers will be more concerned with how to build a product
- Providing fresh eyes
- How many times have you heard yourself say “Can you take a look at this please?” when you’re looking for reassurance that something you’ve produced actually works? Well…that’s the same for Developers who have produced software…they too need a fresh pair of (Test Analyst) eyes to ensure that what they’ve done is correct
- Allows early testing to be done
- By being separate, the test team will be brought in right at the very beginning of a software project to work alongside Business Analysts, Designers and Developers. With this…earlier testing can be done to spot errors that would be a lot more expensive to fix down the line
5. Principles of Software Testing
Testing can be a very complex activity, one that can confuse and create problems for Test Analysts who are not following any strategy or set of processes. However…the following important standard General Testing Principles, laid out by the testing world, have been drawn up to help. These principles, together with the testing techniques that I will be showing you in section 5, will give you a good basis to guide and help you so as to avoid mistakes made by many others over the years.
Yes…the following principles are a little dry and “wordy” to understand, but you will see as you progress through the tutorial how they are quite pertinent. They are, however, only concepts and, therefore, you will not need to know them verbatim…phew 🙂 !
- Testing shows the presence of defects
- The main purpose of testing is to find defects. Therefore, tests should be designed to find as many of them as possible
- Finding defects reduces the probability of finding more
- Testing is no guarantee that there are no defects present
- Exhaustive testing is “impossible”
- We cannot test everything. E.g. if we have 10 fields on a screen with 1000 possible values (0 to 999) and wish to test every scenario, then we would require 10 x 1000 tests (10,000 tests)
- With this in mind, we must prioritise our testing effort using a Risk Based Approach
- Early testing
- Testing should start as early as possible in the development life cycle. You should not be waiting for coding to have finished…you can even dry test the design, as we will be explaining later on
- The earlier you start to test your product or software, the better you will find that you are utilising your test time, saving more errors and cost down the line
- The testing activities should be focused on the activities outlined in the test strategy
- Defect clustering
- You will find that defects are found in clusters, they are not evenly distributed. As of Pareto’s law: 80% of errors are uncovered in 20% of a system
- This should be used as an important consideration in prioritising your testing
- Pesticide paradox
- When pesticides are used to eliminate insects from crops, if the same pesticide mix is being used, then the insects will develop a resistance to it. Therefore, new mixes of the pesticide must be introduced for it to be effective
- The statement above is the same for testing. If the same set of tests are being repeatedly used, then the tests will become ineffective at discovering new defects
- When we test, we identify bugs. Developers respond to the bugs (fix them)
- As the bugs are fixed, the software improves
- As software improves, the effectiveness of previous tests becomes less and less, therefore, we must learn to introduce new tests to keep the tests alive
- Testing is context dependent
- Testing strategy is dependant upon the system that is being tested
- When lives are at stake (e.g. using a traffic airline control system or a hospital drug dispensing system), then these will obviously be more vigorously checked than say a system where it is calculating one’s pension outcome. An ecommerce site will be tested differently to a website that is merely used as a browsing site
- Testing of a browsing website could maybe have only 50% coverage, whereas a traffic airline control system could have up to 95%
- Risk, therefore, would obviously determine the testing coverage needed and perhaps the level of testing for security reasons
- Absence of errors is a myth
- Just because a system has no errors does not mean it is fit for purpose and therefore ready to be “put live”
- If the system does not match the customer’s / user’s expectations, then really it is not complete!
6. The Software Test Process (Fundamental Test Process)
Testing is a process. The testing process begins at the start of a project (in our case, a project to deliver a piece of software) and ends with the project / software being delivered. This project can be something from a large project to a small agile sprint (which really is just like a small project…more on this later in the tutorial).
For testing to take place (i.e. tests to be executed), there must be planning and design of the tests, along with the environment being set up for the tests to take place on. The tests must be monitored to see how testing is progressing and reports must be produced to identify if the end goal is being reached. All these activities put together can be known as the Test Process.
This Test Process consists of five activities:
Testing activities are an iterative process
The stages of the Test Process, in the diagram above, are shown as though they will be sequentially performed. However, in real life, this is not the case. Earlier activities will almost certainly be revisited in a large project. Each activity may be executed several times in order to fulfil a project’s exit criteria. Activities may overlap or even be executed in parallel with other activities.
The following diagram shows instances of showing how iterative the process can be:
Let’s take a brief look at each of the Test Process activities:
6.1 Test Planning and Control
Once the requirements gathering has been completed, the requirements analysis done and the requirements documentation produced, then the Test Manager can start planning, so as to determine what is going to be tested and how the testing will be achieved. This will involve:
- Specifying and implementing a test strategy
- Producing a Test Plan, so as to:
- Identify and set the objectives for testing
- Determine:
- the scope of testing
- the test approach
- the required test resources
- Schedule:
- test analysis and design tasks
- test implementation, execution and evaluation
- Determine the Exit Criteria
Controlling is an ongoing activity throughout the project life cycle, comparing actual progress against the Test Plan, involving:
- Measuring and analysing the test results
- Monitoring the progress of the test coverage and the exit criteria and producing reports for all the relevant stakeholders
- Making decisions by taking corrective action to meet the objectives of the project and allowing for any deviations from the plan
6.2 Test Analysis and Design
This activity is the bridge between Test Planning (Strategy and Test Plan) and the actual running of the tests. It uses the Test Plan’s strategy, analyses the project requirements, the architectural design and detailed system design to produce the Test Conditions and hence the Test Cases.
The main tasks being:
- Review the project’s requirements and design documents (Requirements, Architectural Design, Functional and Detailed System Design)
- Design High Level Tests from the Requirements and Design documents, identifying and prioritising Test Conditions and the associated Test Data required
- Document Test Conditions and associated Test Data in a Test Design Specification
- Design and prepare the test environment set-ups
- Identify any required infrastructure and tools
- Design and prioritise Test Cases
6.3 Test Implementation and Execution
This stage of the project involves setting up test environments, preparing the test data needed, development and running of the test cases, recording test results and re-testing fixes that are made.
Main tasks:
- Set up environments to be able to run tests
- Prepare test data
- Prepare test cases and prioritise. Most important ones to be run first
- Run test cases
- Regression test. To ensure changes have not affected other parts of the system
- Record test results – actual against expected
- Raise test incidents where defects or discrepancies are found
- Re-test fixed incidents
6.4 Evaluating Exit Criteria and Reporting
Throughout the project, at the end of each test stage, the exit criteria should be evaluated to determine if the testing of the application is complete and the objectives in the Test Plan have been met. Obviously, once the criteria has been met, the test closure activities can begin.
Main tasks:
- Check whether the exit criteria, set in Test Planning, has been achieved
- If criteria has not been met, assess:
- Does Exit Criteria need to be amended?
- Are more tests needed?
- Do tests need to be re-tested?
- Prepare Test Summary Report for stakeholders
6.5 Test Closure activities
Once testing has finished, it is important that all the test documentation is tidied up and completed, closure reports are written and uncompleted development work and incidents (defects) are either closed or handed over to another Project or Support (Maintenance) Team for them to be completed / fixed in the future.
Real life testing: When most development projects finish (i.e. the project’s exit criteria has been met), this invariably does not mean everything is fixed or development has been completed 100%! The Exit Criteria may mean that the project has reached, for example, 75% test coverage or could even mean that the project has run out of time or budget!
Main Test Closure tasks:
- Built into a development’s Project Plan will be a phase for the project to “handover” to a Support (Maintenance) Team once the new System or changes have gone “live”. This will involve the project handing over to the Support Team:
- All the Project’s documentation (e.g. Requirements, Design, Test documents, Skill Requirements to support the system going forward)
- List of all outstanding development work
- All incidents, including the outstanding ones, which will probably be handled by the Support Team as ongoing support with the customers / users
- Live, development and test environments for future work, incident fixes and support of the live environment itself
- At the end of all projects, there should be a “lessons learned” activity, where perhaps a meeting takes place between all the stakeholders and it is discussed, and agreed, what went well and what could have been done better. The results from this can then be used for future projects
Real life testing: Development Projects hate completing their handover of documentation! Why…because they just want to develop or test things using the shiny new development / test tools or brand new “get paid more to know” skills! They dislike the mundane routine work of supporting a live system! How do I know…because I have been that developer who just wants to develop using the new stuff and I have been on the other side of being in charge of a £multi-million Support Team having work begrudgingly handed over to them 🙂 !
7. Summary
As said in the Introduction Section, Software Testing can at times appear a little complex if you are learning it for the first time! Again, please persevere with your learning…the rewards will be both more than personally satisfying, from an achievement point of view, and very, very financially rewarding once you have your first job in IT Testing!
In this section, you will have learned:
- What Software Testing is and why we do it
- Who actually performs testing throughout a Project’s Life Cycle
- Why we have a separate team to perform testing
- The all important standard Principles of Software Testing
- What the Software Test Process is and how it is performed through a Software Project
With this knowledge, you are now in a good position to move onto looking at what a Software Development Life Cycle is and where testing actually fits into the scheme of things.
8. Test Yourself
Tests to gauge your progress…
At the end of each section of this tutorial, I have produced a link to some Test Yourself quizzes to see how much you have learned.
Having previously come across tests myself where the answers are so ambiguous that you end up doubting what you know and don’t know…I have attempted to make these quizzes, not as to catch you out, but so as to help you reaffirm your understanding and to give you confidence that all you’ve learned makes sense!
Before you move on to the next section…
So, let’s see how much you have learned about the Concepts of Software Testing. In this first test, you will be given 12 questions, with the answers at the end of the test.
Once complete, check your answers, if you do not score that well (let’s say for this section…not getting at least 9 correct), please either go back over the tutorial’s section again, perhaps making more notes as you go, or re-do the test to ensure you have a good understanding.
It is always important that you understand the section that you have read / learned before you move on 🙂 !
Click here to Test Yourself.
And next…
The next 3 sections in this tutorial will be taught around the V Model, my preferred model, where the Test Process is broken down into 2 main areas: Static and Dynamic Testing.
Prev lesson ⇒ 1. Intro to Software Testing
Next lesson ⇒ 3. Understanding the SDLC