Why White box testing when black box testing is there to test conformance to requirements
Logic errors and incorrect assumptions most likely to be made when coding for "special cases". Need to ensure these execution paths are tested.
May find assumptions about execution paths incorrect, and so make design errors. White box testing can find these errors.
Typographical errors are random. Just as likely to be on an obscure logical path as on a mainstream path.
Software Testing Training
Software testing institute
corporate training software testing
Monday, June 29, 2009
Regression Testing
Regression Testing
Testing conducted for the purpose of evaluating whether or not a change to the system (all CM items) has introduced a new failure. Regression testing is
often accomplished through the construction, execution and analysis of product and system tests.
Regression testing is an expensive but necessary activity performed on modified software to provide confidence that changes are correct and do not adversely
affect other system components. Four things can happen when a developer attempts to fix a bug. Three of these things are bad, and one is good:
Because of the high probability that one of the bad outcomes will result from a change to the system, it is necessary to do regression testing.
It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Most industrial testing is done via test
suites; automated sets of procedures designed to exercise all parts of a program and to show defects. While the original suite could be used to test the
modified software, this might be very time-consuming. A regression test selection technique chooses, from an existing test set, the tests that are deemed
necessary to validate modified software.
There are three main groups of test selection approaches in use:
Minimization approaches seek to satisfy structural coverage criteria by identifying a minimal set of tests that must be rerun.
Coverage approaches are also based on coverage criteria, but do not require minimization of the test set. Instead, they seek to select all tests that
exercise changed or affected program components.
Safe attempt instead to select every test that will cause the modified program to produce different output than original program.
An interesting approach to limiting test cases is based on whether we can confine testing to the "vicinity" of the change. (Ex. If I put a new radio in my
car, do I have to do a complete road test to make sure the change was successful?) A new breed of regression test theory tries to identify, through program
flows or reverse engineering, where boundaries can be placed around modules and subsystems. These graphs can determine which tests from the existing suite
may exhibit changed behavior on the new version.
Regression testing has been receiving more attention as corporations focus on fixing the 'Year 2000 Bug'. The goal of most Y2K is to correct the date
handling portions of their system without changing any other behavior. A new 'Y2K' version of the system is compared against a baseline original system. With
the obvious exception of date formats, the performance of the two versions should be identical. This means not only do they do the same things correctly,
they also do the same things incorrectly. A non-Y2K bug in the original software should not have been fixed by the Y2K work.
A frequently asked question about regression testing is 'The developer says this problem is fixed. Why do I need to re-test?’ to which the answer is 'The
same person probably told you it worked in the first place'
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Testing conducted for the purpose of evaluating whether or not a change to the system (all CM items) has introduced a new failure. Regression testing is
often accomplished through the construction, execution and analysis of product and system tests.
Regression testing is an expensive but necessary activity performed on modified software to provide confidence that changes are correct and do not adversely
affect other system components. Four things can happen when a developer attempts to fix a bug. Three of these things are bad, and one is good:
Because of the high probability that one of the bad outcomes will result from a change to the system, it is necessary to do regression testing.
It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Most industrial testing is done via test
suites; automated sets of procedures designed to exercise all parts of a program and to show defects. While the original suite could be used to test the
modified software, this might be very time-consuming. A regression test selection technique chooses, from an existing test set, the tests that are deemed
necessary to validate modified software.
There are three main groups of test selection approaches in use:
Minimization approaches seek to satisfy structural coverage criteria by identifying a minimal set of tests that must be rerun.
Coverage approaches are also based on coverage criteria, but do not require minimization of the test set. Instead, they seek to select all tests that
exercise changed or affected program components.
Safe attempt instead to select every test that will cause the modified program to produce different output than original program.
An interesting approach to limiting test cases is based on whether we can confine testing to the "vicinity" of the change. (Ex. If I put a new radio in my
car, do I have to do a complete road test to make sure the change was successful?) A new breed of regression test theory tries to identify, through program
flows or reverse engineering, where boundaries can be placed around modules and subsystems. These graphs can determine which tests from the existing suite
may exhibit changed behavior on the new version.
Regression testing has been receiving more attention as corporations focus on fixing the 'Year 2000 Bug'. The goal of most Y2K is to correct the date
handling portions of their system without changing any other behavior. A new 'Y2K' version of the system is compared against a baseline original system. With
the obvious exception of date formats, the performance of the two versions should be identical. This means not only do they do the same things correctly,
they also do the same things incorrectly. A non-Y2K bug in the original software should not have been fixed by the Y2K work.
A frequently asked question about regression testing is 'The developer says this problem is fixed. Why do I need to re-test?’ to which the answer is 'The
same person probably told you it worked in the first place'
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Performance Testing - Volume Testing
Performance Testing - Volume Testing
Volume Testing: Testing where the system is subjected to large volumes of data. As regards faults that should be found through volume testing are those,
where the behaviour of the software deviates from that expected for a specified volume of data. Thus a bank system will be tested for faults at much larger
volumes of data, than that of small retailer software. A fault which is only manifested on a table with a million records will be of no concern to the retail
software, but will be picked up by the bank testers.
For example, a window often used window object is populated with data, by calling a database object which runs a complex SQL query. Supposing the component
is tested against tables with only 4-5 records. Of course it will return within seconds. Everything seems fine. It is then integrated with the window, and
system tested. Again everything seems ok. It is only when the application is in User Acceptance (or even gone live) and it is tested against 100,000 records,
is it discovered that, the SQL was not properly optimized and the tables not indexed. Thus it should have been tested at the component level.
Volume testing needs two things. Firstly clear expected outcomes of how the software is to behave for a given level of data. Secondly, data, and lots of it.
The expected behaviour at various levels, should be in the specification documentation. Ideally this will say something like "the customers details will be
returned returned on the screen within 3 seconds, from a database with 1 million customer records." This gives the tester a benchmark to base a test case on.
The second requirement for data, needs either real life data, or simulated data. Usually, real life data will come in the form of a customer database, that
has had private information, such as names and account numbers scrambled. Alternatively records can be created from scratch using automated tools or by
adding rules directly on to the database, with SQL.
As with all testing, proper records must be kept showing the inputs, outputs other information, to aid potential debugging and audit purposes
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Volume Testing: Testing where the system is subjected to large volumes of data. As regards faults that should be found through volume testing are those,
where the behaviour of the software deviates from that expected for a specified volume of data. Thus a bank system will be tested for faults at much larger
volumes of data, than that of small retailer software. A fault which is only manifested on a table with a million records will be of no concern to the retail
software, but will be picked up by the bank testers.
For example, a window often used window object is populated with data, by calling a database object which runs a complex SQL query. Supposing the component
is tested against tables with only 4-5 records. Of course it will return within seconds. Everything seems fine. It is then integrated with the window, and
system tested. Again everything seems ok. It is only when the application is in User Acceptance (or even gone live) and it is tested against 100,000 records,
is it discovered that, the SQL was not properly optimized and the tables not indexed. Thus it should have been tested at the component level.
Volume testing needs two things. Firstly clear expected outcomes of how the software is to behave for a given level of data. Secondly, data, and lots of it.
The expected behaviour at various levels, should be in the specification documentation. Ideally this will say something like "the customers details will be
returned returned on the screen within 3 seconds, from a database with 1 million customer records." This gives the tester a benchmark to base a test case on.
The second requirement for data, needs either real life data, or simulated data. Usually, real life data will come in the form of a customer database, that
has had private information, such as names and account numbers scrambled. Alternatively records can be created from scratch using automated tools or by
adding rules directly on to the database, with SQL.
As with all testing, proper records must be kept showing the inputs, outputs other information, to aid potential debugging and audit purposes
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Performance Testing - Stress testing 2
Performance Testing - Stress testing 2
Stress testing :is subjecting a system to an unreasonable load while denying it the resources (e.g., RAM, disc, mips, interrupts, etc.) needed to process
that load. The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful. The system is not
expected to process the overload without adequate resources, but to behave (e.g., fail) in a decent manner (e.g., not corrupting or losing data). Bugs and
failure modes discovered under stress testing may or may not be repaired depending on the application, the failure mode, consequences, etc. The load
(incoming transaction stream) in stress testing is often deliberately distorted so as to force the system into resource depletion.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Stress testing :is subjecting a system to an unreasonable load while denying it the resources (e.g., RAM, disc, mips, interrupts, etc.) needed to process
that load. The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful. The system is not
expected to process the overload without adequate resources, but to behave (e.g., fail) in a decent manner (e.g., not corrupting or losing data). Bugs and
failure modes discovered under stress testing may or may not be repaired depending on the application, the failure mode, consequences, etc. The load
(incoming transaction stream) in stress testing is often deliberately distorted so as to force the system into resource depletion.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Performance Testing - Load testing
Performance Testing - Load testing
Load testing: is subjecting a system to a statistically representative (usually) load. The two main reasons for using such loads is in support of software
reliability testing and in performance testing. The term "load testing" by itself is too vague and imprecise to warrant use. For example, do you mean
representative load," "overload," "high load," etc. In performance testing, load is varied from a minimum (zero) to the maximum level the systemcan sustain
without running out of resources or having, transactions suffer (application-specific) excessive delay.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Load testing: is subjecting a system to a statistically representative (usually) load. The two main reasons for using such loads is in support of software
reliability testing and in performance testing. The term "load testing" by itself is too vague and imprecise to warrant use. For example, do you mean
representative load," "overload," "high load," etc. In performance testing, load is varied from a minimum (zero) to the maximum level the systemcan sustain
without running out of resources or having, transactions suffer (application-specific) excessive delay.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Integration Testing
Integration Testing
One of the most difficult aspects of software development is the integration and testing of large, untested sub-systems. The integrated system frequently
fails in significant and mysterious ways, and it is difficult to fix it
Integration testing exercises several units that have been combined to form a module, subsystem, or system. Integration testing focuses on the interfaces
between units, to make sure the units work together. The nature of this phase is certainly 'white box', as we must have a certain knowledge of the units to
recognize if we have been successful in fusing them together in the module.
There are three main approaches to integration testing: top-down, bottom-up and 'big bang'. Top-down combines, tests, and debugs top-level routines that
become the test 'harness' or 'scaffolding' for lower-level units. Bottom-up combines and tests low-level units into progressively larger modules and
subsystems. 'Big bang' testing is, unfortunately, the prevalent integration test 'method'. This is waiting for all the module units to be complete before
trying them out together.
Integration tests can rely heavily on stubs or drivers. Stubs stand-in for finished subroutines or sub-systems. A stub might consist of a function header
with no body, or it may read and return test data from a file, return hard-coded values, or obtain data from the tester. Stub creation can be a time
consuming piece of testing.
The cost of drivers and stubs in the top-down and bottom-up testing methods is what drives the use of 'big bang' testing. This approach waits for all the
modules to be constructed and tested independently, and when they are finished, they are integrated all at once. While this approach is very quick, it
frequently reveals more defects than the other methods. These errors have to be fixed and as we have seen, errors that are found 'later' take longer to fix.
In addition, like bottom up, there is really nothing that can be demonstrated until later in the process.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
One of the most difficult aspects of software development is the integration and testing of large, untested sub-systems. The integrated system frequently
fails in significant and mysterious ways, and it is difficult to fix it
Integration testing exercises several units that have been combined to form a module, subsystem, or system. Integration testing focuses on the interfaces
between units, to make sure the units work together. The nature of this phase is certainly 'white box', as we must have a certain knowledge of the units to
recognize if we have been successful in fusing them together in the module.
There are three main approaches to integration testing: top-down, bottom-up and 'big bang'. Top-down combines, tests, and debugs top-level routines that
become the test 'harness' or 'scaffolding' for lower-level units. Bottom-up combines and tests low-level units into progressively larger modules and
subsystems. 'Big bang' testing is, unfortunately, the prevalent integration test 'method'. This is waiting for all the module units to be complete before
trying them out together.
Integration tests can rely heavily on stubs or drivers. Stubs stand-in for finished subroutines or sub-systems. A stub might consist of a function header
with no body, or it may read and return test data from a file, return hard-coded values, or obtain data from the tester. Stub creation can be a time
consuming piece of testing.
The cost of drivers and stubs in the top-down and bottom-up testing methods is what drives the use of 'big bang' testing. This approach waits for all the
modules to be constructed and tested independently, and when they are finished, they are integrated all at once. While this approach is very quick, it
frequently reveals more defects than the other methods. These errors have to be fixed and as we have seen, errors that are found 'later' take longer to fix.
In addition, like bottom up, there is really nothing that can be demonstrated until later in the process.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Functional testing
Functional testing
Functional testing: Application of test data derived from the specified functional requirements without regard to the final program structure. Also known as
black-box testing.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Functional testing: Application of test data derived from the specified functional requirements without regard to the final program structure. Also known as
black-box testing.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Exploratory Testing
Exploratory Testing
An interactive process of concurrent product exploration, test
design, and test execution. The heart of exploratory testing can be
stated simply: The outcome of this test influences the design of the
next test.
The plainest definition of exploratory testing is test design and
test execution at the same time. This is the opposite of scripted
testing (predefined test procedures, whether manual or automated).
Exploratory tests, unlike scripted tests, are not defined in advance
and carried out precisely according to plan. This may sound like a
straightforward distinction, but in practice it's murky. That's
because "defined" is a spectrum. Even an otherwise elaborately
defined test procedure will leave many interesting details (such as
how quickly to type on the keyboard, or what kinds of behavior to
recognize as a failure) to the discretion of the tester. Likewise,
even a free-form exploratory test session will involve tacit
constraints or mandates about what parts of the product to test, or
what strategies to use. A good exploratory tester will write down
test ideas and use them in later test cycles. Such notes sometimes
look a lot like test scripts, even if they aren't. Exploratory
testing is sometimes confused with "ad hoc" testing. Ad hoc testing
normally refers to a process of improvised, impromptu bug searching.
By definition, anyone can do ad hoc testing
What kinds of specifics affect ET? Here are some of them:
1. the mission of the test project
2. the mission of this particular test session
3. the role of the tester
4. the tester (skills, talents, and preferences)
5. available tools and facilities
6. available time
7. available test data and materials
8. available help from other people
9. accountability requirements
10. what the tester‘s clients care about
11. the current testing strategy
12. the status of other testing efforts on the same product
13. the product, itself- its user interface - its behavior - its
present state of execution - its defects- its testability- its
purpose
14. what the tester knows about the product- what just happened in
the previous test - known problems with it- past problems with it -
strengths and weaknesses - risk areas and magnitude of perceived risk
- recent changes to it - direct observations of it- rumors about it -
the nature of its users and user behavior - how it‘s supposed to work
- how it‘s put together - how it‘s similar to or different from other
products
15. what the tester would like to know about the product
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about
software testing.
http://www.crestech.in
href="http://www.crestechsoftware.com.au">http://www.crestechsoftware
.com.au
An interactive process of concurrent product exploration, test
design, and test execution. The heart of exploratory testing can be
stated simply: The outcome of this test influences the design of the
next test.
The plainest definition of exploratory testing is test design and
test execution at the same time. This is the opposite of scripted
testing (predefined test procedures, whether manual or automated).
Exploratory tests, unlike scripted tests, are not defined in advance
and carried out precisely according to plan. This may sound like a
straightforward distinction, but in practice it's murky. That's
because "defined" is a spectrum. Even an otherwise elaborately
defined test procedure will leave many interesting details (such as
how quickly to type on the keyboard, or what kinds of behavior to
recognize as a failure) to the discretion of the tester. Likewise,
even a free-form exploratory test session will involve tacit
constraints or mandates about what parts of the product to test, or
what strategies to use. A good exploratory tester will write down
test ideas and use them in later test cycles. Such notes sometimes
look a lot like test scripts, even if they aren't. Exploratory
testing is sometimes confused with "ad hoc" testing. Ad hoc testing
normally refers to a process of improvised, impromptu bug searching.
By definition, anyone can do ad hoc testing
What kinds of specifics affect ET? Here are some of them:
1. the mission of the test project
2. the mission of this particular test session
3. the role of the tester
4. the tester (skills, talents, and preferences)
5. available tools and facilities
6. available time
7. available test data and materials
8. available help from other people
9. accountability requirements
10. what the tester‘s clients care about
11. the current testing strategy
12. the status of other testing efforts on the same product
13. the product, itself- its user interface - its behavior - its
present state of execution - its defects- its testability- its
purpose
14. what the tester knows about the product- what just happened in
the previous test - known problems with it- past problems with it -
strengths and weaknesses - risk areas and magnitude of perceived risk
- recent changes to it - direct observations of it- rumors about it -
the nature of its users and user behavior - how it‘s supposed to work
- how it‘s put together - how it‘s similar to or different from other
products
15. what the tester would like to know about the product
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about
software testing.
http://www.crestech.in
href="http://www.crestechsoftware.com.au">http://www.crestechsoftware
.com.au
Boundary Value Analysis
Large number of errors tend to occur at boundaries of the input domain.
BVA leads to selection of test cases that exercise boundary values.
BVA complements equivalence partitioning. Rather than select any element in an equivalence class, select those at the ''edge' of the class.
Examples:
For a range of values bounded by a and b, test (a-1), a, (a+1), (b-1), b, (b+1).
If input conditions specify a number of values n, test with (n-1), n and (n+1) input values.
Apply 1 and 2 to output conditions (e.g., generate table of minimum and maximum size).
If internal program data structures have boundaries (e.g., buffer size, table limits), use input data to exercise structures on boundaries.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
BVA leads to selection of test cases that exercise boundary values.
BVA complements equivalence partitioning. Rather than select any element in an equivalence class, select those at the ''edge' of the class.
Examples:
For a range of values bounded by a and b, test (a-1), a, (a+1), (b-1), b, (b+1).
If input conditions specify a number of values n, test with (n-1), n and (n+1) input values.
Apply 1 and 2 to output conditions (e.g., generate table of minimum and maximum size).
If internal program data structures have boundaries (e.g., buffer size, table limits), use input data to exercise structures on boundaries.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
About Software Testing
About Software Testing
Process of evaluation of Software application: To verify it satisfy
the specified requirements With intent to find errors To Check if the
system ?Fit for Purpose? Simulation of real time scenarios.
Testing is not a one-time activity—applications need to be tested
throughout their lifecycle. Every version upgrade, module addition,
or enhancement, as well as every implementation at a new site or
increase in user load, needs to be put through comprehensive testing.
The Program Doesn‘t Work Nobody would pay you to test if their
program didn‘t have bugs. All programs have bugs. Any change to a
program can cause new bugs, and any aspect of the program can be
broken. You DON‘T ?verify that the program is working.? You FIND
bugs. “If you set your mind to show that a program works correctly,
you’ll be more likely to miss problems than if you want and expect
the program to fail.” Complete Testing is Impossible:
1. There are a nearly infinite number of paths through any
non-trivial program.
2. There are a virtually infinite set of combinations of data that
you can feed the program.
You can’t test them all.
1. Therefore, your task is to find bugs --not to find all the bugs.
2. You want to find as many bugs as possible
3. Find the most serious bugs
4. Find bugs as early as possible
5. Your challenges will require judgment, trade-offs, and efficiency.
“Q: How many QA testers does it take to change a lightbulb?
A: QA testers don't change anything. They just report that it's
dark.” Kerry Zallar?
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about
software testing.
http://www.crestech.in
href="http://www.crestechsoftware.com.au">http://www.crestechsoftware.com.au
Process of evaluation of Software application: To verify it satisfy
the specified requirements With intent to find errors To Check if the
system ?Fit for Purpose? Simulation of real time scenarios.
Testing is not a one-time activity—applications need to be tested
throughout their lifecycle. Every version upgrade, module addition,
or enhancement, as well as every implementation at a new site or
increase in user load, needs to be put through comprehensive testing.
The Program Doesn‘t Work Nobody would pay you to test if their
program didn‘t have bugs. All programs have bugs. Any change to a
program can cause new bugs, and any aspect of the program can be
broken. You DON‘T ?verify that the program is working.? You FIND
bugs. “If you set your mind to show that a program works correctly,
you’ll be more likely to miss problems than if you want and expect
the program to fail.” Complete Testing is Impossible:
1. There are a nearly infinite number of paths through any
non-trivial program.
2. There are a virtually infinite set of combinations of data that
you can feed the program.
You can’t test them all.
1. Therefore, your task is to find bugs --not to find all the bugs.
2. You want to find as many bugs as possible
3. Find the most serious bugs
4. Find bugs as early as possible
5. Your challenges will require judgment, trade-offs, and efficiency.
“Q: How many QA testers does it take to change a lightbulb?
A: QA testers don't change anything. They just report that it's
dark.” Kerry Zallar?
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about
software testing.
http://www.crestech.in
href="http://www.crestechsoftware.com.au">http://www.crestechsoftware.com.au
Subscribe to:
Posts (Atom)