Software Quality Characteristics

 

http://thetesteye.com/posters/TheTestEye_SoftwareQualityCharacteristics.pdf

 

Go through the list and think about your product/features. Add specifics for your context, and transform the list to your own.

 

Capability. Can the product perform valuable functions?

 

• Completeness: all important functions wanted by end users are available.
• Accuracy: any output or calculation in the product is correct and presented with significant digits.
• Efficiency: performs its actions in an efficient manner (without doing what it’s not supposed to do.)
• Interoperability: different features interact with each other in the best way.
• Concurrency: ability to perform multiple parallel tasks, and run at the same time as other processes.
• Data agnosticism: supports all possible data formats, and handles noise.
• Extensibility: ability for customers or 3rd parties to add features or change behavior.

 

Reliability. Can you trust the product in many and difficult situations?

 

• Stability: the product shouldn’t cause crashes, unhandled exceptions or script errors.
• Robustness: the product handles foreseen and unforeseen errors gracefully.
• Stress handling: how does the system cope when exceeding various limits?
• Recoverability: it is possible to recover and continue using the product after a fatal error.
• Data Integrity: all types of data remain intact throughout the product.
• Safety: the product will not be part of damaging people or possessions.
• Disaster Recovery: what if something really, really bad happens?
• Trustworthiness: is the product’s behavior consistent, predictable, and trustworthy? 

 

Usability. Is the product easy to use?

 

• Affordance: product invites to discover possibilities of the product.
• Intuitiveness: it is easy to understand and explain what the product can do.
• Minimalism: there is nothing redundant about the product’s content or appearance.
• Learnability: it is fast and easy to learn how to use the product.
• Memorability: once you have learnt how to do something you don’t forget it.
• Discoverability: the product’s information and capabilities can be discovered by exploration of the user interface.
• Operability: an experienced user can perform common actions very fast.
• Interactivity: the product has easy-to-understand states and possibilities of interacting with the application (via GUI or API).
• Control: the user should feel in control over the proceedings of the software.
• Clarity: is everything stated explicitly and in detail, with a language that can be understood, leaving no room for doubt?
• Errors: there are informative error messages, difficult to make mistakes and easy to repair after making them.
• Consistency: behavior is the same throughout the product, and there is one look & feel.
• Tailorability: default settings and behavior can be specified for flexibility.
• Accessibility: the product is possible to use for as many people as possible, and

 

Charisma. Does the product have “it”?

 

• Uniqueness: the product is distinguishable and has something no one else has.
• Satisfaction: how do you feel after using the product?
• Professionalism: does the product have the appropriate flair of professionalism and feel fit for purpose?
• Attractiveness: are all types of aspects of the product appealing to eyes and other senses?
• Curiosity: will users get interested and try out what they can do with the product?
• Entrancement: do users get hooked, have fun, in a flow, and fully engaged when using the product?
• Hype: should the product use the latest and greatest technologies/ideas?
• Expectancy: the product exceeds expectations and meets the needs you didn't know you had.
• Attitude: do the product and its information have the right attitude and speak to you with the right language and style?
• Directness: are (first) impressions impressive?
• Story: are there compelling stories about the product’s inception, construction or usage?

 

Security. Does the product protect against unwanted usage?

 

• Authentication: the product’s identifications of the users.
• Authorization: the product’s handling of what an authenticated user can see and do.
• Privacy: ability to not disclose data that is protected to unauthorized users.
• Security holes: product should not invite to social engineering vulnerabilities.
• Secrecy: the product should under no circumstances disclose information about the underlying systems.
• Invulnerability: ability to withstand penetration attempts.
• Virus-free: product will not transport virus, or appear as one.
• Piracy Resistance: no possibility to illegally copy and distribute the software or code.
• Compliance: security standards the product adheres to.

 

Performance. Is the product fast enough?

 

• Capacity: the many limits of the product, for different circumstances (e.g. slow network.)
• Resource Utilization: appropriate usage of memory, storage and other resources.
• Responsiveness: the speed of which an action is (perceived as) performed.
• Availability: the system is available for use when it should be.
• Throughput: the products ability to process many, many things.
• Endurance: can the product handle load for a long time?
• Feedback: is the feedback from the system on user actions appropriate?
• Scalability: how well does the product scale up, out or down? 

 

IT-bility. Is the product easy to install, maintain and support?

 

• System requirements: ability to run on supported configurations, and handle different environments or missing components.
• Installability: product can be installed on intended platforms with appropriate footprint.
• Upgrades: ease of upgrading to a newer version without loss of configuration and settings.
• Uninstallation: are all files (except user’s or system files) and other resources removed when uninstalling?
• Configuration: can the installation be configured in various ways or places to support customer’s usage?
• Deployability: product can be rolled-out by IT department to different types of (restricted) users and environments.
• Maintainability: are the product and its artifacts easy to maintain and support for customers?
• Testability: how effectively can the deployed product be tested by the customer?

 

Compatibility. How well does the product interact with software and environments?

 

• Hardware Compatibility: the product can be used with applicable configurations of hardware components.
• Operating System Compatibility: the product can run on intended operating system versions, and follows typical behavior.
• Application Compatibility: the product, and its data, works with other applications customers are likely to use.
• Configuration Compatibility: product’s ability to blend in with configurations of the environment.
• Backward Compatibility: can the product do everything the last version could?
• Forward Compatibility: will the product be able to use artifacts or interfaces of future versions?
• Sustainability: effects on the environment, e.g. energy efficiency, switch-offs, power-saving modes, telecommuting.
• Standards Conformance: the product conforms to applicable standards, regulations, laws or ethics.

 

Internal Software Quality Characteristics

 

These characteristics are not directly experienced by end users, but can be equally important for successful products.

 

Supportability. Can customers’ usage and problems be supported?

 

• Identifiers: is it easy to identify parts of the product and their versions, or specific errors?
• Diagnostics: is it possible to find out details regarding customer situations?
• Troubleshootable: is it easy to pinpoint errors (e.g. log files) and get help?
• Debugging: can you observe the internal states of the software when needed?
• Versatility: ability to use the product in more ways than it was originally designed for.

 

Testability. Is it easy to check and test the product?

 

• Traceability: the product logs actions at appropriate levels and in usable format.
• Controllability: ability to independently set states, objects or variables.
• Observability: ability to observe things that should be tested.
• Monitorability: can the product give hints on what/how it is doing?
• Isolateability: ability to test a part by itself.
• Stability: changes to the software are controlled, and not too frequent.
• Automation: are there public or hidden programmatic interface that can be used?
• Information: ability for testers to learn what needs to be learned...
• Auditability: can the product and its creation be validated?

 

Maintainability. Can the product be maintained and extended at low cost?

 

• Flexibility: the ability to change the product as required by customers.
• Extensibility: will it be easy to add features in the future?
• Simplicity: the code is not more complex than needed, and does not obscure test design, execution and evaluation.
• Readability: the code is adequately documented and easy to read and understand.
• Transparency: Is it easy to understand the underlying structures?
• Modularity: the code is split into manageable pieces.
• Refactorability: are you satisfied with the unit tests?
• Analyzability: ability to find causes for defects or other code of interest.

 

Portability. Is transferring of the product to different environments enabled?

 

• Reusability: can parts of the product be re-used elsewhere?
• Adaptability: is it easy to change the product to support a different environment?
• Compatibility: does the product comply with common interfaces or official standards?
• Internationalization: it is easy to translate the product.
• Localization: are all parts of the product adjusted to meet the needs of the targeted culture/country?
• User Interface-robustness: will the product look equally good when translated?

 

Rikard Edgren, Henrik Emilsson and Martin Jansson - thetesteye.com v1.1
This work is licensed under the Creative Commons Attribution-No Derivative License
inspired by James Bach’s CRUSSPIC STMPL, ISO 9126-1, Wikipedia:Ilities and more…

 

_______________________________________________________________________________________________________

 

Test Heuristics Cheat Sheet

 

http://testobsessed.com/wp-content/uploads/2007/02/testheuristicscheatsheetv1.pdf

 

Data Type Attacks

 

Paths/Files

• Long Name(>255 chars)
• Special Characters in Name(space * ? / \ | < > , . ( ) [ ] { } ; : ‘ “ !@ # $ % ^ &)
• Non-Existent
• Already Exists
• No Space
• Minimal Space
• Write-Protected
• Unavailable
• Locked
• On Remote Machine
• Corrupted

 

Time and Date

• Timeouts
• Time Difference between Machines
• Crossing Time Zones
• Leap Days
• Always Invalid Days (Feb 30, Sept 31)
• Feb 29 in Non-Leap Years
• Different Formats(June 5, 2001; 06/05/2001; 06/05/01; 06-05-01; 6/5/2001 12:34, US vs Europe 5/6/2001)
• ISO week/year (53 week years)
• Daylight Savings Changeover
• Date fields saved as date/time format (time saved as "00:00:00" causes date change on Daylight Savings or Time Zone shift)
• Reset Clock Backward or Forward

 

Numbers

• 0
• 32768 (2^15)
• 32769 (2^15 + 1)
• 65536 (2^16)
• 65537 (2^16 +1)
• 2147483648 (2^31)
• 2147483649 (2^31 + 1)
• 4294967296 (2^32)
• 4294967297 (2^32 + 1)
• Scientific Notation (1E-16)
• Negative
• Floating Point/Decimal (0.0001)
• With Commas (1,234,567)
• European Style (1.234.567,89)
• All the Above in Calculations

 

Strings

 

• Long (255, 256, 257, 1000, 1024, 2000, 2048 or more characters)
• Accented Chars(àáâãäåçèéêëìíîðñòôõöö, etc.)
• Asian Chars ( �� )
• Common Delimiters and Special Characters ( “ ‘ ` | / \ , ; : & < > ^ ? Tab )
• Leave Blank
• Single Space
• Multiple Spaces
• Leading Spaces
• End-of-Line Characters (^M)
• SQL Injection ( ‘select from customer )
• With All Actions (Entering, Searching, Updating, etc.)
• Right to Left Languages (Hebrew, Arabic)

 

General

 

• Violates Domain-Specific Rules (an ip address of 999.999.999.999, an email address with no “@”, an age of -1)
• Violates Uniqueness Constraint

 

Web Tests

 

Navigation

 

• Back (watch for ‘Expired’ messages and double-posted transactions)
• Refresh
• Bookmark the URL
• Select Bookmark when Logged Out
• Hack the URL (change/remove parameters; see also Data Type Attacks)
• Multiple Browser Instances Open

 

Input

See also Data Type Attacks

 

• HTML/JavaScript Injection (allowing the user to enter arbitrary HTML tags and JavaScript commands can lead to security vulnerabilities)
• Check Max Length Defined on Text Inputs
• > 5000 Chars in TextAreas

 

Syntax

 

• HTML Syntax Checker (http://validator.w3.org/)
• CSS Syntax Checker (http://jigsaw.w3.org/css-validator/)

 

Preferences

 

• Javascript Off
• Cookies Off
• Security High
• Resize Preferences Browser Window
• Change Font Size

 

Testing Wisdom

 

• A test is an experiment designed to reveal information or answer a specific question about the software or system.
• Stakeholders have questions; testers have answers.
• Don’t confuse speed with progress.
• Take a contrary approach.
• Observation is exploratory.
• The narrower the view, the wider the ignorance.
• Big bugs are often found by coincidence.
• Bugs cluster.
• Vary sequences, configurations, and data to increase the probability that, if there is a problem, testing will find it.
• It’s all about the variables.

 

Heuristics

 

Variable Analysis - Identify anything whose value can change. Variables can be obvious, subtle, or hidden.

Touch Points - Identify any public or private interface that provides visibility or control. Provides places to provoke, monitor, and verify the system.

Boundaries - Approaching the Boundary (almost too big, almost too small), At the Boundary

Goldilocks - Too Big, Too Small, Just Right

CRUD - Create, Read, Update, Delete

Follow the Data - Perform a sequence of actions involving data, verifying the data integrity at each step. (Example: Enter → Search → Report → Export → Import → Update → View)

Configurations - Varying the variables related to configuration (Screen Resolution; Network Speed, Latency, Signal Strength; Memory; Disk Availability; Count heuristic applied to any peripheral such as 0, 1, Many Monitors, Mice, or Printers)

Interruptions - Log Off, Shut Down, Reboot, Kill Process, Disconnect, Hibernate, Timeout, Cancel

Starvation - CPU, Memory, Network, or Disk at maximum capacity

Position - Beginning, Middle, End (Edit at the beginning of the line, middle of the line, end of the line)

Selection - Some, None, All (Some permissions, No permissions, All permissions)

Count - 0, 1, Many (0 transactions, 1 transactions, Many simultaneous transactions)

Multi-User - Simultaneous create, update, delete from two accounts or same account logged in twice.

Flood - Multiple simultaneous transactions or requests flooding the queue.

Dependencies - Identify “has a” relationships (a Customer has an Invoice; an Invoice has multiple Line Items). Apply CRUD, Count, Position, and/or Selection heuristics (Customer has 0, 1, many Invoices; Invoice has 0, 1, many Line Items; Delete last Line Item then Read; Update first Line Item; Some, None, All Line Items are taxable; Delete Customer with 0, 1, Many Invoices)

Constraints - Violate constraints (leave required fields blank, enter invalid combinations in dependent fields, enter duplicate IDs or names). Apply with the Input Method heuristic.

Input Method - Typing, Copy/Paste, Import, Drag/Drop, Various Interfaces (GUI v. API)

Sequences - Vary Order of Operations, Undo/Redo, Reverse, Combine, Invert, Simultaneous

Sorting - Alpha v. Numeric, Across Multiple Pages

State Analysis - Identify states and events/transitions, then represent them in a picture or table. Works with the Sequences and Interruption heuristics.

Map Making - Identify a “base” or “home” state. Pick a direction and take one step. Return to base. Repeat.

Users & Scenarios - Use Cases, Soap Operas, Personae, Extreme Personalities

 

Frameworks

 

Judgment Inconsistencies

 

• Absences, and Extras with respect to Internal
• External – Specific, or External –
• Cultural reference points. (James Lyndsay, Workroom Productions)

 

Observations

 

• Input
• Output
• Linkage (James Lyndsay, Workroom Productions)

 

Flow

 

• Input
• Processing
• Output

 

Requirements

 

• Users
• Function
• Attributes
• Constraints (Gause & Weinberg Exploring Requirements)

 

Nouns & Verbs

 

• The objects or data in the system and the ways in which the system manipulates it.
• Adjectives (attributes) such as Visible, Identical,
• Verbose and Adverbs (action descriptors) such as Quickly, Slowly, Repeatedly, Precisely, Randomly. Good for creating random scenarios.

 

Deming’s

 

• Cycle
• Plan
• Do
• Check
• Act

 

'This cheat sheet includes ideas from Elisabeth Hendrickson, James Lyndsay, and Dale Emery'

'www.qualitytree.com Copyright © 2006 Quality Tree Software, Inc.'

 

_______________________________________________________________________________________________________

 

Heuristic test strategy model

 

http://www.satisfice.com/tools/satisfice-tsm-4p.pdf

 

General Test Techniques

 

A test technique is a way of creating tests. There are many interesting techniques. The list includes nine general techniques.

By “general technique” I mean that the technique is simple and universal enough to apply to a wide variety of contexts.

Many specific techniques are based on one or more of these nine. And an endless variety of specific test techniques can be

constructed by combining one or more general techniques with coverage ideas from the other lists in the Heuristic Test

Strategy Model.

 

Function Testing

Test what it can do

 

1. Identify things that the product can do (functions and subfunctions).
2. Determine how you’d know if a function was capable of working.
3. Test each function, one at a time.
4. See that each function does what it’s supposed to do and not whatit isn’t supposed to do.

 

Domain Testing

Divide and conquer the data

 

1. Look for any data processed by the product. Look at outputs as well as inputs.
2. Decide which particular data to test with. Consider things like boundary values, typical values, convenient values, invalid
3. values, or best representatives.
4. Consider combinations of data worth testing together.

 

Stress Testing

Overwhelm the product

 

1. Look for sub-systems and functions that are vulnerable to being overloaded or “broken” in the presence of challenging data or constrained resources.
2. Identify data and resources related to those sub-systems and functions.
3. Select or generate challenging data, or resource constraint conditions to test with: e.g., large or complex data structures,
4. high loads, long test runs, many test cases, low memory conditions.

 

Flow Testing

Do one thing after another

 

1. Define test procedures or high level cases that incorporate multiple activities connected end-to-end.
2. Don’t reset the system between tests.
3. Vary timing and sequencing, and try parallel threads.

 

Scenario Testing

Test to a compelling story

 

1. Begin by thinking about everything going on around the product.
2. Design tests that involve meaningful and complex interactions with the product.
3. A good scenario test is a compelling story of how someone who matters might do something that matters with the product.

 

Claims Testing

Verify every claim

 

1. Identify reference materials that include claims about the product(implicit or explicit).
2. Analyze individual claims, and clarify vague claims.
3. Verify that each claim about the product is true.
4. If you’re testing from an explicit specification, expect it and the product to be brought into alignment.

 

User Testing

Involve the users

 

1. Identify categories and roles of users.
2. Determine what each category of user will do (use cases), how they will do it, and what they value.
3. Get real user data, or bring real users in to test.
4. Otherwise, systematically simulate a user (be careful—it’s easy to think you’re like a user even when you’re not).
5. Powerful user testing is that which involves a variety of users and user roles, not just one.

 

Risk Testing

Imagine a problem, then look for it.

 

1. What kinds of problems could the product have?
2. Which kinds matter most? Focus on those.
3. How would you detect them if they were there?
4. Make a list of interesting problems and design tests specifically to reveal them.
5. It may help to consult experts, design documentation, past bug reports, or apply risk heuristics.

 

Automatic Testing

Run a million different tests

 

1. Look for opportunities to automatically generate a lot of tests.
2. Develop an automated, high speed evaluation mechanism.
3. Write a program to generate, execute, and evaluate the tests

 

Scenario Testing

Test to a compelling story

 

1. Begin by thinking about everything going on around the product.
2. Design tests that involve meaningful and complex interactions with the product.
3. A good scenario test is a compelling story of how someone who matters might do something that matters with the product.

 

Project Environment

 

''Creating and executing tests is the heart of the test project. However, there are many factors in the project environment that are critical to your decision about what particular tests to create. In each category, below, consider how that factor may help or hinder your test design process. Try to exploit every resource.''

 

Customers. Anyone who is a client of the test project.

 

• Do you know who your customers are? Whose opinions matter? Who benefits or suffers from the work you do?
• Do you have contact and communication with your customers? Maybe they can help you test.
• Maybe your customers have strong ideas about what tests you should create and run.
• Maybe they have conflicting expectations. You may have to help identify and resolve those.

 

Information. Information about the product or project that is needed for testing.

 

• Are there any engineering documents available? User manuals? Web-based materials?
• Does this product have a history? Old problems that were fixed or deferred? Pattern of customer complaints?
• Do you need to familiarize yourself with the product more, before you will know how to test it?
• Is your information current? How are you apprised of new or changing information?
• Is there any complex or challenging part of the product about which there seems strangely little information?

 

Developer Relations. How you get along with the programmers.

 

• Hubris: Does the development team seem overconfident about any aspect of the product?
• Defensiveness: Is there any part of the product the developers seem strangely opposed to having tested?
• Rapport: Have you developed a friendly working relationship with the programmers?
• Feedback loop: Can you communicate quickly, on demand, with the programmers?
• Feedback: What do the developers think of your test strategy?

 

Test Team. Anyone who will perform or support testing.

 

• Do you know who will be testing?
• Are there people not on the “test team” that might be able to help? People who’ve tested similar products before and might have advice? Writers? Users? Programmers?
• Do you have enough people with the right skills to fulfill a reasonable test strategy?
• Are there particular test techniques that the team has special skill or motivation to perform?
• Is any training needed? Is any available?

 

Equipment & Tools. Hardware, software, or documents required to administer testing.

 

• Hardware: Do we have all the equipment you need to execute the tests? Is it set up and ready to go?
• Automation: Are any test automation tools needed? Are they available?
• Probes: Are any tools needed to aid in the observation of the product under test?
• Matrices & Checklists: Are any documents needed to track or record the progress of testing?

 

Schedule. The sequence, duration, and synchronization of project events.

 

• Test Design: How much time do you have? Are there tests better to create later than sooner?
• Test Execution: When will tests be executed? Are some tests executed repeatedly, say, for regression purposes?
• Development: When will builds be available for testing, features added, code frozen, etc.?
• Documentation: When will the user documentation be available for review?

 

Test Items. The product to be tested.

 

• Scope: What parts of the product are and are not within the scope of your testing responsibility?
• Availability: Do you have the product to test?
• Volatility: Is the product constantly changing? What will be the need for retesting?
• New Stuff: What has recently been changed or added in the product?
• Testability: Is the product functional and reliable enough that you can effectively test it?
• Future Releases: What part of your tests, if any, must be designed to apply to future releases of the product?

 

Deliverables. The observable products of the test project.

 

• Content: What sort of reports will you have to make? Will you share your working notes, or just the end results?
• Purpose: Are your deliverables provided as part of the product? Does anyone else have to run your tests?
• Standards: Is there a particular test documentation standard you’re supposed to follow?
• Media: How will you record and communicate your reports?

 

Product Elements

 

''Ultimately a product is an experience or solution provided to a customer. Products have many dimensions. So, to test well, we must examine those dimensions. Each category, listed below, represents an important and unique aspect of a product. Testers who focus on only a few of these are likely to miss important bugs.''

 

Structure. Everything that comprises the physical product.

 

• Code: the code structures that comprise the product, from executables to individual routines.
• Interfaces: points of connection and communication between sub-systems.
• Hardware: any hardware component that is integral to the product.
• Non-executable files: any files other than multimedia or programs, like text files, sample data, or help files.
• Collateral: anything beyond software and hardware that is also part of the product, such as paper documents, web links and content, packaging, license agreements, etc..

 

Functions. Everything that the product does.

 

• User Interface: any functions that mediate the exchange of data with the user (e.g. navigation, display, data entry).
• System Interface: any functions that exchange data with something other than the user, such as with other programs, hard disk,network, printer, etc.
• Application: any function that defines or distinguishes the product or fulfills core requirements.
• Calculation: any arithmetic function or arithmetic operations embedded in other functions.
• Time-related: time-out settings; daily or month-end reports; nightly batch jobs; time zones; business holidays; interest
• calculations; terms and warranty periods; chronograph functions.
• Transformations: functions that modify or transform something (e.g. setting fonts, inserting clip art, withdrawing money from account).
• Startup/Shutdown: each method and interface for invocation and initialization as well as exiting the product.
• Multimedia: sounds, bitmaps, videos, or any graphical display embedded in the product.
• Error Handling: any functions that detect and recover from errors, including all error messages.
• Interactions: any interactions or interfaces between functions within the product.
• Testability: any functions provided to help test the product, such as diagnostics, log files, asserts, test menus, etc.

 

Data. Everything that the product processes.

 

• Input: any data that is processed by the product.
• Output: any data that results from processing by the product.
• Preset: any data that is supplied as part of the product, or otherwise built into it, such as prefabricated databases, defaultvalues, etc.
• Persistent: any data that is stored internally and expected to persist over multiple operations. This includes modes or states of the product, such as options settings, view modes, contents of documents, etc.
• Sequences: any ordering or permutation of data, e.g. word order, sorted vs. unsorted data, order of tests.
• Big and little: variations in the size and aggregation of data.
• Noise: any data or state that is invalid, corrupted, or produced in an uncontrolled or incorrect fashion.
• Lifecycle: transformations over the lifetime of a data entity as it is created, accessed, modified, and deleted.

 

Platform. Everything on which the product depends (and that is outside your project).

 

• External Hardware: hardware components and configurations that are not part of the shipping product, but are required (or optional) in order for the product to work: CPU's, memory, keyboards, peripheral boards, etc.
• External Software: software components and configurations that are not a part of the shipping product, but are required (or optional) in order for the product to work: operating systems, concurrently executing applications, drivers, fonts, etc.
• Internal Components: libraries and other components that are embedded in your product but are produced outside your project. Since you don’t control them, you must determine what to do in case they fail.

 

Operations. How the product will be used.

 

• Users: the attributes of the various kinds of users.
• Environment: the physical environment in which the product operates, including such elements as noise, light, and distractions.
• Common Use: patterns and sequences of input that the product will typically encounter. This varies by user.
• Disfavored Use: patterns of input produced by ignorant, mistaken, careless or malicious use.
• Extreme Use: challenging patterns and sequences of input that are consistent with the intended use of the product.

 

Time. Any relationship between the product and time.

 

• Input/Output: when input is provided, when output created, and any timing relationships (delays, intervals, etc.) among them.
• Fast/Slow: testing with “fast” or “slow” input; fastest and slowest; combinations of fast and slow.
• Changing Rates: speeding up and slowing down (spikes, bursts, hangs, bottlenecks, interruptions).
• Concurrency: more than one thing happening at once (multi-user, time-sharing, threads, and semaphores, shared data).

 

Quality Criteria Categories

 

''A quality criterion is some requirement that defines what the product should be. By looking thinking about different kinds of criteria, you will be better able to plan tests that discover important problems fast. Each of the items on this list can be thought of as a potential risk area. For each item below, determine if it is important to your project, then think how you would recognize if the product worked well or poorly in that regard.''

 

Operational Criteria

 

• Capability: Can it perform the required functions?
• Reliability: Will it work well and resist failure in all required situations?
• Error handling: the product resists failure in the case of errors, is graceful when it fails, and recovers readily.
• Data Integrity: the data in the system is protected from loss or corruption.
• Safety: the product will not fail in such a way as to harm life or property.
• Usability: How easy is it for a real user to use the product?
• Learnability: the operation of the product can be rapidly mastered by the intended user.
• Operability: the product can be operated with minimum effort and fuss.
• Accessibility: the product meets relevant accessibility standards and works with O/S accessibility features.
• Security: How well is the product protected against unauthorized use or intrusion?
• Authentication: the ways in which the system verifies that a user is who she says she is.
• Authorization: the rights that are granted to authenticated users at varying privilege levels.
• Privacy: the ways in which customer or employee data is protected from unauthorized people.
• Security holes: the ways in which the system cannot enforce security (e.g. social engineering vulnerabilities)
• Scalability: How well does the deployment of the product scale up or down?
• Performance: How speedy and responsive is it?
• Installability: How easily can it be installed onto its target platform(s)?
• System requirements: Does the product recognize if some necessary component is missing or insufficient?
• Configuration: What parts of the system are affected by installation? Where are files and resources stored?
• Uninstallation: When the product is uninstalled, is it removed cleanly?
• Upgrades: Can new modules or versions be added easily? Do they respect the existing configuration?
• Compatibility: How well does it work with external components & configurations?
• Application Compatibility: the product works in conjunction with other software products.
• Operating System Compatibility: the product works with a particular operating system.
• Hardware Compatibility: the product works with particular hardware components and configurations.
• Backward Compatibility: the products works with earlier versions of itself.
• Resource Usage: the product doesn’t unnecessarily hog memory, storage, or other system resources.

 

Development Criteria

• Supportability: How economical will it be to provide support to users of the product?
• Testability: How effectively can the product be tested?
• Maintainability: How economical is it to build, fix or enhance the product?
• Portability: How economical will it be to port or reuse the technology elsewhere?
• Localizability: How economical will it be to adapt the product for other places?
• Regulations: Are there different regulatory or reporting requirements over state or national borders?
• Language: Can the product adapt easily to longer messages, right-to-left, or ideogrammatic script?
• Money: Must the product be able to support multiple currencies? Currency exchange?
• Social or cultural differences: Might the customer find cultural references confusing or insulting?

 

''Designed by James Bach, james@satisfice.com, www.satisfice.com, Copyright 1996-2006, Satisfice, Inc.''

 

_______________________________________________________________________________________________________

 

Michael Kelly Tours

 

FCC CUTS VIDS

 

http://www.testingreflections.com/node/view/2823

 

• Feature tour: Move through the application and get familiar with all the controls and features you come across. 
• Complexity tour: Find the five most complex things about the application. 
• Claims tour: Find all the information in the product that tells you what the product does.

 

• Configuration tour: Attempt to find all the ways you can change settings in the product in a way that the application retains those settings. 
• User tour: Imagine five users for the product and the information they would want from the product or the major features they would be interested in. 
• Testability tour: Find all the features you can use as testability features and/or identify tools you have available that you can use to help in your testing. 
• Scenario tour: Imagine five realistic scenarios for how the users identified in the user tour would use this product. 

 

• Variability tour: Look for things you can change in the application - and then you try to change them.
• Interopeability tour: What does this application interact with? 
• Data tour: Identify the major data elements of the application. 
• Structure tour: Find everything you can about what comprises the physical product (code, interfaces, hardware, files, etc...).

 

Based upon an idea by Michael Kelly http://www.michaeldkelly.com

 

_______________________________________________________________________________________________________

 

HICCUPPS

 

Consistency (“this agrees with that”) - an important theme in oracles

 

• History: If a product is inconsistent with previous versions of itself, we suspect that there might be a problem.
• Image: If a product is inconsistent with an image that the company wants to project, we suspect a problem.
• Comparable Products: When a product seems inconsistent with a comparable product, we suspect that there might be a problem.
• Claims: When a product is inconsistent with claims that important people make about it, we suspect a problem.
• User Expectations: When a product is inconsistent with expectations that a reasonable user might have, we suspect a problem.
• Purpose: When a product is inconsistent with its designers’ explicit or implicit purposes, we suspect a problem.
• Product: When a product is inconsistent internally as when it contradicts itself we suspect a problem.
• Statutes and Standards: When a product is inconsistent with laws or widelyaccepted or relevant standards, we suspect a problem.

 

Orignal idea from James Bach and Michael Bolton
http://www.developsense.com/articles/2005-01-TestingWithoutAMap.pdf