Software Reliability and Software Quality
(SENG 521)
Course Outline

TBA

1. Nothing yet!

1. Course Description and Outline

SENG521 is a step by step description of software quality and software reliability engineering process. It includes introduction to software quality, prediction and measurement of software size and cost, software reliability engineering process, defining necessary reliability, developing operational profiles, decision making based on the test results, techniques to improve and predict software reliability, application of quality concept to agile and incremental software development processes. The focus is on the reliability of object-oriented software systems. A workshop (project) is designed to reinforce the presented material. In the workshop, the students will actually go through the estimation and evaluation of quality of a realistic software project.

2. Course Web Site

The SENG 521 course home page contains links to up-to-date course information, problem assignments, announcements, as well as laboratory and examination scheduling. The SENG 521 course home page is available through the B.H. Far�fs home page at the URL:
(http://www.enel.ucalgary.ca/People/far/Lectures/SENG521/)

3. Allocation of Marks

4. Problem Assignments and Laboratories

Regular problem assignments will be given out of the course textbook, but are not required to be handed in for marking.

During the course the students are grouped into teams composed of 2-3 members. Each team is asked to design a moderately realistic software project and test it. A list of projects that the students can select among them will be posted on the course web page. The laboratories reports (design and test documents) of various phases of the assigned project should be handed in for check and marking. The reports are reviewed and a group discussion will be held in the tutorial periods. Unlike many undergraduate courses, and similar to what is actually performed in industrial practice, the laboratories hours are for group discussion and review of the projects. Additional discussion and review hours can be arranged by the students, TAs and the instructor.

5. Details of The Assignments

1. Unlike many other undergraduate courses and similar to industrial practices, the laboratory hours are not for delivering tutorials or solving problems. The designated lab hours are for group discussion and review of the projects. The meetings for review and discussion can be held either in the designated lab space or outside.
2. The designated lab hours are the core time that the TAs and course Instructor are available to review the projects. It is the students' responsibility to arrange meetings and seek advice from them during these hours. Meeting other than the core time is not guaranteed and requires prior arrangement with the TAs or Instructor and is subject to their availability. Attending the review sessions and discussions during the lab hours is necessary and failing to attend will cost each team/member about 10% deduction of the total mark for the corresponding assignment.
3. Assignments are to be delivered by their respective due dates. The due date for each assignment is explicitly mentioned on the assignments handouts. Unexcused delayed submission or failing to submit the required deliverables will be treated as zero. Students must submit ALL assignments to pass the course.
4. For group assignment one copy for the group will be sufficient.
5. The language for the deliverables must be English with ASCII or ISO-8859-1 character set.
6. The deliverable pages should be numbered and each assignment should have a cover page bearing the assignment number, student(s) name and student(s) ID number(s), and submission date. The cover page is excluded from the minimum and maximum required pages.
7. Assignments must be delivered in both electronic and hardcopy form. In case of electronic submission only formatted documents (MS Word, HTML, Postscript or PDF) are acceptable. Using other word processor software is encouraged but the output should be saved or converted to one of the above mentioned formats. Unformatted text or LaTeX files should be compiled or converted to Postscript or PDF before submission. HTML formatted files are acceptable only if accompanied by all the links and inline graphics, etc., in a single file (i.e., web archive).
8. Electronic version of assignments must be uploaded to the course Blackboard page at the due date together with the printed ones. Notice that electronic assignments WILL NOT be accepted if delivered via email.
9. For electronic submissions, the student is responsible to have the electronic media made readable. Defective material will be rejected and marked zero.
10. For electronic submissions, the student is responsible to have the files and attachments scanned for viruses before submission. Virus infected deliverables will be deleted automatically and marked zero.
11. Inquiries about the assignments can be forwarded to either TA or the course instructor. Email inquiries are acceptable, provided that the keyword SENG521 and the assignment number (ASSIGNMENT no. 1 to 5), e.g.,
    SENG 521: ASSIGNMENT no. 1
    be mentioned in the Subject line of the message. Inquiries not complying with this will be discarded with no action.

• Motivation
• Learning
• Interest and progress
• Retention
• Ability degree
• Attention
• Effort

6. Detail Contents

• Introducing the course.
• What is software reliability?
• What factors affect software quality?
• What is software reliability engineering?

• Software size
• Size: Length (code, specification, design)
• Size: Reuse
• Size: Functionality (function point, feature point, object point, use-case point)
• Size: Complexity

• Software cost model
• COCOMO and COCOMO II
• Constraint model
• Software lifecycle management (SLIM)
• Cost models: advantages and drawbacks

• Software quality
• Software quality models: Boehm�fs model, McCall�fs model, ISO 9126 model, etc.
• Software quality standards and metrics
• Measuring customer satisfaction
• Software quality assurance (SQA)
• Software quality management

• Hardware and software reliability
• The SRE process

• Basic Features of the Software Reliability Models.
• Single Failure Model.
• Reliability Growth Model.
• Exponential Failure Class Models.
• Weibull and Gamma Failure Class Models.
• Infinite Failure Category Models.
• Bayesian Models.
• Early Life-Cycle Prediction Models.

• Introducing failure severity class and failure intensity objective concepts.
• Steps in defining necessary reliability.
• Computing failure intensity objective for developed software.
• Case studies.

• System reliability
• Reliability Block Diagram (RBD)
• Serial and parallel configuration
• Active redundancy
• Hazard analysis: FMEA, FTA

• Fault prevention strategy: software product and process improvement using ISO 9000-3.
• Introducing fault tolerance concepts and definitions. Coincident, correlated and dependent faults.
• Fault tolerance phases.
• Recovery block mechanism and Acceptance testing.
• Exception handling, expected and unexpected events.
• Construction of robust software systems.
• Defensive programming, dual software technique.
• Adjudication by voting.
• Recovery blocks, N-version programming, Consensus recovery block, Acceptance voting, N self-checking programming.

• Defining function, operation, run, run type, operational mode, operational profile.
• Representation of operational profile.
• Procedure to define operational profile.
• Create functions/operations list.
• Determine occurrence rate of individual operations.
• Determine occurrence probabilities.

• Direct and indirect input variables.
• Operation, load and regression test.
• What is a test case?
• How to manage test cases?
• Test procedure.
• Equivalence classes and boundary conditions.
• How to allocate test time among system components based on test type (feature test, regression test, load test) and operation modes?
• In what order the test should be carried on?
• Running a successful test
• How to document the execution results?

• Introducing SRE simulation and reliability growth tools such as: SMERFS, SRMP, SoftRel, CASRE.
• Case study using CASRE.

• Guiding decisions related to certification test.
• Guiding decisions related to reliability growth test.
• Guiding decisions related to adequacy of tests.
• Handling program evolution, ureported failures and variations of operational profiles.

• Software Quality System (SQS); Software Quality Assurance (SQA) and Software Reliability Engineering (SRE)
• Quality, test and data plans
• Roles and responsibilities
• Sample quality and test plan
• Defect reporting procedure

• SRE for Agile development
• Cleanroom software development

• Final review of the course (9/Dec./2011)
• Final examination TBA.

7. Textbooks and Suggested References

• Software Reliability Engineering: More Reliable Software Faster and Cheaper, John D. Musa, (632 p.), Authorhouse, 2nd edition, 2004.
  ISBN 1418493872.

• Metrics and Models in Software Quality Engineering, Stephen H. Kan, 2nd ed. (560 p.), Addison-Wesley Professional (2002).
  ISBN: 0201729156.

• Handbook of Software Reliability Engineering, Michael R. Lyu (Editor), McGraw Hill (1996).
  ISBN: 0-07-039400-8.

• Software Reliability: Measurement, Prediction and Application, J.D. Musa, A. Iannini, K. Okumoto, (621 p.), McGraw-Hill (1987).
  ISBN 0-07-044093-X.

1. Addison-Wesley - A Practical Guide to Testing Object Oriented Software (2001).
2. Addison-Wesley - Critical Testing Processes Plan Prepare Perform Perfect (2003).
3. Addison-Wesley - Effective Software Testing (2002).
4. Addison-Wesley - Software Test Automation (1999).
5. Addison-Wesley - Test Driven Development By Example (2002).
6. Addison-Wesley - Testing Embedded Software (2003).
7. Addison-Wesley - Testing Extreme Programming (2002).
8. Apress - Software Exorcism A Handbook for Debugging and Optimizing Legacy Code (2003).
9. Apress - Test Driven Development A J2EE Example (2005).
10. Artech House - Building Reliable Component based Software Systems (2002).
11. Artech House - ISO 9001 2000 Quality Management System Design (2003).
12. Artech House - Practical Guide To Software Quality Management 2Ed (2003).
13. Artech House - Software Fault Tolerance Techniques and Implementation (2001).
14. Artech House - Testing and Quality Assurance for Component Based Software (2003).
15. CMU TR022-96 - Cleanroom Software Engineering Reference Model.
16. CRC Press - Design for Reliability (2001).
17. Elsevier Science - Reliability Maintainability and Risk (2001).
18. IEEE - Preventive Risk Management For Software Projects.
19. IEEE - SWEBOK Software Engineering.
20. IEEE 1008-1987 - Standard for Software Unit Testing (1986).
21. Kluwer - Computing System Reliability Models and Analysis (2004).
22. Manning - JUnit Recipes Practical Methods for Programmer Testing (2005).
23. Microsoft Press - Testing Software Patterns (2003).
24. Morgan Kaufmann - Unit Testing in Java How Tests Drive the Code (2003).
25. NIST - The Economic Impacts of Inadequate Infrastructure for Software Testing (2002).
26. OReilly - Perl Testing A Developers Notebook (2005).
27. Pragmatic Programmers - Pragmatic Unit Testing in C# with Nunit (2004).
28. Pragmatic Programmers - Pragmatic Unit Testing in Java With Junit (2003).
29. Pragmatic Programmers - Pragmatic Unit Testing in Java with JUnit (2003).
30. Prentice-Hall - Java Testing And Design From Unit Testing To Automated Web Tests (2004).
31. Prentice-Hall - Java Testing Design and Automation (2004).
32. Prentice-Hall - Software Development Building Reliable Systems (1999).
33. Springer Verlog - Practical Software Testing (2003).
34. Springer Verlog - Reliability Engineering Theory and Practice 3Ed (1999).
35. Thomson - Game Testing All in One (2005).
36. Wiley - Automated Web Testing Toolkit - Expert Methods For Testing and Managing Web Applications (2001).
37. Wiley - Case Studies in Reliability and Maintenance (2003).
38. Wiley - Optimal Reliability Modeling Principles and Applications (2003).
39. Wiley - Reliability Survivability and Quality of Large Scale Telecommunication Systems (2003).
40. Wiley - Reliability of Computer Systems and Networks - Fault Tolerance Analysis and Design (2002).
41. Wiley - Software Quality Engineering - Testing, Quality Assurance and Quantifiable Improvement (2005).
42. Wiley - Software Testing Fundamentals Methods and Metrics (2003).
43. Wiley - Statistical Design and Analysis of Experiments 2Ed (2003).
44. Wiley - Statistical Methods for Survival Data Analysis 3Ed (2003).
45. Wiley - The Art of Software Testing 2Ed (2004).
46. Wiley - The Web Testing Companion (2003).

8. Other Information

• IEEE Computer Society Technical Committee on Software Reliability Engineering:
  Publishes a newsletter and sponsors the annual International Symposium on Software Reliability Engineering. Membership application at: http://www.tcse.org
  
• Software reliability engineering:   John D. Musa web site
• Reliability Engineering, University of Maryland, USA
• Software QA/Test Resource Center
• National Software Testing Labs
• Software Testing Online Resources
• NIST Engineering Statistics Handbook
• Quality Assurance and Software Testing Books
• Journals:
  
  1. IEEE Transactions on Reliability.
  2. Reliability Engineering and System Safety, Elsevier Science.
     
  3. IEEE Software.
     
  4. IEEE Transactions on Software Engineering.
  5. IEE Proceedings - Software.
  6. Transactions on Software Engineering and Methodology (TOSEM), ACM.
     
  7. Information and Software Technology, Elsevier Science.
  8. Annals of Software Engineering, Kluwer.
  9. Automated Software Engineering, Kluwer.
  10. Software -- Practice and Experience, Wiley.
  11. Journal of Software Maintenance, Wiley.
  12. International Journal of Software Engineering and Knowledge Engineering, World Scientific.
• Newsletters:
  
  1. The DoD Software Tech News
     Provides cutting-edge topics on Software Technology information.
• Conferences:
  
  1. International Symposium on Software Reliability Engineering (ISSRE)
     
  2. International Conference on Software Maintenance (ICSM)
     
  3. International Conference on Software Engineering (ICSE)
     
  4. Reliability and Maintainability Symposium (RAMS)