Charter Document

Enterprise Level Course Management System Evaluation

1. Executive Summary

The purpose of this document is to explore various build or buy alternatives for the Ohio State University Course Management System and propose a method of evaluation.

Upon completion of this phase, the case for building or buying a Course Management System (CMS) will be presented to the Executive Sponsors for approval. Upon approval, evaluation of the alternatives and recommendation to the CIO will be the next steps. A final step is a charter for the implementation project that details the implementation and maintenance of the CMS.

2. Introduction

The Ohio State University's Course Management System, WebCT (version 3.8), operated by the Office of Information Technology, with separate systems operated by the Ohio State Medical School and the College of Mathematical and Physical Sciences, has served faculty and students since 1998. The system provides faculty with syllabus, email, chat, and grade book functions; it provides students with course material, communication functions, and online evaluation; it provides the University with student enrollment listing and grading functions.

While the system is currently in place and familiar to many students, faculty, and staff, there are pressures to change the system: the office of the CIO and OIT would like to move the system to an "enterprise-level," that functionally integrates with a Student Information System. At the same time, WebCT has announced a new enterprise-level system (Vista) with significantly increased costs.

In the fall of 1998, Ohio State University invited ten or more CMS vendors to demonstrate their products on campus. About 50 people came to the demonstrations. Of the systems represented, WebCT seemed to meet faculty's needs best; in addition, early adopters of CMS systems on campus were generally using WebCT. Since 1998, Ohio State University has supported use of WebCT and Course Sorcerer, a home-grown Ohio State product that provides enrollment and grade book functions.

Over the years since 1998, Ohio State University has supported use of WebCT and provided adequate human resources to support faculty use of the system. The university has also paid for premium level support from the company. On the other hand, the university has provided less funding for professional training of those working with faculty in learning WebCT.

For a year or more, six colleges bought their own licenses for WebCT and ran their own instances of the CMS. In 2000 (?), the university moved to a campus-wide license, although some colleges still run their own instance.

Currently, the university has 28,000 users of WebCT enrolled in 300-500 courses. The largest WebCT courses are in the departments of biology, statistics, chemistry, and theater. The WebCT functions most often used are updating class rosters, posting class materials, linking to external resources, and email. There is less usage of the chat or drop box features.

Ohio State University is one of a very few very large institutions that has one course management system. The current version of WebCT for the university (not individual instances such as the College of Medicine) runs on one central server.

The main advantage of WebCT 3.8 is that it provides flexible instructional strategies (at the expense of user interface). The disadvantages of WebCT 3.8 include scalability problems, difficult user interface, and organization that does not allow for easy control or management of learning objects.

The new system should be easily scalable to numbers greater than 50,000; it should have an easily learned user interface; it should allow for organization at the learning object level.

3. Drivers for Change

The goal for a CMS is to provide the best functionalities possible to Ohio State faculty and students, the best infrastructure to the administration and staff of the University, and to meet anticipated future needs of the University.

To determine how best to deliver this goal, it is important to:

• Know what functionalities the enterprise level CMS have today
• Identify what functionalities OSU's faculty need
• Identify what functionalities OSU's students need
• Understand the relative criticality of those functions as they apply to the CIO's mission, the Academic Plan, and PlanIT.
• Identify the features and infrastructure that will best serve the current and future needs of the University
• Identify the external factors affecting this decision (budget restructuring).
• Understand the cost and funding of the various CMS alternatives.
• Understand the advantages and disadvantages of multiple alternatives for providing these functionalities, features, and infrastructure.

4. Requirements/Success Factors

• Evaluation of the alternatives and recommended solution must be unbiased and must be perceived to be a fair evaluation with all stakeholders' positions considered.
• For the "Buy" products, alternatives must be at enterprise level and must have been implemented in a similar environment.
• The costs must meet the guidelines/budget set by the executive sponsors.
• The schedule must meet a schedule set by the executive sponsors.

5. Stakeholders for CMS Evaluation

Constituencies we have identified include:

• Faculty
• Students (undergraduate and graduate)
• Courseware designers
• help desk representatives
• ADA compliance representatives
• System Administrator representatives
• College representatives (MAPS, Pharmacy, College of Medicine, etc.)
• The registrar's office/operators of the Student Information Systems
• Libraries
• Regional campuses
• Continuing education
• OSU extension

5.1 Stakeholders/ Governance (see appendix):

1. Executive sponsors -- Ilee Rhimes (CIO), deans and vice-provosts
2. Primary executive --Susan Metros
3. Advisory team -- TELR Coordinating Council, program directors, students, other constituencies

6. Approach for CMS Evaluation

Phase I: evaluate various CMS alternatives and
Identify the top several "buy" alternatives
Identify a top "build" alternative
Investigate a "wait for a specified period of time" alternative
And document the decisions made

Phase II:

• consider the various "affiliated questions" -- building blocks, portal, how system should work with library and OCLC
• Investigate other universities' experience
  • What do they like/not like
  • Experience with company/ build your own
  • What are the problems of their systems
  • What are the "repair records" of their systems
  • What advice would they give us
  • What system has been most cost-effective
• Identify a prioritized list of features desired
  • Identify functionality available
  • Prioritize the list of functionality desired
• Identify a prioritized list of infrastructure considerations desired
  • Identify infrastructure features available
  • Prioritize the list of infrastructure features available
• Evaluation
  • Create a decision matrix using the evaluation criteria
  • Evaluate the various buy alternatives against each other; determine top choice
  • Evaluate various build alternatives against each other; determine top choice
  • Evaluate the wait for a specified period of time alternative
• review and approve the top choices.

Phase III

• Evaluate the buy and build and wait alternatives against each other
• Identify the recommended solution
• Expected outcomes:
  • A comprehensive pricing of the top choices
  • Identify specific tangible benefits
  • Full information on cost, schedule, infrastructure, risks of each alternative

Phase IV

• Recommendation to the CIO
• Write charter document for implementation project

6.2 Alternatives to Evaluate

Build Alternative:

• Build a completely new system with others in the CIC consortium

Buy Alternative:

• WebCT Vista
• Blackboard 6.0
• Angel 5.5
• Other possibilities (Anlon, CentraOne, Claroline, ClassWeb, Click2Learn, Colloquia, COSE, Desire2Learn, eCollege, Educator, EduSystem, e-education, Eledge, FirstClass, Fle, Internet course assistant, IntraLearn, Manhattan Virtual Classroom, MimerDesk, The Learning Manager, Theorix, VirtualU, WebTraining Toolbox)

Wait Alternative

• Wait a specified period of time and then make a switch.

6.3 Evaluation Criteria

The various alternatives will be evaluated based on the following factors:

• Schedule
  • What are timeline and staffing impacts?
  • How would these staffing levels impact faculty and student services?
  • How long would the implementation take for each of the options?
  • How quickly would value and benefits be realized?
• Costs
  • Hardware
  • Software
  • Vendor Consulting
  • License fees
  • Staff person-hours
  • Other costs (hidden?)
• Functionality
  • How will each option meet Ohio State's functional and strategic requirements?
  • What functions are currently being used by faculty?
  • Fit of the features with our business processes. What will we have to change in the way we do things to make the product work in our environment?
  • Will any of the functions impact on Ohio State University's relationship with other institutions?
• Technology
  • How well does each option leverage Ohio State's technology investment and match University's technology direction?
  • Service and strategy (Outward)

    How would each option enable OSU's CIO organization to deliver superior service to clients (students, faculty, and staff)? To what extent would the option affect the organization's ability to change to meet the service needs?

  • Infrastructure (Inward)

    How does each option support or fit with OSU's current technical infrastructure? What impact would each option have on hardware and software standards (i.e., function within OSU's existing environment, require new hardware or software tools)?

• Comparative/Competitive
  • How would each option enable Ohio State to be competitive in the higher education marketplace?
• Risk
  • Level of risk / Ability to achieve success
    • Each option will have different level of risk. Specifically, each option will be assessed based on the likelihood of the success of the overall effort.
  • User Acceptance
    • How will the campus react to/support the option? How well does each option match current campus expectations?
    • Will some areas of the university be affected by the option more than others? What kind of support will those areas need?
  • Flexibility to support a changing strategic direction of the University
    • With whatever option is chosen, the University should have the ability to consistently re-evaluate it's strategic direction and to know that the technology and business processes will be able to support the new direction.
  • Vendor Stability
    • Will the company survive and continue to support its products?
  • Stability of the product
    • Will the product change over the next 1-5 years, imposing greater costs (support, training, money) on Ohio State?
• Governance
  • How much time, effort and intensity is required by executive sponsors?
  • The extent that sponsorship and advisory committees would be required to play a role in the governance and control of the implementation.
• Change Management
  • Identify the value of new product features/functions to what currently exists. What is important to the various stakeholders? How much are they willing to "pay" for the new features? i.e. time, effort.
  • What is the breakdown of your current user population? Basic, intermediate, advanced users. How will this change over time? How will their needs change over time?
  • What is the organizational learning curve?
  • Assess current state of organization to support change.
  • Who will spend time doing what? centralized vs. decentralized impact
  • What is the balance of talent/skills needed, centralized vs. decentralized?
• Return on Investment/ Value on Investment
  • What are the costs and likely quantifiable benefits of each option?
  • The cost decision factor focuses on total project cost and long-term maintenance costs. While the project costs may be short term or long term depending on the option, the overall cost for implementation, resources, software and hardware will be taken into account
  • Time to realize value and benefit