An Evaluation Guide
















By Andy Conrad, Stacy Phelps, and Cliff De Long






Oglala Lakota College (OLC) is seeking to institutionalize technology as a method of disseminating quality, full-time faculty to students in outlying district college centers.  The investment of time and resources over the past five years has been very significant.  The decentralized structure of OLC lends itself to implement technology effectively as a solution to overcome the vast distances and the issues that are associated with the decentralized structure.  I like this introduction.  Short succinct, an it sets a good stage for the next pieces of this plan.



Evaluation Purpose

Our overall plan is to evaluate the current technology in the area of technical support with an eye to monitor the possibility of building capacity in the area of technical support. Not real clear what you mean here.  I think youre saying you want to assess the need for additional support by evaluating the current level of tech support.  Right? Oglala Lakota College (OLC) is currently placing special emphasis on staff training components and technical support (help desk). This evaluation plan is a means by which a more comprehensive institutional technology plan will be put into place at OLC.

Audience and Stakeholders

Our target audience includes the administration of OLC; its students, support staff, and faculty.

Evaluation Context

Proposals for implementation include a staffed hotline for technical support for videoconferencing and computer lab problems.  We are contemplating upgrading our collaboration server to Exchange 2002 for increased web-based communication and collaboration capabilities.  In order to better serve the institution with technical support, restructuring of the existing positions and job descriptions has been proposed and is under consideration by administration.  Okay, this explains the first sentence of your purpose statement.  Some front end of this would help in the evaluation purpose section. Remember, Im trying to see this through the eyes of a lay technologist.  So, keep your reader in mind as you plan and write.


Oglala Lakota College.

OLC is a North Centrally accredited academic institution, which provides associates, baccalaureate, and graduate degrees in many different fields of study.  Since its inception, OLC has been a distance learning institution.  OLC traditions of distance learning evolve from the early 1980s when instructors traveled to individual sites (trailers, homes, public buildings, etc.) across the reservation to have classes.  This tradition of learning has evolved to now the 10 different learning centers across the reservation and in Rapid City. 


OLC was one of the first tribally controlled colleges in the United States. The concept of a tribally controlled college is that it be sanctioned by an Indian tribe, it's governed by an Indian tribe, it's governing body be made up of tribal members; and meet the needs of reservation people in their pursuit of higher education. said Tom Short Bull, President of OLC.


Presently, there are over 1200 students at the 10 different learning centers.  The average age of an OLC student is 32, and there are almost 4 female students for every male student.  Most classes typically meet once a week, so they range from one to four hours in length.  There are no classes on Friday thus allowing various meeting to be held for both faculty and students.  Good.


Technical Systems.

 Each center is connected to the central site for OLC (Piya Wiconi) by a dedicated T-1 line.  Piya Wiconi has been operating a VideoServer bridge since 1997.  On-line registration is in use, and all of the centers have broadband Internet access.  Also, each of the centers has the capability of originating and receiving videoconferences through the use of PictureTel equipment.  During the current semester, 17 classes are being offered over the PictureTel equipment and a few classes are being offered totally over WebCT.  Numerous classes are using WebCT as a supplemental classroom source. All of the centers have microcomputer labs, workgroup servers, routers, and networked laser printers.  Piya Wiconi is connected to the world by a pair of T-1 lines for Internet access as well as a frame-relay 56k line for secure VPN financial transactions for the administration offices.  OLC currently maintains in production: 29 servers, 15 routers and 514 workstations.  This maintenance is provided by the full-time faculty, adjunct faculty and by students.  Although the wide area network and technology infrastructure in our opinion has been relatively stable and trouble free, this may not be the perception of the administration, staff, faculty, and student body.  The entire technology infrastructure has been acquired through grants (corporate, private and federal no state).  Institutional funds are not used for technology supplies, repair, maintenance or acquisition.  The Bush Foundation pays for advanced degrees for the faculty if they wish to pursue them.  Training grants have been received in the past, and are currently under review.  Professional commensuration is above average for South Dakota but still falls below the national average.


Over the Christmas break (2001) our college will be implementing a new video bridge, new routers, and new videoconferencing devices.  We have been using web-based communication tools such as NetMeeting and WebCT chat for instruction and are interested in their quality.  Many of our CIS and MIS courses are taught at nine different computer labs across the reservation and at the He Sapa center.  We recently implemented workgroup servers and print servers as well as Windows 2000 workstations in those labs, and are interested in the stability of this design as well as end-user satisfaction.  We have a software package / server for technical support Track-It for which we would like to determine effectiveness.  An Elron firewall, Microsoft proxy server and Network Address Translation (NAT) at the router levels aid in providing network security.  Internet filtering is in use, denying access to objectionable sites.  The Acceptable Use Policy was updated in the spring of 2001.


Staff Support.

Training is offered to faculty in the use of PictureTel equipment and WebCT on a regular basis.  MIS, CIS and IT courses are offered every semester for the students of the institution, and all faculty and staff are allowed and encouraged to attend those courses at no cost.

Evaluation Approach

We would prefer to use external evaluators due to our own bias and would like a summative approach to the evaluation using a spectrum of evaluative instruments.  The overall goal of this evaluation is to improve the educational process for all involved parties at OLC, but most especially for the student.  Evaluating the effectiveness of the technology is fundamentally necessary at this point in time for our upcoming North Central review.


Im going ahead with my review even though it looks like you still need to clean up here in  this section.  I cant put it of any longer.  Not too much of a deduction anyway.    


These front end pieces read much more sequentially and informatively with your recent changes.  Good job with the adjustments.


The current technology at OLC moves us to ask the following guiding evaluation questions.

Guiding Evaluation Questions:

1. In what ways is technology used for instruction and student/staff productivity?

2. What is the availability of technical support and does it meet your student, faculty, and administrative needs?

3. How can the stability and functionality of our overall technology delivery system be improved? 

4. What has been the quality and utility of the technology training that you have receivedprovided to users of the system?

5. Is the current technology inventory sufficient to meet the instructional and administrative needs of the college?



You paired down the questions a good deal.  They seem reasonable though and give you flexibility as you investigate them.  Also, we may be adding a question or two in the last few weeks of plan development.

Description of Evaluation Approach:

We need to evaluate query (you wouldnt be evaluating the users as much as evaluating the technology system. administration, staff, instructors, adjunct instructors and students.  The query with both formative and summative questions.  These questions would be of Likert scale and essay type Here Im led to believe that youre only conducting a survey, which would not be consitant with the breadth and depth of information you need to get.  A survey would be a good initial strategy, but dont focus on it up front..  Evaluation of administration, staff, and instructors would be conducted during the December 2001 All-Staff meeting.  Typically at the December All-Staff, there are over 200 in attendance, which would should yield give us good results.  Evaluation ofQuery adjunct instructors would be completed by mail.  An SASE envelope would be included to help insure a high probability of response.  Students would complete their technology questionnaires evaluations at the same time that they would complete their course evaluation forms. evaluate their classes.  Another component that we feel is necessary is to conduct face-to-face interviews, using the Institutional Director of Assessment, with our 10 centers staff.  The instruments will be paper surveys.  We feel that this face-to-face interview method would provide us with the most accurate assessment of the technology. this will get our highest rate of response. 


Evaluation Methods:


There will be several evaluation methods used to gather data from administrators, staff, instructors, students, college center personnel and computer audits; questionnaires; group interviews; and by viewing computer logs and application programs loaded on center computers. 


Information would be gathered in a short period of time to produce a brief  snap shot of how technology is viewed at OLC.


Pertinent survey questions will be asked of administrators, staff, instructors and students.  The results from this paper survey will be manually tabulated and categorized using standard statistical methods and the results will be disseminated on the colleges website via user-friendly tables and charts for all stakeholders to review.


Information Technology Internship students will conduct computer audits.  They will audit a random sample of 33% of each center computers paying attention to computer logs as well as the name and type of application program that has been loaded onto each of the audited machine.  A listing of the different names and types of application software will be added to the website with provisions for instructors to request the application software to be loaded onto machines at specific college centers.  good


The Assessment Office at each of the college centers will perform focus group interviews.  The Assessment Office will ask probing questions regarding the technology available to each college center and how each center feels that the technology is being utilized.  Results from the interviews will be tabulated into list form for inclusion on the website.


The following chart depicts the methods of data collection and data analysis interpretation as they relate to each question..


Evaluation Questions

Why the Question Is Important

Data Sources

Information Needed to Answer the Question  Lets change this column to Data Sources, and then see what happens

When and How the Information Will Be Collected

Analysis and Interpretation

1.  In what ways is technology used for instruction and student/staff productivity?


       Discover ways that technology is used that we were not aware of.

       In what ways is the technology being used.

       In what ways does the technology increase students and staff productivity?




       Lesson plans

       Computer lab logs

       Collection of instrument fromSurvey to admin, staff and faculty at all-staff.

       Survey from students during student course evaluation.Collection of instrument from students during student course evaluation

       Collection of instrument fromFocus group of college cCenter personnel during on-site interview.

       Randomly audit college center computers for installation of applications beyond original configurations suite.


Categorize and tabulate participants responses.  Group results that have reoccurred in different respondent queries together.  Results will be placed in a productivity table.  Placement in the table will occur based on the frequency of it occurrence in different surveyed groups.  If a technology is identified in each of the query methods, it will be rated higher in the table.  We will be able to determine which technologies are deemed more valuable for productivity based on where that technology is ranked in the productivity table.

2. What is the availability of technical support and how does it meet your needs?


       To ensure that the lack of technical support does not hampers the educational process.

       Results from Track It


       College center maintenance logs


       Instructor survey


       College center focus groups.

           The data would be collected using qualitative and quantitative surveys with instructional staff.

           Same survey will be completed during focus group evaluations of college center staff.


           All calls for service in whatever form they arrive will be entered into the Track It program and a maintenance ticket will be opened on them.  Track It will be queried for summation of service calls for duration of service tickets.


           Collect and summarize maintenance logs from college center, monitors computer service and distance learning service work on-site.


The results of the service request will be grouped and sorted in a table.  The table will categorize results based on length on time that calls take to get resolved, method of service request, commonality of service request, location of service needed.  We will also summarize what technicians are responding to calls and what technicians, if different from those that first responded, are fixing certain types of problems.  The table will help us to identify commonalities of problems with technology at OLC, how long problems are taking to get repaired, and what methods are preferred to request services.  We will be able to determine training needs of technicians based on what technicians are not responding to calls based on familiarity with technology and will be able to prioritize and technicians to handle different technology needs in different locations.  We can also determine if we need to do more for disseminating of methods to request technical assistance and how we can reduce the amount of time, if necessary, that is needed to complete service calls.

3. How can the stability and functionality of our overall technology delivery system be improved? 

       This will help us to improve the technology of the institution.

       All stakeholders

       Collection of instrument fromSurvey to admin, staff and faculty at all-staff.

       Survey from students during student course evaluation.Collection of instrument from students during student course evaluation

       Collection of instrument fromFocus group of college cCenter personnel during on-site interview.


Results of this open-ended question would go under two levels of analysis.  The first level will sort comments by relevant themes.  The second level will consist of a table will be constructed sowing the frequency and percentage of comments attributed to these themes.  Comments from each theme will then be summarized into a written report in ways that illuminate the perceived functionality and stability the technology has.

4. What has been the quality and utility of the technology training that you have received?


       Are the current trainings being offered helpful, do we need to offer other types of trainings.

       Is the content of the training being deployed in their use of technology.


       Survey of instructors at All-Staff.

The results will be tabulated based on trainings that are identify by instructors and which of those trainings are identified as useful.  Tabled results will show what trainings are being taken advantage (needs of instructors), what forms of training are considered helpful, what trainings are deemed valuable and which trainings need to be reconstructed to better suit the needs of the instructors.  A narrative summary will be constructed from suggestions as to how to improve trainings that are not identified as valuable.

5. Is the current technology asset list sufficient to meet the instructional and administrative needs of the college? If not, what is needed and why?

       From the perspective of the stakeholders is there sufficient technology in place (hardware, software, and web ware) to provide enough resources to meet their needs and the mission of OLC.

       All stakeholders

       9-12 students from Pine Ridge Reservation high schools.

       Interview stakeholders during All-Staff meetings

       Interview 9-12 students from high schools during Lakota Nation Winter Festival and college recruitment days at local Pine Reservation high schools.

Results of this open-ended question would go under two levels of analysis.  The first level will sort comments by relevant themes.  The second level will consist of a table will be constructed sowing the frequency and percentage of comments attributed to these themes.  Comments from each theme will then be summarized into a written report in summarizing what new technology is needed and currently not available at OLC for use.



Reporting Section:


The evaluation process will occur over the Christmas break time frame (December-January).  The surveys for faculty can be distributed during the December all staff meeting and the student surveys can be distributed during the fall course evaluation time period, which also occurs in mid to late November time frame.  The focus groups will be facilitated by the Institutional Assessment director, but will be completed during the Christmas break time frame.  The college center staff has more flexibility in time as their duties are reduced once the semester has been completed.  The random hardware audits will be completed during the Christmas break as well.  We do scheduled sweeps of each college center during the semester break as well.


The results of the evaluation will be compiled during the mid-January and early-February time frame.  The summarized finding will be posted on the college website and emailed in mass to all the faculty and staff.  A hardcopy of the report will also be placed in each college center facility.


At the March All-Staff meeting will make a formal presentation of the results and will ask for more recommendations of other areas that the faculty and staff would like to see collected.  Recommendation and findings that were concluded from the evaluation will be implemented in a fashion to reflect current practices and to change current practices that are not as valuable as viewed by the stakeholders.  New trainings and updates to technology as recommended by the stakeholders will be included in an institutional technology plan that will be presented to the Board of Trustees of OLC.  In presenting we will identify budgetary commitments and policy needs that will be sent through the committee system at OLC for implementation.  The evaluation will occur again in the fall semester each successive year and will be refined as the needs change and technology evolves based on the findings.


I like the short time-frame of this evaluation.  It can be clean, quick, and provide needed information in a timely manner.




The purpose of this evaluand is to see what types of software have been added to random sampled computer workstations.  If instructors are adding software to only a few workstations, it may be due to limits in the number of licenses available. 




Computer Serial Number                                                                   




O/S Version                                                                                        


Is the basic Office Suite installed?                  Yes                  No


Is Publisher installed                                       Yes                  No


Other software installed:













The purpose of this evaluand is to see if the hardware that is available is being utilized as intended.  This will be determined by looking at the log associated with the hardware.




Hardware Serial Number                                                                    


MIE/OLC Serial Number                                                                   


Hardware type                                                                                   


Amount of UseageUsage        Little               Average                       Above average

                                    1          2          3                      4                      5



Would information on the quality of performance of the technology components, and/or its utility given the instructional goals of the institution be useful?


Complaints about system:





The strength of this design is that you are very explicit with your data analysis procedures.  You also have very strong pre matrix components.  I like the writing you did there and I found little if any instances where I though you were elaborating on trivial events or issues. Kudos on the level of writing.


Unfortunately, you go a little sloppy, in the handling of the document and making revisions.  Thats probably a casualty of organizing the team work and making each other accountable.


Overall, good job.  Youve applied the basics of evaluation planning with good accuracy.  With some well placed revisions,  this is work that can provide some quality outcomes for you at OLC. It has the structure and definition to be a good guiding document for evaluating the technology support program.


Your score 110/125