An Evaluation Guide
OLC
2001-2002
By Andy
Conrad, Stacy Phelps, and Cliff De Long
Introduction
Oglala Lakota College (OLC)
is seeking to institutionalize technology as a method of disseminating quality,
full-time faculty to students in outlying district college centers. The investment of time and resources
over the past five years has been very significant. The decentralized structure of OLC lends itself to implement
technology effectively as a solution to overcome the vast distances and the
issues that are associated with the decentralized structure. I like this
introduction. Short succinct, an
it sets a good stage for the next pieces of
this plan.
Our overall plan is to evaluate the
current technology in the area of technical support with an eye to monitor the
possibility of building capacity in the area of technical support. Not
real clear what you mean here. I
think youÕre saying you
want to assess the need for additional support by evaluating the current level of
tech support. Right? Oglala Lakota College (OLC) is currently placing special
emphasis on staff training components and technical support (help desk). This evaluation plan is a means by which a more
comprehensive institutional technology plan will be put into place at OLC.
Our target audience includes the
administration of OLC; its students, support staff, and faculty.
Proposals for implementation include a
staffed hotline for technical support for videoconferencing and computer lab
problems. We are contemplating
upgrading our collaboration server to Exchange 2002 for increased web-based
communication and collaboration capabilities. In order to better serve the institution with technical
support, restructuring of the existing positions and job descriptions has been
proposed and is under consideration by administration. Okay, this explains the first sentence
of your purpose statement. Some
front end of this would help in
the evaluation purpose section. Remember, IÕm trying to see this through the
eyes of a lay technologist. So,
keep your reader in mind as you plan and write.
OLC is a North Centrally accredited academic institution, which provides associates, baccalaureate, and graduate degrees in many different fields of study. Since itÕs inception, OLC has been a distance learning institution. OLC traditions of distance learning evolve from the early 1980Õs when instructors traveled to individual sites (trailers, homes, public buildings, etc.) across the reservation to have classes. This tradition of learning has evolved to now the 10 different learning centers across the reservation and in Rapid City.
ÒOLC was one of the first tribally controlled colleges in the United States. The concept of a tribally controlled college is that it be sanctioned by an Indian tribe, it's governed by an Indian tribe, it's governing body be made up of tribal members; and meet the needs of reservation people in their pursuit of higher education.Ó said Tom Short Bull, President of OLC.
Presently, there are over 1200 students at the 10 different
learning centers. The average age
of an OLC student is 32, and there are almost 4 female students for every male
student. Most classes typically
meet once a week, so they range from one to four hours in length. There are no classes on Friday thus
allowing various meeting to be held for both faculty and students. Good.
Each center is connected to the central site for OLC (Piya
Wiconi) by a dedicated T-1 line.
Piya Wiconi has been operating a VideoServer bridge since 1997. On-line registration is in use, and all
of the centers have broadband Internet access. Also, each of the centers has the capability of originating
and receiving videoconferences through the use of PictureTel equipment. During the current semester, 17 classes
are being offered over the PictureTel equipment and a few classes are being
offered totally over WebCT.
Numerous classes are using WebCT as a supplemental classroom source. All
of the centers have microcomputer labs, workgroup servers, routers, and
networked laser printers. Piya
Wiconi is connected to the world by a pair of T-1 lines for Internet access as
well as a frame-relay 56k line for secure VPN financial transactions for the
administration offices. OLC
currently maintains in production: 29 servers, 15 routers and 514
workstations. This maintenance is
provided by the full-time faculty, adjunct faculty and by students. Although the wide area network and technology
infrastructure in our opinion has been relatively stable and trouble free, this
may not be the perception of the administration, staff, faculty, and student
body. The entire technology
infrastructure has been acquired through grants (corporate, private and federal
Ð no state). Institutional funds
are not used for technology supplies, repair, maintenance or acquisition. The Bush Foundation pays for advanced degrees
for the faculty if they wish to pursue them. Training grants have been received in the past, and are
currently under review.
Professional commensuration is above average for South Dakota Ð but
still falls below the national average.
Over the Christmas break (2001) our
college will be implementing a new video bridge, new routers, and new
videoconferencing devices. We have
been using web-based communication tools such as NetMeeting and WebCT chat for
instruction and are interested in their quality. Many of our CIS and MIS courses are taught at nine different
computer labs across the reservation and at the He Sapa center. We recently implemented workgroup
servers and print servers as well as Windows 2000 workstations in those labs,
and are interested in the stability of this design as well as end-user
satisfaction. We have a software package
/ server for technical support Track-It for which we would like to determine
effectiveness. An Elron firewall,
Microsoft proxy server and Network Address Translation (NAT) at the router
levels aid in providing network security.
Internet filtering is in use, denying access to objectionable
sites. The Acceptable Use Policy
was updated in the spring of 2001.
Training is offered to faculty in the use
of PictureTel equipment and WebCT on a regular basis. MIS, CIS and IT courses are offered every semester for the
students of the institution, and all faculty and staff are allowed and
encouraged to attend those courses at no cost.
We would prefer to use external
evaluators due to our own bias and would like a summative approach to the
evaluation using a spectrum of evaluative instruments. The overall goal of this evaluation is
to improve the educational process for all involved parties at OLC, but most
especially for the student.
Evaluating the effectiveness of the technology is fundamentally
necessary at this point in time for our upcoming North Central review.
IÕm going ahead with
my review even though it looks like you still need to clean up here in
this section. I canÕt put it of any longer. Not too much of a deduction anyway.
These front end
pieces read much more sequentially and informatively with your recent
changes. Good job with the
adjustments.
The current technology at OLC moves us to
ask the following guiding evaluation questions.
1. In what ways is technology used for instruction and student/staff productivity?
2. What is the availability of technical support and does it
meet your student, faculty, and administrative needs?
3. How can the stability and functionality of our overall technology delivery system be improved?
4. What has been the quality and utility of the technology
training that you have receivedprovided to users of
the system?
5. Is the current technology inventory sufficient to meet
the instructional and administrative needs of the college?
5.
6.
You paired down the questions a good deal. They seem reasonable though and give you flexibility as you
investigate them. Also, we may be
adding a question or two in the last few weeks of plan development.
We need to evaluate query (you wouldnÕt be evaluating the users as much as
evaluating the technology system. administration,
staff, instructors, adjunct instructors and students. The query with both formative
and summative questions. These
questions would be of Likert scale and essay type Here IÕm led to
believe that youÕre only conducting a survey, which would not be consitant with the breadth and
depth of information you need to get.
A survey would be a good initial strategy, but donÕt focus on it up
front.. Evaluation of
administration, staff, and instructors would be conducted during the December 2001 All-Staff
meeting. Typically at the December
All-Staff, there are over 200 in attendance, which would should yield give
us good results. Evaluation
ofQuery
adjunct instructors would be completed by mail. An SASE envelope would be included to help insure a high
probability of response. Students
would complete their technology questionnaires evaluations at
the same time that they would complete their course evaluation forms. evaluate
their classes. Another component that we feel is necessary is to conduct
face-to-face interviews, using the Institutional Director of Assessment, with
our 10 centerÕs staff. The
instruments will be paper surveys. We feel that this face-to-face interview method would provide us with the
most accurate assessment of the technology. this
will get our highest rate of response.
There will be several evaluation methods
used to gather data from administrators, staff, instructors, students, college
center personnel and computer audits; questionnaires; group interviews; and by
viewing computer logs and application programs loaded on center computers.
Information would be gathered in a short period of time to produce a brief Òsnap shotÓ of how technology is viewed at OLC.
Pertinent survey questions will be asked
of administrators, staff, instructors and students. The results from this paper survey will be manually tabulated
and categorized using standard statistical methods and the results will be
disseminated on the colleges website via user-friendly tables and charts for
all stakeholders to review.
Information Technology Internship
students will conduct computer audits.
They will audit a random sample of 33% of each center computers paying
attention to computer logs as well as the name and type of application program
that has been loaded onto each of the audited machine. A listing of the different names and
types of application software will be added to the website with provisions for
instructors to request the application software to be loaded onto machines at
specific college centers. good
The Assessment Office at each of the
college centers will perform focus group interviews. The Assessment Office will ask probing questions regarding
the technology available to each college center and how each center feels that
the technology is being utilized.
Results from the interviews will be tabulated into list form for inclusion
on the website.
The following chart depicts the methods
of data collection and data analysis interpretation as they relate to each
question..
Evaluation Questions |
Why the Question Is Important |
Data Sources
|
When and How the Information Will Be Collected |
Analysis and Interpretation |
|
1. In what ways is technology used for instruction and
student/staff productivity? |
á
Discover ways that technology
is used that we were not aware of. |
á
Students á
Staff
á
Lesson plans á
Computer lab logs |
á
á
Survey from students
during student course evaluation. á
á
Randomly audit
college center computers for installation of applications beyond original
configurations suite. |
Categorize and tabulate
participantsÕ responses. Group
results that have reoccurred in different respondent queries together. Results will be placed in a
productivity table. Placement in
the table will occur based on the frequency of it occurrence in different
surveyed groups. If a technology
is identified in each of the query methods, it will be rated higher in the
table. We will be able to determine
which technologies are deemed more valuable for productivity based on where
that technology is ranked in the productivity table. |
|
2. What is the availability
of technical support and how does it meet your needs? |
á
To ensure that the lack
of technical support does not hampers the educational process. |
á
Results from Track It á
College center
maintenance logs á
Instructor survey á
College center focus
groups. |
á
The data would be
collected using qualitative and quantitative surveys with instructional
staff. á
Same survey will be
completed during focus group evaluations of college center staff. á
All calls for service
in whatever form they arrive will be entered into the Track It program and a
maintenance ticket will be opened on them. Track It will be queried for summation of service calls
for duration of service tickets. á
Collect and summarize
maintenance logs from college center, monitors computer service and distance
learning service work on-site. |
The results of the service
request will be grouped and sorted in a table. The table will categorize results based on length on time
that calls take to get resolved, method of service request, commonality of
service request, location of service needed. We will also summarize what technicians are responding to
calls and what technicians, if different from those that first responded, are
fixing certain types of problems.
The table will help us to identify commonalities of problems with
technology at OLC, how long problems are taking to get repaired, and what
methods are preferred to request services. We will be able to determine training needs of technicians
based on what technicians are not responding to calls based on familiarity
with technology and will be able to prioritize and technicians to handle
different technology needs in different locations. We can also determine if we need to do more for
disseminating of methods to request technical assistance and how we can
reduce the amount of time, if necessary, that is needed to complete service
calls. |
|
3. How can the stability and
functionality of our overall technology delivery system be improved? |
á
This will help us to
improve the technology of the institution. |
á
All stakeholders |
á
á
Survey from students
during student course evaluation. á
|
Results of this open-ended question would go under
two levels of analysis. The
first level will sort comments by relevant themes. The second level will consist of a table will be
constructed sowing the frequency and percentage of comments attributed to
these themes. Comments from each
theme will then be summarized into a written report in ways that illuminate
the perceived functionality and stability the technology has. |
|
4. What has been the quality
and utility of the technology training that you have received? |
á
Are the current
trainings being offered helpful, do we need to offer other types of
trainings. á
Is the content of the
training being deployed in their use of technology. |
á
Instructors |
á
Survey of instructors
at All-Staff. |
The results will be
tabulated based on trainings that are identify by instructors and which of
those trainings are identified as useful. Tabled results will show what trainings are being taken
advantage (needs of instructors), what forms of training are considered
helpful, what trainings are deemed valuable and which trainings need to be
reconstructed to better suit the needs of the instructors. A narrative summary will be
constructed from suggestions as to how to improve trainings that are not
identified as valuable. |
|
5. Is the current technology
asset list sufficient to meet the instructional and administrative needs of
the college? If not, what is needed and why? |
á
From the perspective of
the stakeholders is there sufficient technology in place (hardware, software,
and web ware) to provide enough resources to meet their needs and the mission
of OLC. |
á
All stakeholders á
9-12 students from Pine
Ridge Reservation high schools. |
á
Interview stakeholders
during All-Staff meetings á
Interview 9-12 students
from high schools during Lakota Nation Winter Festival and college
recruitment days at local Pine Reservation high schools. |
Results of this open-ended
question would go under two levels of analysis. The first level will sort comments by relevant
themes. The second level will
consist of a table will be constructed sowing the frequency and percentage of
comments attributed to these themes.
Comments from each theme will then be summarized into a written report
in summarizing what new technology is needed and currently not available at
OLC for use. |
The evaluation process will occur over the Christmas break time frame (December-January). The surveys for faculty can be distributed during the December all staff meeting and the student surveys can be distributed during the fall course evaluation time period, which also occurs in mid to late November time frame. The focus groups will be facilitated by the Institutional Assessment director, but will be completed during the Christmas break time frame. The college center staff has more flexibility in time as their duties are reduced once the semester has been completed. The random hardware audits will be completed during the Christmas break as well. We do scheduled sweeps of each college center during the semester break as well.
The results of the evaluation will be compiled during the mid-January and early-February time frame. The summarized finding will be posted on the college website and emailed in mass to all the faculty and staff. A hardcopy of the report will also be placed in each college center facility.
At the March
All-Staff meeting will make a formal presentation of the results and will ask
for more recommendations of other areas that the faculty and staff would like
to see collected. Recommendation
and findings that were concluded from the evaluation will be implemented in a
fashion to reflect current practices and to change current practices that are
not as valuable as viewed by the stakeholders. New trainings and updates to technology as recommended by
the stakeholders will be included in an institutional technology plan that will
be presented to the Board of Trustees of OLC. In presenting we will identify budgetary commitments and
policy needs that will be sent through the committee system at OLC for
implementation. The evaluation
will occur again in the fall semester each successive year and will be refined
as the needs change and technology evolves based on the findings.
I like the short time-frame of this evaluation. It can be clean, quick, and provide needed information in a timely manner.
The purpose of this evaluand is to see what types of software have been added to random sampled computer workstations. If instructors are adding software to only a few workstations, it may be due to limits in the number of licenses available.
Center
Computer Serial Number
O/S
O/S Version
Is the basic Office Suite installed? Yes No
Is Publisher installed Yes No
Other software installed:
The purpose of this evaluand is to see if the hardware that is available is being utilized as intended. This will be determined by looking at the log associated with the hardware.
Center
Hardware Serial Number
OR
MIE/OLC Serial Number
Hardware type
Amount of UseageUsage Little Average Above
average
1 2 3 4 5
Would information on the
quality of performance of the technology components, and/or itÕs utility given
the instructional goals of the institution be useful?
Complaints about system:
Team,
The
strength of
this design is that you are very explicit with your data analysis
procedures. You also have very
strong pre matrix components. I
like the writing you did there and I found little if any instances where I though you were
elaborating on trivial events or issues. KudoÕs on the level of
writing.
Unfortunately, you go a
little sloppy, in the handling of the document and making revisions. ThatÕs probably a casualty of
organizing the team work and making each other accountable.
Overall, good job. YouÕve applied the basics of evaluation planning with good
accuracy. With some well placed revisions, this is work that can provide some quality
outcomes for you at OLC. It has the structure and definition to be a good
guiding document for
evaluating the technology support
program.
Your score 110/125