Standard 5

Candidates demonstrate knowledge, skills, and dispositions to evaluate the adequacy of instruction and learning by applying principles of problem analysis, criterion-referenced measurement, formative and summative evaluation, and long-range planning.

5.1 Problem Analysis
Problem analysis involves determining the nature and parameters of the problem by using information-gathering and decision-making strategies.

Artifact 1: Murtagh Model of Technology Integration and Instructional Design
Course: Edtech 503 - Instructional Design
Format: Scribd Document

Artifact 2: Request For Proposal: Evaluation of Determining Instructional Purposes (DIP) training program
Course: Edtech 505 - Evaluation for Educational Technologists
Format: Scribd Document

I created my Franken model, also known as the Murtagh Model of Technology Integration and Instructional Design, for Edtech 503 to illustrate the instructional design process that I use with students and teachers. The circular design includes observation, collaboration, implementation, support and evaluation stages and emphasizes the idea that the design process never really ends, but rather evolves on a continuum. Feedback constantly drives design changes. I use this model frequently in my activities, and have found that by displaying the graphic on my wall, the people I am working with get a better appreciation and understanding of the process that I take them through. I have also found that after they complete a project with me using a formal design approach, that they often try to replicate the process in their own activities.

A second artifact that demonstrates mastery of problem analysis is the response for proposal (RFP) evaluation (Determining Instructional Purposes (DIP) training program) that I completed for Edtech 505. For this project, Far West Laboratory for Education Research and Development (FWL) issued a request for solutions to a help them determine the market availability of their training program. Acting as a senior research analyst, I had to analyze their problem, devise a unique solution, assemble a team, and then present a compelling argument as to why my proposal should be chosen amongst competition from my peers. Boulmetis and Dutwin (2011) make an emphatic point that when it comes to evaluation that “Your first task - not your last - is to ask who is interested in the results of this evaluations” (p.17). When it comes to problem analysis, I feel you can generalize this quote and apply it to more than just the evaluation process. As an instructional designer, I need to identify who is this project for, and then work with them to determine what they need. The emphasis of making this first step in the process has helped me shorten delivery times and also made for better projects with fewer revisions.   

5.2 Criterion-Referenced Measurement
Criterion-referenced measurement involves techniques for determining learner mastery of pre-specified content.

Artifact 1: Getting Started with Gmail Online Course (video overview of Canvas course)
Course: Edtech 512 - Online Course Design
Format: Video Presentation

Artifact 2: Getting Started with Gmail Online Course (website with links to Canvas course)
Course: Edtech 512 - Online Course Design
Format: Website

For my Edtech 512 class, I created an online course using Canvas to support our district’s email migration to the Google Gmail product. Because the Canvas platform requires students to register for courses, there are links to two separate artifacts. The first artifact is a video overview of the course that demonstrates how the course is designed and functions without having to login. The second artifact is the website that provides an introduction to this online course and a link with registration details.

For the start of the 2014-15 academic year, our district needed to get several hundred staff and faculty members fully operational on the Gmail platform. Our technology department does not have the resources to directly support this type of demand, so an online solution made great sense to offload some of the work. This course allows learners to proceed at their own pace to acquire the necessary skills to conduct professional electronic correspondence with Gmail. Essential components built into the Gmail course are assessment activities designed to give users feedback on their understanding of core concepts. Canvas provides basic quiz type functionality to create online assessments which can be used to measure discrete knowledge. Since a critical Gmail skill users need is the ability to create and reply to email, I had to go beyond the basic true/false, multiple choice, and short answer question types in Canvas. By creating fictitious email accounts on our district server, I was able to setup email addresses that will auto-reply a code word to the sender. The learner in my course is asked to send an email to a designated address, and if done correctly, they will receive an email reply with a code word, which is then used to answer a fill in the blank question in a Canvas quiz. This enhanced quiz functionality allows the learner to assess that they have mastered the two most essential skills required to conduct email correspondence. The course has proven to be a tremendous resource for the district’s faculty and staff, and it is expected that it will be used again when we offer email accounts to our students throughout this upcoming year.

5.3 Formative and Summative Evaluation
Formative evaluation involves gathering information on adequacy and using this information as a basis for further development. Summative evaluation involves gathering information on adequacy and using this information to make decisions about utilization.

Artifact: Getting Started with Gmail Design Document
Course: Edtech 512 - Online Course Design
Format: Scribd Document

The design document created for Edtech 512’s Gmail 101 online course is an excellent artifact documenting both the formative and the summative evaluation process. As part of the development stage of my online course, I conducted a formative evaluation using a Google survey to ascertain whether learners can independently access their GPS Gmail account from work and at home, use Gmail to engage in professional / academic electronic communication, and customize Gmail for their personal preferences. The survey consisted of 15 questions with a 5 point Likert scale. Individuals representing all the major stakeholder groups were asked to participate. The results of the evaluation were summarized and the online course was modified to enhance those areas which needed improvement. The formative feedback confirmed that certain aspects of the course were very effective, but also identified some weaknesses around taking the course on iPads.

The task of recruiting subjects to take the survey and explore the course design was challenging. I had to rely on the good nature of co-workers whose time is at a premium. At first I felt that the formative review process was an unnecessary burden, but after compiling the results, and talking with the volunteers informally, it turns out that the process was absolutely necessary and their feedback improved the overall project.  I also had to craft a plan for a summative evaluation for the online course, even though we did not have to actually complete the evaluation for the course. Based on the success of the formative review, I am convinced that the summative will yield equally good results. The plan for the summative was to observe faculty who took the course as they used Gmail when they returned to school, and then administer a comprehensive survey to everyone that participated in the course in late Fall. As a result of creating the two evaluations, I gained new insight into the art of building effective surveys. To collect meaningful data, you must begin with worthwhile questions, and then have a scale that lets you interpret the results. When I look at surveys now, I find myself examining the questions and the answers, to see if they are designed well.

5.4 Long-Range Planning
Long-range planning that focuses on the organization as a whole is strategic planning. Long-range is usually defined as a future period of about three to five years or longer. During strategic planning, managers are trying to decide in the present what must be done to ensure organizational success in the future.

Artifact: Networking Analysis / Wireless Upgrade
Course: Edtech 541 - Integrating Technology into Classroom Curriculum
Format: Video / Slideshow

In Edtech 541, I analyzed the wireless network infrastructure for one of our district’s middle schools. The results were summarized in a video presentation describing the school’s then current network configuration, as well as a proposal for the implementation of a new 802.11ac standard wireless network. By upgrading our 802.11b/g network to the 802.11ac standard, the district expects to gain a twenty-fold increase in bandwidth. It is estimated that the proposed system will satisfy wifi needs for the next five years if technology adoption continues at the current rate. To validate the networking hardware, a small scale pilot of the wireless network was conducted last spring during our Smarter Balanced testing which required hundreds of students to simultaneously access the online assessment over wireless connections. The pilot results exceeded expectations, and over the summer of 2014, two schools in our district began the wireless upgrade process. My proposal for the revised network included floor plans and product descriptions that eventually became part of our district’s bidding process. It is anticipated that the remaining five schools in our district will also implement this wireless upgrade over the next two years.