Assessing Impact of an Online Training

One of the challenges of online training is that it can feel extra difficult to assess how people are feeling, what they are learning, and generally what impact your training has had.  At the same time, training online offers some options to easily generate data you can use to help you analyze your impact, including chat transcripts, polling data, participant data, etc. Using the data online training tools offer you becomes especially helpful when you want to measure the impact of large online trainings.  

MAKING AN IMPACT ASSESSMENT PLAN

It takes a lot of time and effort to design and lead online trainings, so it’s worth taking some time to evaluate what impact you are making towards your goal. For the most helpful assessment, don’t wait until after your training to think about how you will evaluate it!

In making an impact assessment plan, start like you would when designing the training, by getting clear on the goals of the workshop:

• What is the organizing ASK for your training? (What you want people to do after the training.) 
• What are your impact goals? (The Knowledge, Skills, and Attitudes they will need to do your ASK.) 
• What other goals do you have for the workshop, for example, do you have a fundraising goal? A goal of  ensuring the participation of a particular audience?

These are the goals to focus on in your impact plan. For each goal, ask how you will be able to assess it. We encourage including engagements in your workshop that are specifically designed to help you get a sense of where your participants are in light of your objectives.  After the workshop, you can use the results of these activities (in chat transcript, in scribed notes, in poll results) in your assessment. If you use a post-training evaluation form, it is important to also think through about the questions to include in it. Remember that having many people “like” your training does not give you much information about its impact.

TIP: During your training, if you paste each activity prompt in the chat box with a row of characters (like an equal sign) at the start of that activity, it makes it much easier to analyze the chat transcript later. For example:

===========

Share in chat: How helpful is this example on a scale of 1 to 5?

===========

A review of your gathered statistics can be part of your post-workshop debrief and also make it easier to compare two sessions of the same workshop. Did changes you made to the design increase the impact?

See below for some examples and tips that may help as you build and implement your plan.

EXAMPLE STATISTICS

Here are a few examples of the kinds of statistics we have used in the past to help us analyze an online training’s impact.

TECH TIP FOR CHAT ANALYSIS

To make the above chat analysis easier, especially for really large trainings, it can be helpful to convert a text chat transcript into a spreadsheet.  Here is an example of how to do this with a Zoom chat transcript and Microsoft Office tools:

1. Open your chat transcript document
2. Remove any private chats. Tip: search “Direct”
3. Copy and paste the chat transcript into a Word document (for the next step to work, this must be Word, not any text editor)
   
4. In the Word toolbar, go to Insert -> Table -> Convert Text to Table.  Choose the option to separate text at Tabs. The result will be a table with two columns, the first column is the time stamp, the second column starts with From <name>
5. Copy and paste your new table into a spreadsheet like Google sheets, Excel, etc. 

Now that you have this two column spreadsheet, you can do things like:

• Sort the whole spreadsheet by column 2 (name) and hide all the chats from people you don’t want to include in your analysis (for example, staff)
• Sort the whole spreadsheet by column 2 to easily count how many different people participated or how many times each person participated
• Sort the whole spreadsheet by column 1 (time stamp) to count how many answers or analyze the answers for a specific prompt

TECH TIP FOR PARTICIPATION ANALYSIS

Zoom accounts include various types of reports that can give you useful information. They include registration, attendee (which includes the number of minutes people stayed connected to the call), and poll reports. You can access a tutorial about this feature at: https://support.zoom.us/hc/en-us/articles/201363213-Getting-started-with-reports 

If you are using a platform other than Zoom, check out what analytics features it offers.