Evaluate Implementation

Review the implementation. After your implementation has been up and running for a defined time period, evaluate the system and make any adjustments. During the operational planning phase, you decided what would constitute success for your repository and you developed an evaluation plan. Now that the repository has been implemented, you can begin to review the pilot deployment and identify successes and weaknesses in the implementation. If your project team does not include an experienced evaluator, you may wish to contract with one for advice and/or assistance with conducting the evaluation. Consider the stakeholder groups that you may need to satisfy with your evaluation data, as well as any current political priorities that your project data might support. Especially if your results will be highly visible and/or critical for future funding, ensure you have followed rigorous procedures for your evaluation and analysis, so that everyone can have confidence in the results. Another alternative is to hire an external evaluator or locate one from another government agency, to ensure confidence in the study results.

Gather data. As suggested in the planning phase, review all the data you collected while monitoring the implementation (e.g. user feedback, system reports, service desk contacts, usage statistics) and assess these opinions and outcomes. Examine feedback from participants in your training sessions (face-to-face, online, and recorded) regarding their opinions and suggestions for improvement. Your evaluation might include following up your first analysis of data with focus groups or interviews to gather more in-depth information and insights into your data. Gather input from your strategic planning partners and other non-user stakeholders.

Analyze your data. This is an area where multiple perspectives can help you dig deeper, and debate the meaning of the data. For example, a large number of users can indicate that many people are interested in the repository. By comparing this information with visits to the repository, you may find more useful information.If you have perhaps 1000 registered users, but only 10 visitors to the repository a day, this likely indicates that users are not making regular use of the repository. Lots of people may have registered for accounts in the excitement surrounding the launch, but they may not have followed up and made use of the repository resources in their teaching. Or, it could indicate that users are downloading resources for use, in which case you may not have the ability to track usage directly. Surveys or interviews could help here.

Review user response. Similarly, you may want to look at how users who are visiting the repository are interacting with it. Are they simply searching for objects or are they downloading or linking to them as well? A user who visits the repository but does not use the resources might not be finding what they are looking for. Which items are getting the most hits/downloads? It might be beneficial to find out what makes these resources so attractive. You may want to seek out similar types of item to add to the repository. Are users contributing items to the repository? Why or why not? Following up on these questions could yield valuable information for future improvements.

Report on your evaluation, especially if you have positive results. You may want to pull supporting data or statistics and incorporate that information into your marketing for current/future implementations