Phoebe Phase 2 Evaluation Plan
Phoebe Phase 2 Evaluation Plan
- Document history
- Research questions for the Phase 2 evaluation
- Specific questions to ask (stemming from the research questions)
- Evaluators for Phase 2
- Evaluation methods
- Phase 2 evaluation schedule
- Version 1: 03/10/07
- Minor revisions: 10/01/08, 18/03/08
Research questions for the Phase 2 evaluation
As in Phase 1 we are evaluating Phoebe as a proof-of-concept tool; however, this time we are testing it in more authentic environments, in order to address the following questions (adapted from Phase 1):
I Is Phoebe a tool that practitioners in post-compulsory education find usable, helpful and relevant to their needs, whether they are
a) beginning or experienced teachers looking to use technology in their teaching for the first time, or
b) are familiar with e-learning but are looking for new ideas re technology and pedagogy?
Specifically, does it encourage practitioners to think about their practice in a structured way?
II Can one tool address all sectors of post-16 education, or are separate tools required for, say ACL, WBL, FE versus HE?
III Is Phoebe suitable as a tool for teacher education and/or a tool for supporting everyday practice?
IV Is there a perceived need for Phoebe to be customised to meet the needs of individual organisations (i.e. to function as a community-owned artefact)?
Specific questions to ask (stemming from the research questions)
1. Key aspects of usability
1.1 Ease of learning to use
1.1.1 What "getting started" guidance is needed?
1.2 Ease of use once learned
1.2.1 Does Phoebe work? 1.2.2 What does Phoebe do well? 1.2.3What does Phoebe do badly?
1.3 Productivity/usefulness/effectiveness (task accomplishment) (also covered in practice-related issues below)
1.4 Affective response (enjoy using it/choose to use it/recommend to others)
1.5 Sustainability? (Not just new features, but also projections re effort likely to be involved in maintaining currency of content and examples.)
2. Integration into individual pedagogic practice
2.1 Mapping to their established practice (processes, functionality)
2.2 Usefulness at different levels: course/scheme of work planning vs topic planning vs session planning
2.3 To what extent does Phoebe encourage them to think about their practice in a structured way? If yes, why and how? If no, why not?
2.4 Relevance to their established practice (content) (focus on generic vs examples vs institution-specific examples)
2.5 Extent to which they feel they have learned something new/been inspired to experiment/innovate in their own practice
2.6 Relationship between their use of Phoebe and their continued use of existing tools: Do practitioners combine the use of Phoebe's planning functionality with their familiar tools?
2.7 Would practitioners want to use Phoebe in their normal work?
2.7.1 If yes, what role would Phoebe play? 2.7.2 If no, why not and what, if anything, would induce them to use it again?
2.8 Do they consider Phoebe to be useful to others? At what stage of career/at what level of experience wrt e-learning?
3. Integration into pedagogic practice of the institution (community dimension)
3.1 Staff trainers: does it map to the models of pedagogy which they are trying to communicate to their trainees?
3.1.1 If no, how much effort is required to make it map? Could simple customisation options (e.g. ability to edit "default" guidance, add own case studies and examples, components of plan, terminology?) work, or would they need a totally customised tool?
3.2 Can the tool be used by practitioners working alone or does it work best in an institutional setting, as a community artefact?
3.3 To what extent does Phoebe facilitate or, conversely, impede established practice relating to session planning in the user's institution?
Evaluators for Phase 2
- E-learning experts:
- Teaching staff who are already familiar with the use of technology in their teaching/learning
- Staff developers with responsibility for promoting e-learning and tutors in ITT who are actively promoting
- IT support officers and learning technologists who are responsible for disseminating the use of technology in their institution/department and/or are involved in specific 'projects' (e.g. creating an image bank; redesigning a F2F course to run in a blended or fully online environment)
- E-learning novices:
- Teachers undergoing CPD aimed at introducing them to e-learning
- Teaching staff who are new to technology but not participating in a CPD e-learning programme
- Students undergoing ITT (NB they may or may not be experienced in using digital technologies)
Possible sources of evaluators:
NB This is just a list of 'possibles': none of these groups has yet been approached.
- Members of JISC Pedagogy Experts group
- Selected participants from the Learning Design Tools project (mailing list is still open)
- PIs and/or their colleagues (esp. where originally approached to take part in 'embedding' workshops)
- Serendipitous contacts, including those who find out about Phoebe through third-party connections or Web searches (see also "Friends of Phoebe" and list of people who contacted us for user accounts in Phase 1)
- Workshop participants in the University of Greenwich staff conference September 2007 who attended a demonstration of Phoebe and expressed interest in another workshop
We envisage recruiting only a small minority of evaluators from the D4L programme itself as we feel that the evaluation will be more fruitful (and objective?) if evaluators are from outside the D4L "community."
We will use methods that focus on capturing a rich set of qualitative data from a small group of (experienced) people.
The proposed methods are shown in the following table:
|Instrument||Subjects||Purpose||Qual/Quant? data?||Notes & issues|
|Heuristic evaluation||Project team||Optimise usability of interface so as to eradicate unwanted factors in evaluation of pedagogical aspects of tools.||Qual + Quant||Base on user-centred and learner-centred principles; modify, extend and/or replace the Nielsen and Sharples & Beale guidelines. (See other lit on UI design in e-learning, incl. Hedberg et al. paper.)|
|Observation, optionally with recorded think-alouds or short semi-structured interviews||Evaluators||Affective responses to experience; Usability problems||Qual||Feed into case studies|
|Semi-structured group discussions||Evaluators||Affective responses to experience (reflective)||Qual||Feed into case studies|
|Session plans created during workshops?||Evaluators||Capture examples of output - components used in different plans, structures (ordering of components)||Qual||Feed into case studies|
|Questionnaires||Evaluators||Affective responses to Phoebe; other attitudinal data||Qual + Quant||Draw for inspiration from questionnaires written for LAMS evaluation, LD Tools project; feed into case studies?|
|Webserver request logs||Evaluators||Investigate users' behaviour re the planning and reference aspects of Phoebe||Quant||From Dave B. 28/08/07: "with the guidance and functionality running as independent websites, correlating users in their logs will be necessary if we wish to examine paterns connecting the two parts of Phoebe."|
Phase 2 evaluation schedule
The Phase 2 project plan identifies five key 'events' in the evaluation schedule: workshops involving trainee teachers, in-service training (CPD), a workshop for experienced e-learners, and workshops to evaluate the customisation features of Phoebe and its support for collaborative planning. On reflection, some of these can be combined (e.g. experienced e-learners and customisation) or may prove infeasible to run as "live" workshops (e.g. collaborative planning). On consultation with Glenaffric, we now also propose to capture feedback from practitioners who ask us for a Phoebe account (30+ in Phase 1) through an online survey. However, we have yet to ascertain the likely value of such data in relation to the "live" face-to-face events.
|Timing||Event||Factor to Evaluate||Questions to Address||Method(s)||Measure of Success|
|Oct-Nov 2007||Internal usability tests||UI and functionality of prototype tool||Consistency, usability, error-free functioning||Walkthrough by project team members||Number of issues identified and resolved|
|31 Oct 2008||Trainee teachers' workshop (HE), University of Greenwich||Usability; embedding in ITT; suitability for novice teachers||Usability; value of structured support for the planning process; value of the reference materials in inspiring thinking||Observation, discussion, session plans, Web request logs||Levels of satisfaction and acceptability expressed by participants; extent to which the tool is judged to be of use to trainees|
|16 Nov 2007; 28-30 Jan 2008||In-service training: staff development workshops at Oxford Brookes University||Embedding in CPD; usability and usefulness of tool||Usability; integration into personal pedagogic practice; integration into institutional practice||Observation, discussion, session plans, questionnaires (? to capture current practice?), Web request logs||Levels of satisfaction and acceptability expressed by participants; extent to which the tool is judged to be of use to individuals (esp. in helping them to engage with e-learning and to plan in a structured way) and the community|
|14 Jan 2008||Experienced practitioners' workshop held at Oxford University Dept of Continuing Education; attended by practitioners in FE, HE, ACL, WBL||Sustainability; suitability for experienced practitioners; customisation; support for collaborative planning(?)||Integration into personal pedagogic practice; integration into institutional practice||Observation, discussion, session plans, Web request logs||Levels of satisfaction and acceptability expressed by participants; knowledge about the ways in which practitioners might want to customise Phoebe; input on the additional functionality needed to support customisation at an acceptable level; ?input into additional functionality needed to support collaborative planning (incl. enhanced understanding of the ways in which practitioners plan collaboratively)|
|29 Jan 2008||CPD workshop, University of Brighton (led by Oxford Brookes University)||Sustainability; suitability for experienced practitioners; customisation; support for collaborative planning(?)||Integration into personal pedagogic practice; integration into institutional practice||Observation, discussion, session plans, Web request logs||Levels of satisfaction and acceptability expressed by participants; knowledge about the ways in which practitioners might want to customise Phoebe; input on the additional functionality needed to support customisation at an acceptable level; ?input into additional functionality needed to support collaborative planning (incl. enhanced understanding of the ways in which practitioners plan collaboratively)|
|6 Feb 2008||Trainee teachers' workshop (FE), Swansea College||Usability; embedding in ITT; suitability for novice teachers||Usability; value of structured support for the planning process; value of the reference materials in inspiring thinking||Observation, discussion, session plans, Web request logs||Levels of satisfaction and acceptability expressed by participants; extent to which the tool is judged to be of use to trainees|
|4 Mar 2008||Embedding and sustainability||Suitability for embedding in staff development and/or initial teacher-training context; suitability across sectors and domains||Strategic review meeting with representatives from HEA, Becta, ACLearn (joint event with the London Pedagogy Planner project; organised by JISC)||Surveys||Quality of feedback re suitability|
|Feb-Mar 2008||Online evaluation by interested (mainly experienced?) users||Usability; usefulness||Integration into personal pedagogic practice||Questionnaires, Web request logs(?)||Positive feedback on tool + constructive suggestions for future directions (both functional and in terms of deployment)|