User:Leighblackall/Identifying Success in Learning Management System Changeover

This is a critical review of a change in learning management system at a large Australian University, covering the impact on workload and professional development activities in the university. It makes recommendations from the perspective of an educational designer working with teaching and administration staff in the organisation.

From initial consultation through to deployment, the transition has taken twenty four months to date, with significant resources invested. While the merit of the educational and business case for the change was contested at the time, the executive decision to proceed was said to be based on a benefit to the end-users (teachers and students) who needed to become “ready for life and work”.

With a seventy percent casualised workforce, a high turnover of staff and everyone having work plans already accounted for, the workload and professional development needs of teachers and their support staff was underestimated. A quality assurance process was effective in driving professional development, but better criteria to help find more efficient workflow for teachers and to allow for innovation and align development in teaching practice is needed.

Concluding are a number of recommendations for how professional development may continue after the transition, namely a better understanding of staff workload, a review of the quality assurance process, an introduction of peer observation and the development of flexible learning activities that recognise the realities of a casualised workforce.

Keywords: Learning Management Systems, Canvas, Professional Development

Identifying Success in Learning Management System Changeover: Implications for professional development
Leigh Blackall

Preface
“For instance, Thamus warns that the pupils of Theuth will develop an undeserved reputation for wisdom. He means to say that those who cultivate competence in the use of a new technology become an elite group that are granted undeserved authority and prestige by those who have no such competence.”

“Harold Innis, the father of modern communication studies, repeatedly spoke of the "knowledge monopolies" created by important technologies. He meant precisely what Thamus had in mind: those who have control over the workings of a particular technology accumulate power and inevitably form a kind of conspiracy against those who have no access to the specialized knowledge made available by the technology.“

(Postman, 1993)

Reason for writing
In my role as an Educational Designer in Digital Learning for the College of Design and Social Context at RMIT University, I have observed and been part of a recent, university wide change in Information and Communications Technology affecting teaching and learning. With a major stage in that change now coming to an end, I have been asked to help design and manage professional development in the College, centred on “digital learning” curriculum design, and relating to a number of focus areas coming from the University’s Strategic Plan. And so it seems timely to review the impact that the recent change has had on professional development, and begin conceptualising a renewed effort for professional development in the College.

Context and Change at RMIT
RMIT University is a large Melbourne-based university with a significant presence in Ho Chi Minh City and a small presence in Barcelona. It has 10425 staff and 87465 students, and of the 10425 staff only 3237 are employed on a continuing basis (Annual Report p31, 2017). This means the vast majority of employees at RMIT are casually employed, or on fixed term contracts. A casualised workforce like this presents significant challenges for improving the quality of teaching practice through professional development (Watts 2017, 2015; Klopper 2014; Percy 2008).

The Melbourne-based university is made up of Three Colleges, each with a number of Schools or Discipline Areas. Within the Schools and Discipline Areas are Programs of study from short courses and certificates through to undergraduate and postgraduate degrees. The Programs are managed by a Program Manager, and the courses/studios/practical placements or work integrated learning activities within the program are managed by a Course Coordinator. Both Program Managers and Course Coordinators are typically teachers in the program, sometimes casually employed, and are primarily supported by administrative staff in the School, professional development staff in the College, and infrastructural support staff in the university. Research is conducted individually and/or within university research focus areas, and/or within funded projects.

Professional Development and infrastructure resources are allocated centrally in the university, as well as at the College level. It is rare that a School, Discipline Area or Program has its own professional development resources, apart from occasional funds for consultants and events. Centrally, it is Human Resources and the Education Portfolio - under the direction of the Deputy Vice Chancellor, Education, where the university-wide directions for education are planned and managed, with the College resources directed to support the projects of the Education Portfolio where they touch the Schools and Programs in the College.

Introducing a new Learning Management System
The Learning Management System was changed from Blackboard to Canvas and not without dissent (Blackall, Errey, Smithers 2017). The process of gathering requirements from stakeholders took longer than initially expected - This process had dubious merit, probably due to a high turnover of staff managing the process. For example, a prominently desired feature requirement was that the system be fully featured and optimally usable within a mobile browser. Blackboard, Desire2Learn and Canvas were presented as the only systems to be considered against the listed requirements (for reasons unknown to me), and when Canvas was quickly selected, it was clear that it was not fully featured and optimally usable in a mobile browser. It would require the use of mobile applications instead (Canvas Doc Team, 2017).

There have been structural changes affecting the groups that normally manage and support digital infrastructure, media production and professional development. A series of restructures at the same time as the review of Learning Management Systems dramatically changed the staffing and services in Information Technology Services. Another restructure saw the closure of the centrally located Academic Development Group that included a team of educational media producers and technology trainers. Another restructure saw the centralisation of Library services, and another saw Marketing centralised, closing the College based groups who assisted with communications, events and media production around Programs. These restructures have generated a strong sense of precarity among members of the remaining support groups.

Canvas momentum generated
A considerable amount of momentum was generated for Canvas. There was the review and ultimate change of the Learning Management System; the establishment of a new, centrally based management and support group for the transition to Canvas; the introduction of a Quality Assurance process for all courses (see Quality Assurance); an aligned organisational hierarchy using ‘data dashboards’ to survey that process and; the College resources directed to support the transition from Blackboard to Canvas. Implicit with this momentum was that use of the Learning Management System would not be optional.

Impact of support staff
The groups and roles that would normally manage and support ICTs in education underwent significant changes in the overall change plan. Before Canvas, there was a relatively small Teaching and Learning group within the university’s central management who relied on College based academic developers and learning designers to support teaching and learning. To a large degree, those College groups played a leadership role in initiating projects, advising the central group and managing change upwards. With a new Vice Chancellor and a Deputy in Education who, both having worked together at Open University UK, there appeared to be a firm idea of what was to happen at RMIT, citing a sort of mandate from a public consultation called #shapeRMIT that had a very low response rate (Blackall 2016).

With the formation of new and agile project teams, a large number of roles and groups in the older structure of the university were made redundant. Of course, in those roles were people with relations, experience and organisational knowledge. Soon after that restructure, the new project teams became larger and a new group was established. It became clear that this new group was going to play the main role in the change plan ahead.

The College-based academic development groups would continue to work in with the new central group, but not as before. The direction of change became very top down or centre out - from the centre, out to the teachers, not always via the College groups. This generated a certain level of tension between the College groups and the Central group, typically found in most universities, with some anxiety in the College groups that restructure and redundancy was soon to affect their roles as well.

Impact of teaching staff
These changes took place over a period of 2 years and obviously had an impact on the workload and professional development of teaching staff, both full time and casually employed, as well as the staff who support the work of teachers.

Before Canvas, most teachers used Blackboard to manage the content, curriculum and assessment of their courses and programs. Some used websites on Wordpress and Google Sites. A few did not develop any online access or activity points at all (Errey 2015). Use of the Learning Management System was not, and still is not, required in the Course and Program Policy, or the Assessment Policy. For everyone, learning to use Canvas and adapting to its design premises, was more work than usual, and of an unknown quantity.

Within the momentum generated for the change, there was a strongly implied expectation that every course would now use Canvas (see Momentum generated), the teachers who had invested in Blackboard or who had established alternative ways to offer a course website, along with those who had not so far used any online device to support their teaching, faced a rapid transition to Canvas. With their work plans and workloads already set, they needed to find an unspecified amount of time to learn Canvas, attend Canvas meetings, workshops, dropins, ring hotlines and follow online self help resources. Casually employed staff were eventually given either one and a half or three hours pay to attend a workshop (depending on their teaching role), after it was advised by the College that it would take at least twenty hours to become competent in the use of Canvas.

Presumably to help with teacher workload, the central change management group (by then called Project Rewire) initiated a project called “Shift and Lift” where they outsourced the copying over of course content from Blackboard to Canvas, and applied templates into the copied courses. This did little to reduce teacher workload however, as it copied across out of date files and work needing to be improved or sorted, and some of the templates created complexity, such as tables requiring HTML editing, or graphics and images that would easily distort when edited by a beginner/novice. In addition to the templates, a Quality Assurance process was introduced, requiring teachers to meet a set of standards in their Canvas courses, before their students would be given access to it. The templates helped with some of those standards, but some of the standards duplicated other work (such as Course Guides) so it amounted to more work, sometimes substantially more work.

Even with all that working against the adoption of Canvas, most teachers and course coordinators carried a mood of going with it. One because Canvas offered a more attractive interface than Blackboard, two because “working outside the system” with unsupported tools was perceived as too difficult or unsustainable, and three because most students had by now come to expect some form of online experience across courses.

Data lead
The legacy student administration management system had to be wrangled to interface with Canvas in an attempt to automate much of the course creation and access processes. At the point in which teachers began getting access however, the integrity and management of this data began to reveal its weaknesses and limitations. Many courses did not list the true Course Coordinators, therefore delaying communications, support or professional development opportunities from reaching the right person. Many courses were (unofficially) coordinated by casually employed staff who could not be expected to attend professional development, or commit to extra preparation time without an employment contract that covered that time. Many courses, due to an idiosyncrasy of RMIT’s administration of vocational programs, were in fact units of competency within a vocational program, so offering a Canvas site for each competency within a program of study was often undesirable - Program Canvas sites would be needed instead, something Canvas is not well designed for, and this increased the workload on Program Managers, Course Coordinators and support staff who were asked to attend many meetings to try and develop a solution to the problem.

Quality Assurance
The Quality Assurance process consisted of a list of elements (see below) that become a requirement in every Canvas course site before it was to be made available to students. The Course Coordinators were expected to satisfy all the elements of a list, with help available on websites, in scheduled workshops and regular dropins and ‘Live QA’ - where a Canvas course site could be assessed and rectified (within reason) on the spot. Project Rewire employed “Canvas Champions” to carry out the assessments of course quality to the list, and their assessments were recorded and made available to Program Managers, Associate Deans and up the hierarchy in the form of data dashboards. Senior managers and executives were called upon to give pressure and uncover problem areas when necessary.

The 14 Elements to Canvas Success
 * 1) Course Home Page Information and Banner
 * 2) Introductory Announcement
 * 3) The Course Welcome and Orientation Module is Present
 * 4) Welcome to Course: Video and Course Guides
 * 5) Teaching Team Details
 * 6) Canvas in this Course
 * 7) Course Queries: Discussion Thread
 * 8) Course Schedule
 * 9) Required Navigation
 * 10) Additional Navigation
 * 11) Copyright and Active Links
 * 12) Assessment Tasks: Present and Consistent with Course Guide
 * 13) Assessment Tasks: Essential Information and Submission
 * 14) RMIT Style

Apart from the statement describing the 14 Elements to Canvas Success as, “...a set of compulsory items designed to give students a core, consistent experience when using Canvas…” It is not clear where the 14 Elements came from or how they were arrived at. Most information about the Elements is behind the RMIT staff intranet, and does not reference any research or other information to support the claim that they improve the quality experience.

From my own observations they emerged and evolved from a series of projects and individuals with influence on the project, under intense time pressure and before there was sufficient experience in the organisation to know what a quality experience of Canvas might be. Furthermore, it became apparent that little to no consideration had been given to existing workflows and quality assurance procedures, such as the Course Guide process, that had timelines impacting on a teacher’s ability to complete the Canvas elements, and that was work that then had to be duplicated into Canvas to satisfy the elements. One measure of quality might have been the efficient use of the system.

These weaknesses in the Elements of the Quality Assurance process generated a high level of skepticism and resentment among teachers and support staff, with some teachers electing not to do certain elements on principle, or because they believed they had creatively or efficiently met the criteria elsewhere. Many considered the elements to be non essential, and a distraction from more important work, such as compliance to national governing bodies, better assessment tools, or finding efficient ways to use Canvas. These attitudes were met with stiff resistance from the Canvas Champions who had the unenviable task of assessing Canvas sites for their conformity to the Quality Assurance Elements. Their resistance was naturally inconsistent at times, with some Champions being lenient and very helpful, and others being strict and most unhelpful. This, among other things, deepened the resentment from teachers and support staff.

Quality as a driver of professional development
The introduction of a Quality Assurance process within the momentous way that RMIT implemented Canvas, however, has been a powerful mechanism to drive professional development in the organisation. It accelerated the wide scale adoption of Canvas throughout the hierarchy and focused attention on a clear set of objectives said to be directly relevant to the end users. It is unfortunate though, that the quality of the Elements was compromised by a number of factors, thereby undermining the potential depth of its drive of professional development. In many instances poorly thought out elements propelled inefficiency in the organisation, such as the misalignment and duplication between the Course Guides and Canvas, or the inflexibility of the quality assessment stunting innovative solutions to inefficiency or better ways to use Canvas for differing contexts. A thorough and well considered review of the elements, if it could somehow remain reasonably adaptable, capturing the experience and knowledge base as it grows in the organisation, could still capitalise on the strength of the Quality Assurance process for professional development, but not without a further irritation of teachers who are already hostile to the existence of it.

An obvious way to go about a review of the Quality Assurance Elements would be to review and consolidate the base elements to their minimum core level; align a more holistic workflow to course and program management, and invite Programs to select and propose their own elements for quality assurance on top of the consolidated core.

Understanding Workload
“...implementation of new technologies for online teaching has resulted in poorly defined workload expectations.” (Tynan 2013)

“...When “repurposing” existing content (for 60% or more of the blended course) the development ratio average drops to 22:1.” (Chapman in Tynan 2013)

According to Tynan we should have (and still need to) thoroughly interrogate the workload implications of a transition to ‘blended’ and ‘online learning’ via a learning management system, at least in terms of the 22:1 work:output ratio that “repurposing” has (Tynan 2013). In the context of RMIT, that workload calculation must include the work of learning a new system, adjusting to the restructures, satisfying the new quality assurance process, on top of preexisting and far-from-perfect processes and requirements that remain unaccounted for.

To date, extravagant resources have gone to the formation of a centralised support group at RMIT. This includes a significant amount of energy put into the connection of the LMS to the Student Administration Management System, Results Processing System, reporting to audits, as well as data dashboards for management to survey usage of Canvas and compliance to the Quality Assurance process. Little to no resources have been given directly to teachers however, in the form of workload allocation, or compensation for increased workload for professional development, especially casually employed teachers. For example, in 2018 it was estimated by the College group that 20 hours would be needed by teachers to learn and use Canvas effectively in their courses. Casually employed teachers were given 1.5 - 3 hours compensation, depending if they were tutors or coordinators, and full time teachers were expected to fit it into their existing work plans.

In 2018 a Workload Review Committee was established to address the concerns of Vocational Education teachers in the University. That committee did not produce a report or recommendations, however a Workload Management Plan was initiated in 2019 and their output is eagerly awaited.

Professional development planning and implementation at RMIT might consider how to convincingly empathise with teachers’ common complaints of workload. Doing more to understand the subclasses of work under the general area of teaching, and accounting for workload accordingly. This would be a worthwhile first step toward proper recognition and planning. Any professional development offered ought to be within that workload allocation, not on top of it, and link directly to other areas of work, such as payment, promotion and awards. To do this the university needs to review literature that has been published since Tynan’s work in 2013 (Tynan 2013), and attempt to find or develop a breakdown of teaching work so that ‘blended’ and ‘online’ work expectations can be more realistic. At RMIT, this includes the quality assurance process. Ideally, this would have been done well before changing the LMS.

Managers of this work would benefit from some sort of longitudinal assessment of workload, to query the claims of workload efficiency gains commonly used to justify Learning Management Systems and online education (Rumble 2001, Meyer 2006). To do this, it might be useful to follow Tynan’s methods of interviewing faculty (Tynan 2013), but to target and survey teachers who have been developing digital and online practice in their courses for at least 3 iterations of the same course, where the teachers started from little to no online methods. The expectation of this survey being that efficiencies might be found in the 3rd iteration, after the first 2 iterations being less efficient than face to face offline modes. There would be at least some teachers at RMIT who have had the benefit of a consistent and developmental teaching experience, and so will be able to give some insight on workload implications for the rest of the organisation.

In addition to any work done to define workload, including some sort of participatory action ethnographic research, perhaps in the form of “User Experience or UX Design” if that approach is still trending, to more deeply appreciate the attitudes, approaches, ethics and principles of teachers - specifically in relation to “blended’ and ‘online’ teaching practice. This would be an equivalent to the “Student Experience” work that was conducted over 2017/18. This sort of work would of course be unafraid to find the divergent, discordant and dissenting ideas, recognising them and accommodating them. Something that is difficult to attain with a precarious and casualised workforce. This work could be done by people in the School of Design, drawing from the work by Errey previously (Errey 2015), who have experience and expertise in User Experience Design as well as working in RMIT.

Recommendations for professional development
Below are a range of suggested structures, systems and formats for professional development activity that align with and build on current activity in RMIT. They seek to address a range of staff needs relating to workload, flexibility and recognition that have been gleaned from the experiences of the College group, from which this paper derives.

Listing professional development needs
After a requirements gathering period where program managers would be asked to convey a list of professional development needs from their teams, this list would be put into a website configured to take further suggestions and voting up and down list items as well as having category features that would help manage the correlations. From there the activities could be generated and prioritised, demonstrating a responsiveness to suggestions, in an open and generative system.

Professional development events of flexible learning
At RMIT the prevailing format for professional development is undocumented face to face training/seminars/workshops. Working within this existing norm, a schedule of events would be organised and promoted at least a month in advance, but they be reliably webstreamed and recorded for a gradually growing number participants who make use of this flexibility. This would require careful consideration for how the events take place, so that online synchronous and asynchronous participation is not disadvantaged, but perhaps advantaged in many ways. Along with the webcasting and recording, links and resources used in the events must be online and easily shareable with participants and their peers, including casually employed staff who await access to RMIT systems.

Professional development events would need to align with the Educator and Researcher Capability Framework so that participation can be recorded in alignment to that, if and when Human Resources develop a way to track and manage workload, rewards and awards relating to the capability framework.

Staff credentials
A structure of “Staff credentials” around the topics of professional development that are aligned to the Educator and Researcher Capability Framework of RMIT would help to discipline professional development events and make them more of a transactional activity (notwithstanding the many objections to commodified learning, this suggestion is in consideration of present staff sentiments and workload). The structure of a Credentialed activity would help to ensure clearly stated objectives with meaningful outcomes, offering work integrated learning activities and assistance in the recognition and management of teacher workload in relation to professional development.

Credits exchange
A credits exchange system would help to address the dubious and unique position RMIT has in not adequately funding or accounting for professional development in its workforce. Credits would be given to all staff members to use, including required professional development, and the rest at the discretion of the staff member, including between teachers to recognise and appreciate peer to peer teaching and professional development. Used credits are then “cashed in” on the budget necessarily allocated to professional development. Casually employed staff may then have a legitimised participation in professional development activity, and both full time and casually employed staff can have a workload allocation or bonus to their income or benefits.

Catalog of topics
A key device in facilitating a creds and credits scheme for professional development is an adequate catalogue of topics and activities to choose from and plan for at the start of semester. A catalogue with website resembling the short courses of community learning found in most local council jurisdictions in Victoria for example, but one that can manage the credits being traded, and the issuing of creds for staff to use in their profiles, work planning, and payroll claims.

Flexible learning
The formats for professional development activities would need to be aligned to good practice in flexible learning design where face to face activities are augmented by recorded web conference participation options where possible, as well as asynchronous engagement. For example, a face to face workshop that includes a recorded web conference channel, that generates raw footage to be used to enhance the web page resources for that topic, and that holds a forum and chat help function would at least demonstrate a standard in flexible learning to the staff participating.

Embedded support
In addition to this obvious example, the effectiveness of embedding professional development support, to sit in staff rooms and work alongside teachers for a period of time cannot be understated when done in coordination with Program Managers and specific work with stated objectives. With consideration, this sort of activity could find efficiencies beyond demonstrating its inherent if non intuitive efficiency, and work in with the credits trading scheme to support more informal ways of learning.

Peer partnerships
Peer to peer teacher and class observations, in a supportive, respectful and non threatening context, has long been shown to be effective (Wingrove 2017 and 2015). Peer to peer observations are also of obvious value, but need to be recognised and rewarded. The work of Wingrove et al in Peer Partnerships should be reintroduced to an over all professional development effort at RMIT.

Conclusion
RMIT University’s executive have made substantial investments and sacrifices to change the infrastructure that supports teaching and learning. This has demanded more of staff in terms of their professional support and development needs, but without adequate recognition of the problems of casualised work and already full workloads. Of all the training support offered to staff, it was the quality assurance process for Canvas that most successfully drove adoption and professional development. That drive however, and the questionable quality of the criteria in that process, generated further resentment among many teachers for the increase in workload it generated. It will be necessary for professional development efforts in the future to recognise the problems of a casualised workforce and the timelines of full time teachers, and to properly fund and account for the workload implications of such development. An integrated approach to professional development, through a system of Credentials and Credits could assist with such workload management and recognition, and help to establish some consistency to the various offerings relating to professional development in the organisation.