You can read the "Reporting and Analytics Maturity Models, A Guidepost for Developing Institutional Competency" section in its entirety below or read it directly from the A New Age of Learning Management Analytics whitepaper (starting on page 9). As always, if you have any comments or would like to share insight on the current state of LMS, please share it in the comments below or feel free to contact us directly at feedback@moodlerooms.com.
Reporting and Analytics Maturity Models, A Guidepost for Developing Institutional Competency
In the early planning phases of developing institutional capacity for LMS-based information architecture, institutions should take into consideration a maturity model or a designed set of structured levels that clearly articulate the behaviors, practices and processes of a datarich LMS environment that can reliably and sustainably produce required outcomes. For the purposes of this model, at each level, there are distinct profiling and operational attributes. An example of such a maturity model can be found in the following levels of critical path adoption.
Level 1: Informational
The Informational level is primarily defined by access to stored information within the institution’s native LMS system(s) that take into consideration static reporting of basic performance indicators at the course level, based solely on student and faculty data relationships. This level is highly LMS-centric and does not aggregate other sources of enterprise data, operational data stores or multiple data marts. This level of competency is mainly relegated to the extraction and reporting of transaction-level data, which typically consists of artifacts of course-level activity and “static” data informative to faculty and/or a limited universe of information “consumers.” In this case, institutions are defining a set of preconfigured, non-interactive “push” oriented reports created from central IT sources on a one-off reporting basis.
Profiling Capabilities:
Examples of level one reporting and basic analysis typically represent grade tracking for in-course formative and end–of-course assessment, course-level online time audits, time spent in online courses, and the associated student behaviors within a particular course. Other profiling applications might also include class attendance, assignment punctuality and online course persistence and retention.
Operational Capabilities:
The informational level provides basic operating features from simple reporting mechanisms (e.g. SQL Server views, OLAP cube-based data models and other transaction data report generation methods) tied via LMS data integration to be run on demand through the LMS interface with standard “out-of-the-box” reports. Additional features include ad-hoc reporting in multiple export formats, advanced data visualization and built-in capabilities for processing of multivariate data.
Level 2: Adaptive
The Adaptive level of the maturity model moves from a largely static environment to interactive reporting and analysis of multivariate data within an LMS course environment. The Adaptive level of this model gives administrators, faculty and staff the ability to draw inference, analysis and interpretation of multivariate data, thus creating a platform for actionable analytics. For instance, these informed actions might influence online course design, course content and student intervention. Adaptive maturity models often use two-dimensional data analyzing multiple evidences of student behavior and applied effort. In the Adaptive environment, faculty have access to the necessary tools to establish an ongoing practice of using analytics for ongoing intervention, analysis of alignment to course objectives, and ultimately to the improvement of the teaching practice itself.
Profiling Capabilities:
At this level of reporting and analytics competency, profiling capabilities would include the collection and analysis of student interviews and surveys, course evaluation, and behavioral paths that track communication, collaboration and interaction in online teaching, peer group support and group projects and assignments. Another benefit to establishing an Adaptive level of competency is to better track, interpret and understand interactions with online course materials, primary research and other online accessible course support information. Adaptive
levels allow for comparison, contrast and gap analysis for persistent observation of in-class vs. online student activity. At the same time, student and faculty success predictors may emerge from the longitudinal view of aggregate data from intra-course tests, discussions and assignments. Adaptive models also allow advanced analysis of multi-level vertical and horizontal insight at the institutional, section, group, course and individual faculty and student levels.
Operational Capabilities:
Adaptive maturity model levels are typically evident beyond the centralized IT environment and are successful driven down to the end user practitioner levels of the institution. Adaptive reporting and analytic environments can be classified as levels of individual competency in self-service reporting, drill down, data manipulation and shaping with advanced sorting and filtering by predetermined variables with virtually no IT support. This level can often feature capabilities of individualized and shared data application workspaces, offering end users the option to use filters for replicable reports and the ability to monitor, analyze and report on work space data. Also important in establishing Adaptive levels of analytic competency is the ability to independently create cascading notification systems at student, faculty, advisor and program levels. The core requirement at the Adaptive level is the integration beyond LMS architecture to disparate systems in ERP, finance, enrollment portfolio, and other administrative data stores.
Level 3: Transformative
At the peak of the maturity model, advanced reporting and analytic systems create a truly transformative opportunity for virtually all levels of the institution. The Transformative level of the maturity model assumes the technical capacity to aggregate multi-evidence data that provides insight and action into operational, strategic and financial objectives and scorecards. A fully transformative analytic maturity model allows for data discovery capabilities to analyze longitudinal data to evaluate programmatic performance by broad-based objectives. LMS originated analysis provides end users at all levels and expertise with the ability to juxtapose, as well as compare and contrast, large data sets from offline and online programs to audit performance. Transformative levels of system maturity offer unlimited ETL integration from the LMS and a wide variety of disparate legacy data systems, allowing for not only analysis but also applied data modeling for operational, financial and strategic transformation.
Profiling Capabilities:
In advanced data profiling, Transformative levels allow institutions to cascade institutional and programmatic goals and objectives to online learning program, section and course learning objectives to form gap analysis and intervention workflows. Within this level, a wide array of data sets are available, such as student academic purchasing history, registration analysis for course and section capacity planning, and enrollment or retention history. The Transformative levels involve predictive analytics capabilities designed for nons-tatistical end users. Examples might include:
- Predicting outcomes of student risk and academic performance based on pre-matriculation performance data from secondary education.
- Enrollment persistence likelihood for course and section capacity planning.
- Computing ROI models on per-unit cost of instruction.
Predictive capabilities can also provide a snapshot of the student’s course-related work and online behavior analysis compared to that of their peers, providing pattern recognition over time. Using this data, faculty can make informed decisions on intervention and remediation paths on an individual student-to-student basis.
Operational Capabilities:
Operational capabilities of Transformative stages of the maturity model include real-time reporting, end user custom reporting capabilities, and advanced use of dashboards that can partition domains separately for individual business units or learning audiences. Using advanced dashboarding techniques, end users have the ability to filter information by domain, user groups and user-defined variables. At this stage, end user faculty have the ability to personalize and share filtered reports through the use of custom user fields containing detailed user profile data. Most importantly, Transformative stages of development can provide valuable student engagement data through analysis of historical and real-time online student behaviors based on predetermined institutional thresholds and flags.
In one-dimensional testing, quiz and survey instruments, faculty can perform assessment item
analysis with the ability to drill down to question level for ongoing item analysis. Transformative levels also have capabilities of broad based distribution via e-mail list “push” and the ability to create data sharing networks with institutional peer groups. Analytics marries large data sets, statistical techniques, and predictive modeling. It could be thought of as the practice of mining institutional data to produce “actionable intelligence.”
Using factor analysis and logistic regression, a model was developed to predict student success within a course. Six variables were found to be significant:
- ACT or SAT score
- Overall grade-point average
- CMS usage composite
- CMS assessment composite
- CMS assignment composite
- CMS calendar composite
No comments:
Post a Comment