BasicLTI for QuestionMark Perception

Using Sakai to connect to load-balanced QuestionMark Perception using BasicLTI.

Why?

  1. Provide single sign-on capability to avoid maintaining a separate password for each user;
  2. Sakia (at Oxford) allows ‘external’ users to be created  from their email addresses. Where these users are given permission with Sakai and where they have accounts in Perception, they would then be able to access assessments.

How?

By adapting the QuestiionMark BasicLTI connector. This is a community edition project which the very helpful Steve Lay pointed us at: http://projects.oscelot.org/gf/project/lti-qm/frs/

Process as follows (PLEASE NOTE THAT I AM BLOGGING THIS AS I GO SO I APOLOGISE FOR ANY MISUNDERSTANDINGS – I’LL UPDATE AS MY LEVEL OF UNDERSTANDING INCREASES):

  1. Create DB – as we have a load-balanced setup and a clustered MSSQL DB, we had to create another db in our MSSQL server to avoid introducing a single point of failure. Create the tables using the following (adapted from http://www.spvsoftwareproducts.com/php/lti_tool_provider/)
    CREATE TABLE lti_consumer (
     consumer_key varchar(255) NOT NULL,
     name varchar(45) NOT NULL,
     secret varchar(32) NOT NULL,
     lti_version varchar(12) DEFAULT NULL,
     consumer_name varchar(255) DEFAULT NULL,
     consumer_version varchar(255) DEFAULT NULL,
     consumer_guid varchar(255) DEFAULT NULL,
     css_path varchar(255) DEFAULT NULL,
     protected tinyint NOT NULL,
     enabled tinyint NOT NULL,
     enable_from datetime DEFAULT NULL,
     enable_until datetime DEFAULT NULL,
     last_access date DEFAULT NULL,
     created datetime NOT NULL,
     updated datetime NOT NULL,
     PRIMARY KEY (consumer_key)
    );
    
    CREATE TABLE lti_context (
     consumer_key varchar(255) NOT NULL,
     context_id varchar(255) NOT NULL,
     lti_context_id varchar(255) DEFAULT NULL,
     lti_resource_id varchar(255) DEFAULT NULL,
     title varchar(255) NOT NULL,
     settings text,
     primary_consumer_key varchar(255) DEFAULT NULL,
     primary_context_id varchar(255) DEFAULT NULL,
     share_approved tinyint DEFAULT NULL,
     created datetime NOT NULL,
     updated datetime NOT NULL,
     PRIMARY KEY (consumer_key, context_id)
    );
    
    CREATE TABLE lti_user (
     consumer_key varchar(255) NOT NULL,
     context_id varchar(255) NOT NULL,
     user_id varchar(255) NOT NULL,
     lti_result_sourcedid varchar(255) NOT NULL,
     created datetime NOT NULL,
     updated datetime NOT NULL,
     PRIMARY KEY (consumer_key, context_id, user_id)
    );
    
    CREATE TABLE lti_nonce (
     consumer_key varchar(255) NOT NULL,
     value varchar(32) NOT NULL,
     expires datetime NOT NULL,
     PRIMARY KEY (consumer_key, value)
    );
    
    CREATE TABLE lti_share_key (
     share_key_id varchar(32) NOT NULL,
     primary_consumer_key varchar(255) NOT NULL,
     primary_context_id varchar(255) NOT NULL,
     auto_approve tinyint NOT NULL,
     expires datetime NOT NULL,
     PRIMARY KEY (share_key_id)
    );
    
    ALTER TABLE lti_context
     ADD CONSTRAINT lti_context_consumer_FK1 FOREIGN KEY (consumer_key)
     REFERENCES lti_consumer (consumer_key);
    
    ALTER TABLE lti_context
     ADD CONSTRAINT lti_context_context_FK1 FOREIGN KEY (primary_consumer_key, primary_context_id)
     REFERENCES lti_context (consumer_key, context_id);
    
    ALTER TABLE lti_user
     ADD CONSTRAINT lti_user_context_FK1 FOREIGN KEY (consumer_key, context_id)
     REFERENCES lti_context (consumer_key, context_id);
    
    ALTER TABLE lti_nonce
     ADD CONSTRAINT lti_nonce_consumer_FK1 FOREIGN KEY (consumer_key)
     REFERENCES lti_consumer (consumer_key);
    
    ALTER TABLE lti_share_key
     ADD CONSTRAINT lti_share_key_context_FK1 FOREIGN KEY (primary_consumer_key, primary_context_id)
     REFERENCES lti_context (consumer_key, context_id);
  2. Create a user on the DB whose details you will enter below.
  3. Create a new administrator in Enterprise Manager which will be used by QMWISE
  4. Create an MD5 checksum for the QMWISE user as described here: https://www.questionmark.com/perception/help/v5/product_guides/qmwise/Content/Testing/Calculating%20an%20MD5%20Checksum.htm
  5. Change settings in config.php:
    define('CONSUMER_KEY', 'testkey');define('CONSUMER_SECRET', 'testsecret');
    define('DB_NAME', 'sqlsrv:Server=localhost;Database=testdb');
    define('DB_USERNAME', 'dbusername');
    define('DB_PASSWORD', 'dbpassword');
    define('QMWISE_URL', 'http://localhost/QMWISe5/QMWISe.asmx');
    define('SECURITY_CLIENT_ID', 'ClientIDFromQMWISETestHarness');
    define('SECURITY_CHECKSUM', 'ChceksumFromQMWISETestHarness');
    define('DEBUG_MODE', true);
    define('ADMINISTRATOR_ROLE', 'LTI_INSTRUCTOR');
    define('QM_USERNAME_PREFIX', '');
    define('WEB_PATH', '/qmlti');

    DB_NAME should adhere to DSN conventions here: http://www.php.net/manual/en/ref.pdo-sqlsrv.connection.php

    SECURITY_CLIENT_ID is the administrator which is used by QMWISE

    SECURITY_CHECKSUM is the chceksu which can be generated from the QMWISE test harness

    WEB_PATH:  we needed to change to subdirectory where we had placed lti connector code

  6. As detailed here (https://www.questionmark.com/Developer/Pages/apis_documentation004.aspx) you cannot write applications yourself that provide a trust header so switch this off under Administration | Server Management | Server Settings in Enterprise Manager. Don’t forget to reset IIS (Start | All Programs | Accessories right-click Command Prompt and Run as Administrator then type
    iisreset

    ) to ensure it picks up the change in settings.

  7. Upload qmlti folder into C:\inetpub\wwwroot on both of the load-balanced servers
  8. Copy the lti.pip file into the \repository\PerceptionRepository\pip folder on the storage area shared by both load-balanced servers
  9. At this stage, I wanted to be able to launch an assessment as an IMS ‘Learner’/QM Participant from the test_harness. I couldn’t work out how to do this so made the following changes (these will probably be superseded once I have worked out how to set up the tool as an IMS ‘Instructor’ so that the correct assessment is shown based upon the User/Context_ID):
    • added a custom parameters textarea to the test_harness.php (188):
      <div class="row">
      <div class="col1">
      Custom Parameters (each on separate line; param_name=param_value)
      </div>
      <div class="col2">
      <textarea name="custom" rows="5" cols="50" onchange="onChange();" ><?php echo htmlentities($_SESSION['custom']); ?></textarea>
      </div>
      </div>

      ..and on line 45, something to save them:

      set_session('custom');
    • Something to read them and turn them into request parameters in test_launch.php (48):
      $custom_params = explode('\n', str_replace('\n\r','\n',$_SESSION['custom']));
      foreach($custom_params as $custom_param){
      if(strpos($custom_param, '=')){
      $custom_param_parts = explode('=', $custom_param);
      $params['custom_'.$custom_param_parts[0]]=$custom_param_parts[1];
      }
      }
    • And changed the parameter read in index.php (65) to look for the assessment ID with ‘custom_’ prepended as this seems to be what the specification expects:
      $assessment_id = $tool_provider->context->getSetting(custom_ASSESSMENT_SETTING);
  10. Don’t forget that you’ll need to ‘Allow run from integration’ under ‘Control settings’ for any assessment that you want to launch with QMWISE.
  11. Now pouint your browser to http://yourserver/qmlti/test_harness.php. It should pick up your key and secret automatically from config.php so, to launch an assessment as a student, all you need to do is:
    • enter the username of a valid participant in the User Details | ID box;
    • Enter the ID of an assessment for which that student is scheduled in the new custom parameters box as follows:
      ASSESSMENT_SETTING=1234567887654321
    • Click ‘Save data’ to..save your data
    • Then click Launch

It’s worth noting that doing things like this does breaks the strict separation between the various tiers as the PHP application is running on the QPLA servers and accessing the database directly.

Finally, a huge thanks to QuestionMark’s Steve Lay and to Stephen Vickers for his Basic LTI code.

 

Accessibility – Rogo vs Perception

When considering and comparing Rogo and Questionmark Perception, one thing we looked at was the accessibility options available in both. As would be expected both give consideration to accessibility requirements, but the approaches and options vary.

Questionmark Perception (V5)

Rogo

In Rogo, accessibility settings are set for an individual, rather than for an assessment/schedule. Once specified, these settings are used for every assessment that the user sits. The following settings are available:

  • Extra Time (doesn’t actually do anything as assessments are not timed)
  • Font Size
  • Typeface
  • Background Colour
  • Foreground Colour
  • Marks Colour
  • Heading/Theme Colour
  • Labels Colour

User’s cannot adjust any settings for themselves within exam delivery, but this is unlikely to be a problem, as accessibility requirements are almost always known about prior to an assessment. Furthermore, they can easily be changed ‘on the fly’.

Importing Questions into Rogo

In preparation for our first running our first live test of Rogo, we needed to get an exam paper that had been created in Questionmark Perception into Rogo. This seemed simple enough, simply by exporting the QTI XML from Perception and importing it into Rogo. This generally worked reasonably well, but there were a few issues that had to be dealt with:

  • Not unsurprisingly, images needed special attention. The links to the images in Perception (e.g. “%SERVER.GRAPHICS%topicresources/1065007843”, where the number at the end is the topic id) were replaced (using a blanket find/replace) with “/media(/subdirectorypath)”, and the images copied into the appropriate subdirectory.
  • There were some issues with tables where an image was shown side by side with the answer options, which required a hack in Perception. For some questions there were problems in the HTML in Perception (left over from the dark ages), so fixing the HTML in Perception fixed the import problems relating to this.
  • EMQ feedback – the general question feedback from Perception is given as the feedback for all stems in Rogo, which does not have general feedback for EMQs. This can be fixed by removing the feedback from all except the last question, so that it just appears at the end of the responses. Alternatively, the feedback can be split up to give each stem its own feedback. Either of these options would be laborious for large numbers of questions, so the best long-term alternative may be to alter the import code to deal with this.
  • MCQ feedback – when importing MCQs (including those created in Rogo, exported and imported back in without any modification), in which only general feedback is provided, the general feedback is put in the answer feedback for each answer and general feedback is left empty. This has been added to the Rogo Trac system (#683,  https://suivarro.nottingham.ac.uk/trac/rogo/ticket/683)
  • Correct answers for EMQs – after importing all the correct answers are set to A. This is also the case if a question is created in Rogo, exported from Rogo and imported back in. This issue was added to Trac (#682, https://suivarro.nottingham.ac.uk/trac/rogo/ticket/682) and has now been fixed by the Rogo team, presumably for a future release. For this assessment, fixing this was simply, but annoyingly, a case of going through all the questions and changing the correct answers.
  • Rogo does not have a multiple MCQ question type, i.e. several MCQs that share the same stimulus and so are combined into a single question, so that it looks like an EMQ, but there are different answer options for each stem. Such questions are recognised as Matrix questions by Rogo when they are imported, but this format is not appropriate. There is no way of modifying an EMQ in Rogo so that subsets of the answer options are only available for certain stems. We have a large number of questions of this format, and currently the only option is to separate them out into MCQs, which is not a sustainable solution. Therefore, we may need to create a multiple MCQ question type, which hopefully would not be too difficult using the EMQ question type as the basis.

Having fixed all these problems, it seemed as though we had an exact copy of the assessment, with everything working as it should. The only difference between the questions as far as the students were concerned was that the multiple MCQ question (there was only one of this type in this paper) was split into 5 separate MCQs.

However, while running the assessment, we were to come across the Curious Case of the Disappearing Image, which, it turned out, was caused by a problem related to importing.

The importance of paper type in Rogo

One of the problems which very nearly forced us to abandon our first test of Rogo yesterday was our lack of understanding of the importance of paper ‘type’ in assessment delivery in Rogo and, arguably, the degree to which we are used to the way that QuestionMark Perception does things.

Test Type Feedback? Restart? Fire Exit? Multiple Attempts? Review Previous attempts?
Self-Assessment Test Yes, granular control No No Yes Yes
(Quiz) (Yes) (Yes if SAYG) (n/a) (Yes) (Yes if enabled)
Progress Test No – but options shown in Assessment properties screen. Yes No No No
(Test) (User decides) (Yes if SAYG) (n/a) (Yes) (Yes if enabled)
Summative No Yes Yes No No
(Exam) (No) (Yes if SAYG) (n/a) (Yes) (Yes if enabled)

The three main assessment types in Rogo (with a comparison with Perception in brackets)

In Questionmark Perception, ‘Assessment Type’ is a convenience method for setting various parameters of assessment delivery. However, the parameters are set explicitly and are visible to administrators. They are also all individually configurable regardless of assessment type.  In Rogo, paper type is considerably more important as, although it sets very similar parameters to those in Perception, they do not seem then to be independently configurable or, crucially, to be visible to administrators. As a result it is very easy to inadvertently, but radically, change the way in which an assessment is delivered. Or as we found, it was not possible to deliver the assessment in the way required at all.

We wanted to be able to deliver a formative paper under exam conditions which would display marks and feedback to students at the end but which would also allow students to restart their assessment if something went wrong before they had finished. We began by setting paper type to ‘Progress Test’ as this gave us the feedback we required but then realised this wouldn’t allow students to restart in the event of a hardware failure. So we tried ‘Summative’ but, despite having ticked the two feedback tick boxes, no feedback appeared. Luckily, since we were only testing the system, we could nip in and alter the offending bit of code (paper/finish.php, line 234) to allow feedback with a summative paper:

$show_feedback = true;

but this wouldn’t be acceptable on a production system.

It seems to me that, in this respect, the QuestionMark Perception model is better – paper type should help by suggesting appropriate settings not by constraining how an assessment can be delivered.

Question Analysis – Difficulty/Facility and Discrimination

Difficulty/Facility

This is simply a measure of the proportion of students that answered a question correctly and has a value of between 0 and 1. It is calculated by taking the sum of the actual marks for each candidate and dividing it by (the maximum multiplied by the number of candidates).

It is often referred to as difficulty, but should probably be known as facility, as a value of 0 means that no-one answered the question correctly, and a value of 1 means that everyone answered the question correctly.

General wisdom seems to be that questions with facility values of around the pass mark for the assessment (e.g. pass mark = 70%, facility = 0.7) will give the most most useful information about the candidates.

Discrimination

The purpose of item discrimination is to identify whether good students perform better, worse or the same as poor students on a question. Based on Kelley (1939), good and poor students are defined by taking the top and bottom 27% based on overall assessment mark.

Discrimination for a particular answer option is then calculated by subtracting the fraction of the bottom group who gave the answer from the fraction of the top group who gave the answer. So:

  • A positive item discrimination means a higher proportion of people in the top group chose the answer than in the bottom group. A high positive value for the correct answer generally means the question is a good discriminator, which is what we want (but is very difficult to achieve!). A positive discrimination for an incorrect answer may suggest an issue, but could equally just mean that it is a good distractor.
  • An item discrimination of 0 means the same number of people from each group gave the answer, so the answer doesn’t discriminate at all. Questions where everyone got the correct answer will always have a discrimination of 0.
  • A negative item discrimination means a higher proportion of people in the bottom group chose the answer. This would be expected for an incorrect answer. A negative discrimination on a correct answer may indicate something is wrong, as more able students are choosing an incorrect answer.

To make a question a good discriminator, the correct answer should have a high positive discrimination, and the incorrect answers should have a negative discrimination.

Thoughts on Rogo Delivery

The Rogo delivery system seems to be robust and, unlike the admin interface, free from bugs. It also seems fast, and initial load-testing has shown it to be capable of putting up with a reasonable number of students starting or finishing an assessment at the same time, although we have not tested this in a live environment. This is despite us running the system on a single, reasonably-spec’d, but not especially new, server.

One handy feature in the Rogo delivery system, that Perception does not have, is the Fire Alarm button – see https://learntech.medsci.ox.ac.uk/wordpress-blog/?p=15.

However, there are a number of things that are missing, compared with Perception: Auto-save, timed assessments, secure browser.

Auto-save

Rogo is not able to auto-save assessments, e.g. every 5 minutes. It does save when a user moves to another screen (and when the fire alarm button is clicked) but since we prefer to run our exams as a single, scrollable screen containing all of the questions, this is not particularly helpful. However, since the functions are there for saving the current status of an exam while the exam is continuing, it would proably not be a big step to write an AJAX-based auto-save function.

Timed Assessments

Assessments in Rogo are never timed and so do not autosubmit when the time is up. The assessments we have run in Perception have traditionally been timed, so that each user’s time only begins when they enter the assessment page, and the assessment automatically submits when their time runs out. There is also a timer on the page, so the student can see exactly how much time they have left, as this would not be the same for all students. However, we have recently been leaning increasingly towards running untimed assessments, mainly due to the problems of disaster recovery with a timed assessment. For example, if a fire alarm were to go off, it would not be possible to pause the timer and allow students to carry on from where they left off. Therefore, the lack of timed assessments is unlikely to be a major issue, as we were heading in this direction anyway. Nonetheless, it would be good to have the option!

Secure Browser

Unlike Perception, Rogo does not have built in secure browser support. However, there are free secure browsers out there that can be used instead, e.g. Safe Exam Browser – http://safeexambrowser.org. We might cover this more fully at a later date when we’ve done more testing with it. Update: Now covered in this post

A few things that Rogo does better than Perception

Having been tied to Perception for many years, it is refreshing to be able to test out Rogo, which, although still in it’s infancy, has the potential to be a viable alternative. While we have come across some issues, there are also a good number of things that it does better than Perception (and I expect there are plenty more to come):

  1. Quicker, simpler, less clunky, more intuitive – Rogo just feels like a nicer, less 90’s system to work with
  2. Version history for questions – it will be extremely useful to be able to look back at how a question has changed and developed. In Perception, we use a complex system of question naming to keep track of major versions of a question, but this is far from ideal.
  3. Drag and drop question ordering
  4. Easy duplication of exams, with options to copy the questions or just link to the same set of questions
  5. Fire Alarm/Exit button in exam mode – students can click this button to if there is a fire alarm or other problem during an exam and it basically just shows a blank screen, so that students can be stopped and started together, without anyone seeing anyone else’s screen while they leave the room, or benefiting from seeing the exam for an extra amount of time. Assessments are not timed, and do not auto-submit, so this is not used to stop the timer.
  6. Flexible feedback settings

Perhaps the most important benefit of Rogo though is that it is OSS (Open Source Software) and written in PHP, which is where our skills primarily lie. Therefore, when there are things that don’t work as we would like, we can change them. Whereas with Perception, we generally just have to live with them and pay for the pleasure!