Please note that this is a work in progress, and we will add issues as we find them, or remove them as they are fixed (or we find out that they aren’t really issues).
As discussed in other posts, we’re using QTI (Question and Test Interoperability) to create and deliver maths questions. The majority of the questions are being written by modifying (directly in the XML) questions from Dr Sue Milne’s collection. Uniqurate is being used to put questions together into tests, and QTIWorks is being used to deliver the questions.
We have found all the parts of the system – Uniqurate, QTIWorks and the QTI specification in general – very useful, usable and flexible. However, as is inevitable with such a complex and extensive specification and combination of creation and delivery systems, we have come across some issues. None of these are major problems that we cannot work with or around, but I thought I would continue my recent theme of writing “work in progress” blogs, and write one containing all of the issues that we have with each part of the system/process.
QTI (inc. Maxima, MathAssess engine)
- Rounding floats doesn’t seem to work properly. After rounding, there seems to often be an arbitrary (small) value added to or subtracted from the rounded value, giving a value to >10 decimal places. For example, rounding 41.28947 to 2 decimal places, which should give 41.29, might give 41.29000014723.
- There can be issues with recognising 2 equally valid correct answers as such. For example, if 1/x^3 is the correct answer to a question, x^-3 might be considered not fully simplified, even though it is at least as simple and arguably simpler. Another example would be requiring -(3x^2)/y, when (-3x^2)/y or -3x^2/y, or even -3x^2y^-1 would be equally correct. It might be possible to fix this within the XML, but would ideally be built into the Maths comparison engine.
Uniqurate
- Questions are sometimes added to the wrong section of a test, or are not added at all (the latter seems to happen after a question has just been deleted from the test section). Generally returning to the Main Menu and then editing the test again gets rid of the issue temporarily.
- It would be great to be able to add multiple questions to a test section in one go, rather than having to add them one at a time.
- It is possible (and not that difficult) to create invalid combinations of of the “Number of questions to choose from section” and “Show questions only once/Repeat Questions” settings. If you tell it to show more questions that you have added, but leave it set to use each question only once, the test is invalid. It would be good if Uniqurate stopped you from doing this. To me, it seems as though there is significant overlap between these settings, at least when you tell it to show more question instances that there are questions in the section.
- Can only specify the total number of questions in a section, not the number of each question.
- The “Add question to section” link only works when the mouse cursor is an arrow. Depending on the browser (I’m talking about IE10 on Windows), the pointer will be one of an arrow, a pointer (hand with pointed figure) or a text cursor when you hover over the “Add question to section” link, depending on which part of the link you are over. It only seems to actually work as a link when it is the arrow.
- Related to the above point, if you choose to show more questions than there are in the section, and hence set it to repeat questions, you are not guaranteed one of each question. E.g. if you have 2 question, and choose to show 3 questions, you could get 3 of one question and none of the other.
- IDs given to sections are random strings, which makes them hard to identify when looking at results in QTIWorks. It would be good to be able to specify the ID for a section.
- If questions are repeated in a section, the total score for the test is calculated as if each question was to be shown only once.
QTIWorks
- The maths input help box could do with some improvements – see my post about a Help Guide for Maths Entry Questions.
- Would like to be able to display multiple questions on a single page, rather than one question per page. This would be possible to set up by creating multiple questions in a single XML file, but it would be much nicer to be able to do it through the delivery settings.
- Right clicking on a MathML block, then going to Settings > Math Renderer > MathML sometimes shows a simplified version of what is displayed, which for some questions (e.g. division of fractions) might actually be the answer. While it is unlikely someone would realise and do this, and it is only for certain questions where it would give the answer, it is something that could be looked at.
- When a single part question is answered, it shows as “Answered” (in blue). However, when a multi-part question is answered, it is shown as “Finished” (in grey).
- Similarly, if you answer a question (so it is marked as “Answered” and in blue), but then look at the solution, it is then marked as finished. This makes sense if the answer given was incorrect, but if the answer was correct and then the student looked at the solution, it would be good if the status remained as “Answered”.
Fixed/Non Issues
- Would like scores, feedback and/or solutions to be shown when reviewing a test – Feedback and model solutions can be shown when reviewing by adding the following to the testPart section in the assessment.xml file for a test:
<itemSessionControl showFeedback="true" showSolution="true" allowReview="true" />
showFeedback means that the last feedback that the user was shown when attempting the question is shown when they review it. showSolution provides a “Show Solution” button, which puts the correct response (as defined in the response declaration, or by using setCorrectResponse in templateProcessing) in the answer box. However, it seems that if a maths variable is set as the correct response to a question, this will not be displayed properly as the solution when reviewing.