QMP Problem: How many marks is it out of?

Here is a problem I have found with scoring questions in Questionmark Perception. This is in a drop down box question type used to deliver true/false/no answer multi-part questions. When you add outcomes, the score of these is added to a variable in the Question properties called the ceiling – the maximum score. This provides the “out of” value when scoring the question. However, there is a problem when adding further outcomes to allow negative marking. After adding a true (+1) / false (-1) marking system you end up with a ceiling of zero. Because the ceiling is zero any mark above zero is treated as zero. In the example in the video below, any question score with a negative value is correct but any score of 0, 1, 2 or 3 is treated as 0.

At first I thought the ceiling was the sum of the outcomes, and that adding negative values to positive ones – particularly when all outcomes cannot occur simultaneously – is deeply flawed. However, I have now realised that adding any outcome with a value of zero or less resets the ceiling to zero.

In the above video, I add some positive outcomes and the ceiling is appropriately set. However, adding a value of zero or a negative value resets the ceiling to zero. Deleting the offending outcome does not rectify the problem but adding another positive value recalculates correctly. I then delete all three +1 outcomes, but the ceiling is not reduced accordingly until I add another positive outcome. Finally, I add a negative outcome and this again sets the ceiling to zero. Not deleting this, I add another positive outcome to recalculate the ceiling. This time the ceiling is set to 5, which is the sum of all available outcomes, including the negative one! The actual maximum score for this question (weirdly set though it is) is to get parts a=1 and c=2 correct and to answer b anything other than 2, giving a score of 10. However, the automatic ceiling is set to cap scores at 5, so the score recorded for that response would be 5.

So I have observed the following behaviour: Adding a positive outcome recalcuates the ceiling to the sum of all possible outcomes. Adding a zero or negative outcome resets the ceiling to zero. Deleting an outcome does not change the ceiling.

On a related note: We have seen this business of summing the outcomes to give the ceiling is problematic when negative scoring is used. Equally, this method is problematic when different outcomes are assigned to the same choice. In the video below I only have one drop down box and assign three outcomes to it, each worth +1. Obviously only one outcome can be performed at once. I am even careful to not tick “Accumulate” or “Evaluate other outcomes even if this condition is met” (although I have tried it selecting these and there is no difference). The ceiling? 3.

This is a silly example, each option attracts the same score, but you might want to have a drop down box with one option attracting full marks and another attracting partial marks. The automatic ceiling would be set to the sum of these, meaning it is impossible to be assigned 100% for this question, even if your answer is 100% correct. In this case, the automatic ceiling should be the greater of the available options, since each are exclusive outcomes (i.e. this question is not multiple response).

The solution to all of these ceiling problems is to manually override the ceiling when you have finished adding outcomes. Still, this behaviour is erratic and the need to do this is not obvious.

QMP Problem: 0 becomes peter

I have overseen two Questionmark Perception exams in the last two weeks and this has highlighted some interesting behaviour. Here is the first.

The lecturer created their own exam. In the results, all students scored zero for one question. The lecturer told me that on looking at the question, all the 1s in the correct responses are now “f”, so no marks were given as no student entered f in their answer (the system wouldn’t let them anyway). This is very strange. It was working in mid-December, when the lecturer entered a full set of correct answers but was not working by the exam a month later.

What could have caused this? Well, I had a play and came across the following behaviour. The numeric input boxes (“choices” in QMP) are numbered by default. If you change the label from a number to something more identifible, it changed the reference to that choice in the logic that determines whether the answer is correct (the “outcome”). But watch what happens:

So I hypothesise that the lecturer renamed choice 1 to choice f between mid-December and the exam, and the system did the rest.

I realise this could be demonstrated even more straightforwardly. If the answer contains zero and choice zero is renamed the following happens.