We had a student in one QMP exam log in using the option “Show list of any unfinished assessments”. This is a checkbox under the login and I suppose this student just viewed it as “remember me” or “I agree” or any of those things you generally see below login boxes that he must be completely desensitised to reading. I have spoken to others who have had students tick this box when logging in also.

Anyway, if this were done on the day of the exam the student would not see the exam in their list of available assessments, since they haven’t partially completed it. The problem is that they do see any assessments they have partially completed and so might accidentally take, say, a practice test in place of a real one. This applies even if their access to the practice test has expired.

In the following video I have logged in to a test earlier and closed the window before submitting the test (in fact, before the 10 minute autosave period). Logging back in after the schedule has ended (i.e. after my access to start the test has expired), I am able to resume the test and submit a set of solutions. Also, because I closed the window within the 10 minute autosave period the timer has reset. Basically the system thinks I have interrupted my test and resumed it later. The problem is that the schedule that gives me access to the test has expired so, in my view, this should not be possible.

From the student’s point of view, they have logged in and taken the available assessment, so they have a strong argument that they have done nothing wrong. If they happen to have taken the practice test as real that may be judged as not their fault.

The only real solution to make sure practice tests aren’t resumed in place of the real test, etc. is to create every student a seperate, one time use username for each exam. This is an administrative hassle. In the real exams we ran we didn’t do this but we made a point of telling the students not to check the box and getting them to check they were seeing “modulecode_exam” and not “modulecode_practice” when they logged in. It is in their interest to take the real exam. Next time, I will be making sure the practice exam looks obviously different from the real one (different coloured background, for example), so the invigilators can easily look around the room and see anyone who has opened the wrong assessment.

I understand that QMP5 has a different layout for available assessments, so that incomplete assessments are more clearly marked as such.

16:00 03/02/10: There is a section of this post missing, which I have removed to give Questionmark time to explore a particular problem without the problem being in the public domain.

At first I thought the ceiling was the sum of the outcomes, and that adding negative values to positive ones – particularly when all outcomes cannot occur simultaneously – is deeply flawed. However, I have now realised that adding any outcome with a value of zero or less resets the ceiling to zero.

In the above video, I add some positive outcomes and the ceiling is appropriately set. However, adding a value of zero or a negative value resets the ceiling to zero. Deleting the offending outcome does not rectify the problem but adding another positive value recalculates correctly. I then delete all three +1 outcomes, but the ceiling is not reduced accordingly until I add another positive outcome. Finally, I add a negative outcome and this again sets the ceiling to zero. Not deleting this, I add another positive outcome to recalculate the ceiling. This time the ceiling is set to 5, which is the sum of all available outcomes, including the negative one! The actual maximum score for this question (weirdly set though it is) is to get parts a=1 and c=2 correct and to answer b anything other than 2, giving a score of 10. However, the automatic ceiling is set to cap scores at 5, so the score recorded for that response would be 5.

So I have observed the following behaviour: Adding a positive outcome recalcuates the ceiling to the sum of all possible outcomes. Adding a zero or negative outcome resets the ceiling to zero. Deleting an outcome does not change the ceiling.

On a related note: We have seen this business of summing the outcomes to give the ceiling is problematic when negative scoring is used. Equally, this method is problematic when different outcomes are assigned to the same choice. In the video below I only have one drop down box and assign three outcomes to it, each worth +1. Obviously only one outcome can be performed at once. I am even careful to not tick “Accumulate” or “Evaluate other outcomes even if this condition is met” (although I have tried it selecting these and there is no difference). The ceiling? 3.

This is a silly example, each option attracts the same score, but you might want to have a drop down box with one option attracting full marks and another attracting partial marks. The automatic ceiling would be set to the sum of these, meaning it is impossible to be assigned 100% for this question, even if your answer is 100% correct. In this case, the automatic ceiling should be the greater of the available options, since each are exclusive outcomes (i.e. this question is not multiple response).

The solution to all of these ceiling problems is to manually override the ceiling when you have finished adding outcomes. Still, this behaviour is erratic and the need to do this is not obvious.

The lecturer created their own exam. In the results, all students scored zero for one question. The lecturer told me that on looking at the question, all the 1s in the correct responses are now “f”, so no marks were given as no student entered f in their answer (the system wouldn’t let them anyway). This is very strange. It was working in mid-December, when the lecturer entered a full set of correct answers but was not working by the exam a month later.

What could have caused this? Well, I had a play and came across the following behaviour. The numeric input boxes (“choices” in QMP) are numbered by default. If you change the label from a number to something more identifible, it changed the reference to that choice in the logic that determines whether the answer is correct (the “outcome”). But watch what happens:

So I hypothesise that the lecturer renamed choice 1 to choice f between mid-December and the exam, and the system did the rest.

I realise this could be demonstrated even more straightforwardly. If the answer contains zero and choice zero is renamed the following happens.

Firstly, I wanted to see what files I could recover and how easy it would be to edit these into a video. The default output is a video of only the slides or a Flash-based interactive show with slides and speaker in a side window. I think this is fine for most purposes but I wanted to see if I can could get both in the same video file. In some talks very little may happen on screen for a period and I think it is more watchable if you see movement.

Secondly, I wanted to run through a slide with a lot of changes to see the clarity of the output and see if I could confuse the automated chapter marking – it didn’t.

Thirdly, I wanted to plug in an external tablet laptop PC and see if the writing on the screen was captured satisfactorily.

Fourth, I wanted to record a clean audio track with tested kit and compare this with the recording made by the ITR kit. Unfortunately I messed this up and the audio recording with the known-to-work lapel mic failed to record anything but silence.

Anyway, here are the outputs:

Video auto-processed by the system (m4v).

Full interactive Flash-based output (Flash; also auto-processed; best for interactivity but only on limited platforms).

Enchanced podcast (m4b; also auto-processed;completely thrown by my animated slide and tablet PC annotations)

And finally, the video I made by manually editing the output (mp4).

The same video on YouTube:

The video quality on the speaker is poor. I suspect as it was only designed to be viewed in a small window so not as bad at half size. Here is the edited video at half size (mp4).

Edit/update: I have found the raw video file is saved as a .h264 and have converted this to mp4. The result is here:

The next response system class is in 24 minutes as I write this, so no time to throw something experimental into the mix. But I may suggest we try AnyWhere in a fortnight when we run the class again.

Before the class the lecturer was very worried about the software and I was worried about the practicalities of handing out and getting back up to 180 handsets. In fact the software worked just fine as expected and handing out and collecting in the handsets (150, actually) was a doddle with three volunteers from the audience. The students seemed to enjoy using the equipment and the lecturer got useful feedback. Everything very smooth!

One question in particular struck me as an excellent demonstration of the advantages of the technology. The correct answer was A and 72% of the class said A but 16% said D. Would those 16% have identified themselves in a show of hands?

The issue now is for the lecturer to come to a procedure where he can run the process independently of me. I made the PowerPoint file by screen capture of a LaTeX-generated PDF. I have suggested if serious use were to be made of the system in the School we could come to some more automated arrangement using some LaTeX->image or PPT conversion. But in the meantime, the lecturer’s preferred method for presenting lectures is by writing on a tablet PC (see image below). As he can annotate a PowerPoint and store the annotations into the PPT file he is going to try this method. He will, before the class, write the questions and answers (or enough to serve as a reminder – the students will have a print out of the questions) on the PowerPoint and save it. Then TurningPoint will run as last week with images of hand written, rather than LaTeX-ed mathematics.

Tablet PC displaying maths taken with my phone during the class

I am also investigating the use of TurningPoint Anywhere, which promises use of the TurningPoint application over any program, not just PowerPoint.

This week I will attend class again but remain passive. The intention is the lecturer will be able to run the classes by himself in future. Of course, once we are comfortable with the technology we have to find a way to evaluate whether the system is meeting the lecturers aim of encouraging the students to engage with lecture content before formal assessment.

The lecturer has used audience-wide multiple choice questioning in previous years via OMR and a show of hands. This is used formatively at the start of an examples class. I have recommended TurningPoint (the system the University has) as an alternative that should be to the benefit of all concerned.

There are, however, a few issues to be overcome. The question file is created in LaTeX (as it should be) but TurningPoint only sits over PowerPoint (and not, for example, Beamer). The lecturer provided me the LaTeX source, which my MikTeX didn’t have the packages to compile, and the PDF. If, for the sake of time and quality of typesetting, we discount rewriting all the maths into PowerPoint then we are left with inserting the maths as images. This copy is only for display onscreen during voting; if a copy were needed by a student who could not access the images the LaTeX source could be used to create something suitable. There are lots of clever ways to convert LaTeX into images or possibly even LaTeX to Office converters but again for the sake of time a simple screengrab seems the quickest way. I took one screen capture of the five questions and answers and chopped this up into consituent images.

I created a TurningPoint-enhanced PowerPoint file and added a questioning slide. Although my questions are images I have still to enter something into PowerPoint for TurningPoint to get its hooks into (to discover how many possible answers there are; to have something to refer to each question and answer in the report; etc.). For the question prompt I entered the question number “Q1″-”Q5″. For the answers I entered the letter corresponding to the answer  “A” through “E”.

Empty TurningPoint question

Then I positioned the question and answer images in appropriate places. As far as TurningPoint is concerned I am asking some pretty dull questions that just happen to have images nearby. For the students, it should be clear which equation relates to which answer.

Completed TurningPoint question

The report will simply list question numbers and how many students chose each of A-E. This is very unsophisticated use of TurningPoint but it is simple and quick to produce and, crucially, gets the job done.

Dummy test question: I have included a slide at the start as an ice-breaker (“Which of the following is a mathematician famous for his law of universal gravitation?” with answers: Newton, Plato, Wagner, the lecturer himself or Rooney). This is vital because we will be running the TurningPoint kit with up to 180 students and the chances that all the kit works first time are slim indeed. The devices can easily be set to the correct frequency (you may want to do this in various circumstances). Unfortunately this means a student randomly pressing buttons can accidentally change frequency and, in my experience,  in any given box of kit there will be some not working and there is no one with the time to check all 180 devices before each lecture (except if the students check one each). The procedure for setting to the correct channel is easy and the lecturer will print the instruction onto an OHP slide (rather than try to display this as well as the PowerPoint show on the same screen) but it is helpful to have the dummy question to get the students familiar with the kit and allow them to discover whether theirs works. Procudure is: attempt to answer; if you get a green light, you’re done; if you get red, press GO-frequency-GO and try again.

The lecturer is going to practice before the session and we will meet up just before to check everything is still working. I feel reasonably confident but still pretty anxious about the whole thing. One of the major barriers is physically handing out and collecting back up to 180 KeePad voting devices. Beyond that, we just have to hope the kit works.

This document refers to numeric entry questions as very problematic. In this post, I will attempt to recreate the problems related to this question type.

In the video below you will see what happens when the question asks for a number of decimal places but there is a trailing zero. Is the answer QMP allows in fact correct?

In the video below you will see what happens when the correct answer to a numeric question is an integer and a decimal answer is entered.

(N.B. if a decimal answer is used then it works as you would expect. i.e. if you set the answer to be 4.5 then it doesn’t accept 4.51. So if you set the answer to be 4.0 then it expects a decimal and doesn’t truncate the decimal part.)

In the following clip we see what happens when one of the answers is zero.

(N.B. If there is only one numeric entry box this doesn’t happen – QMP says “Question not answered”. But if there is more than one it counts it as answered.)

I find it incredible that anyone would release software with such bugs, especially for paying customers. The only solution offered by my predecessor and those I met at the conference is to find workarounds to avoid the bugs (i.e. no numeric questions where the answer is zero, enter integer answers as decimal when setting the question – 4 as 4.0 and so on,  no trailing zeroes). If anyone has a better idea I would be pleased to hear it!