• Runner
• Runner Provider
• Runner Events
• Runner Life Cycle
• Runner Proxy
  • Security
    • Token
    • TimeLine
• Runner Controller
• Runner Layout
• Runner Helpers
  • Map Helper
  • Messages Helper
• Runner Plugins
• Runner Configuration

Runner

There is currently two versions of the QTI Test Runner.

The first version, let call it the Old Test Runner, is based on iframes and use a monolithic architecture, which is not easy to extend.

The new version, let call it the New Test Runner, is based on a modular approach. It is built using the test engine provided by the taoTests extension. This engine is based on the Delegation design pattern and provide a way to extend the features by using plugins.

Both versions are configurable through the same set of options. A chapter is dedicated to this aspect.

The following documentation is only related to the new version of the Test Runner.

To get more info on the runner implementation, please refer to the taoTests extension.

Provider

The Test Runner engine is based on the Delegation design pattern. This means the engine delegates most of its API to an external provider that brings the implementation.

The runner architecture is mainly built around promises and events. Each API that refers to a life cycle step is asynchronous and relies on a promise to be achieved. Once the promise is resolved, an event is triggered. The rejected promises are routed through the error handling that triggers error events.

The runner maintains a list of internal states, some of them can also persist in the browser storage.

To communicate with the server, the provider instantiates a particular component: the proxy. This component brings a communication API, and this must be the only way to communicate with the outside.

The provider also has in charge to build and inject the UI layout, and this is done by a particular component.

In addition to the standard components, the Test Runner behavior can be extended by plugins, that can hook the life cycle steps or react to events.

To get more info on the provider implementation, please refer to the taoTests extension.

Events

Events take a big place in the runner's life cycle. Here is the list of the events handled by the new Test Runner.

Life Cycle

The life cycle of the runner is quite complex. Here is a very simplified life cycle to illustrate the main workflow of the runner:

init -> render -> loadItem -> renderItem -> unloadItem -> finish -> flush -> destroy

In fact the life cycle of the runner is more complex, as it involves events and states. A test runner does not carry on only one item, so the life cycle is manager through a loop.

To get more info on the standard life cycle of the runner, please refer to the taoTests extension.

Proxy

The proxy brings the ability to communicate with the outside. Like the runner it is built using the Delegation design pattern.

To get more info on this component, please refer to the taoTests extension.

Each time a component needs to interact with the server it must use the proxy. For instance to request a test action:

    runner.getProxy().callTestAction(...);

All the proxy API are linked to backend actions:

To setup the proxy, the provider relies on a config object that will translate and maintain the runner config set. This is the purpose of the qtiServiceConfig component.

Security

Some mechanisms are provided to enforce the securgity of the test. A security token is provided to validate the actions, and the time duration of the test is stored using a time line.

Token

Most of the interactions with the server will require a security token. This security token is unique and is generated with each action.

Each time the client request the server it must provide this token. If it is valid, the server allows the action to be performed, otherwise an error is triggered. Once the action is done, the server generates and sens a new token to the client. The latest will have to provide the new token on the next request, and so on.

Not all actions require a security token, but the majority does.

TimeLine

The time line is a a way to compute the time elapsed during the test both on the server and the client sides, and is intended to synchronize the two.

Client side, the time spent by the test taker to take the test is computed between requests:

• it starts when the item is received and displayed
• it ends when the item responses are submitted

So the time spent to query the server is not added to the elapsed time.

Server side, the time is captured and updated as this:

• it starts when the item data are sent back to the client
• it ends when the item responses are received

As you can see, a delay can be found between the client side and the server side. This is especially true if the client time is not perfectly synchronised with the server. For this reason, the client and the server time lines are separated. However, the client time line must be circumscribed by the server time line.

A time line is a suite of time point pairs: [start, end]. To be able to compute the elapsed time, we need a list of valid pairs. If a pair is incomplete the result cannot be computed. The same if the pair is bad built (like 2 starts for 1 end, or vice-versa). This is why the time points are set as often as possible.

Controller

A particular controller is dedicated to the Test Runner life cycle: taoQtiTest_actions_Runner

A dedicated service is implemented to manage the Test Runner. The controller only provides a link to this service. To maintain the persistence, a service context must be provided each time a service API is called.

Each action of the controller returns a JSON content, which is always formatted like this:

    // success:
    {
        "success": true,
        "token": "<the new security token>"
        ...
    }

    // error
    {
        "success": false,
        "type": "error"|"exception"|"TestState"|"FileNotFound",
        "message: "<some message>",
        "token": "<the new security token if the error is not related to the security (403)>"
        "code": 403|404|500
    }

Beside the action related parameters, each time the client request the server it must provide this set of parameters:

• serviceCallId or testServiceCallId: The identifier of the text execution
• testDefinition: The identifier of the test itself
• testCompilation: The identifier of the test delivery

Mots of the actions are secured, and a security token must also be provided through the HTTP header X-Auth-Token.

The exposed TestRunner actions are:

• init

  • Parameters:

    • clientState: An optional state the client want the server to be aware
    • storeId: An optional identifier for the local storage

  • Security Token: not needed

  • Description: Initializes the delivery session. If the client claimed it was paused, the server will set the state to pause and close the test session. The test taker will then need to resume the test. The client store identifier is also persisted server side, in order to trace the change of computer. This action is called as well as for a new test session or to resume an existing session. So this the fist action called by the client. The timer and the security token are also reset.

  • Response: The action sends back to the client a JSON object that contains:

    • testData: The definition data of the test, this also cover the config and options.
    • testContext The context data of the current item.
    • testMap: The full map of the test.
    • lastStoreId: The identifier of the local storage the client should use.
  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• getTestData

  • Parameters: none

  • Security Token: required

  • Description: Provides the test definition data

  • Response: The action sends back to the client a JSON object that contains:

    • testData: The definition data of the test, this also cover the config and options.
  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• getTestContext

  • Parameters: none

  • Security Token: required

  • Description: Provides the test context object

  • Response: The action sends back to the client a JSON object that contains:

    • testContext The context data of the current item.
  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• getTestMap

  • Parameters: none

  • Security Token: required

  • Description: Provides the full map of the test items

  • Response: The action sends back to the client a JSON object that contains:

    • testMap: The full map of the test.
  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• getItem

  • Parameters:

    • itemDefinition: The identifier of the item

  • Security Token: required

  • Description: Provides the definition data and the state for a particular item. Starts the timer.

  • Response: The action sends back to the client a JSON object that contains:

    • itemData: The definition data of the item
    • itemState: The item state
    • baseUrl: The base url of the item, needed to resolve the assets
    • rubrics: The optional rubric blocks that comes with the item
  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• submitItem

  • Parameters:

    • itemDefinition: The identifier of the item
    • itemDuration: The time elapsed on the item at the client side
    • payload: The item responses and its state are provided through the request payload using a JSON format:
      • itemState: The new item state
      • itemResponse: The item responses
      • emptyAllowed: A flag that tells if empty responses are allowed

  • Security Token: required

  • Description: Stores the state object and the response set of a particular item. End the timer.

  • Response: The action sends back to the client a JSON object that contains:

    • displayFeedbacks: A flag telling if some feedbacks need to be displayed
    • feedbacks: The optional feedbacks to display
    • itemSession: The new item session state
    • notAllowed: A flag telling if the response has been rejected by the server, in that case the client should stay on the current item.
    • message: A message to display if the notAllowed flag is set
    • testContext: The new test context, when the notAllowed flag is set, in order to synchronize the timer
  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• storeTraceData

  • Parameters:

    • itemDefinition: The identifier of the item, when the storage is related to an item. (optional)
    • traceData: The data to store. It must be a JSON content.

  • Security Token: required

  • Description: Allows the client to store information about the test, the section or the item.

  • Response: The action sends back to the client a JSON object that only contains the status.

  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• move

  • Parameters:

    • ref: The identifier of the target position to reach. (optional)
    • direction: The direction of the move, could be 'next' or 'previous'
    • scope: The scope of the move, could be 'item', 'section', 'part'

  • Security Token: required

  • Description: Moves the current position to the provided scoped reference: item, section, part

  • Response: The action sends back to the client a JSON object that contains:

    • testContext: The context data that refers to the new item to display
  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• skip

  • Parameters:

    • ref: The identifier of the target position to reach. (optional)
    • scope: The scope of the move, could be 'item', 'section', 'part'
    • itemDuration: The time elapsed on the item at the client side

  • Security Token: required

  • Description: Skip the current position to the provided scope: item, section, part

  • Response: The action sends back to the client a JSON object that contains:

    • testContext: The context data that refers to the new item to display
  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• timeout

  • Parameters:

    • ref: The identifier of the target position from which the timeout occurred.
    • scope: The scope of the timeout, could be 'item', 'section', 'part'

  • Security Token: required

  • Description: Handles a test timeout. May request a test end.

  • Response: The action sends back to the client a JSON object that contains:

    • testContext: The context data that refers to the new item to display
  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• exitTest

  • Parameters: none

  • Security Token: required

  • Description: Exits the test before its end.

  • Response: The action sends back to the client a JSON object that only contains the status.

  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• finish

  • Parameters: none

  • Security Token: required

  • Description: Finishes the test. It should not be possible to interact with this test later. The next action will be to go back to the index page.

  • Response: The action sends back to the client a JSON object that only contains the status.

  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• pause

  • Parameters: none

  • Security Token: required

  • Description: Sets the test in paused state. It should not be possible to interact with this test unless it is explicitly resumed. The next action will be to go back to the index page.

  • Response: The action sends back to the client a JSON object that only contains the status.

  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• resume

  • Parameters: none

  • Security Token: required

  • Description: Resumes the test from a paused state.

  • Response: The action sends back to the client a JSON object that contains:

    • testContext: The context data that refers to the new item to display
  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• flagItem

  • Parameters:

    • position: The position of the item to flag. By default the current item is flagged. (optional)
    • flag: The flag state. By default the flag is set. (optional)

  • Security Token: required

  • Description: Flag an item for later review.

  • Response: The action sends back to the client a JSON object that only contains the status.

  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• comment

  • Parameters:

    • comment: The text of the comment to store

  • Security Token: required

  • Description: Stores a comment about the test

  • Response: The action sends back to the client a JSON object that only contains the status.

  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

• messages

  • Parameters: payload

  • Security Token: not needed

  • Description: Manage the bidirectional communication through a polling. This action is called every period of time.

  • Response: The action sends back to the client a JSON object that contains:

    • responses: A list of responses for the received requests
    • messages: A list of messages the server has sent to the client
  • Possible error: An error may be triggered if the test session is already finished, or if the test does not exist.

Layout

The new Test Runner does not use iframes anymore, it relies instead on a HTML5 layout, and delegate the access handling to an AreaBroker.

Basically, this AreaBroker gives access to:

• a main container for the whole runner
• a header bar, that contains for instance the title
• a control bar, that contains for instance the progress bar and the timer
• a navigation bar, that contains the navigation controls
• a toolbox, that contains the tools buttons
• a side panel, which is a multi-purpose display area
• a content panel, which obviously contains the current item

The layout is built and injected to the page by the provider when the Test Runner is rendered.

Access to the layout

Each component of the Test Runner must rely on the AreaBroker to get access to the layout. To do so, the Test Runner provides an API to get the AreaBroker instance:

    runner.getAreaBroker();

Then you can request access to the wanted area. Either by relying on the standard areas, which must always be implemented by each provider:

    runner.getAreaBroker().getHeaderArea();
    runner.getAreaBroker().getControlArea();
    runner.getAreaBroker().getContentArea();
    runner.getAreaBroker().getPanelArea();
    runner.getAreaBroker().getToolboxArea();
    runner.getAreaBroker().getNavigationArea();

Or by requesting an extra area, which may be implemented by a particular provider. For instance, the QTI provider bring also access to the actionBar, which contains both toolbox and navigation:

    runner.getAreaBroker().getArea('actionsBar');

Note: The brought AreaBroker always returns jQuery selections.

To get more info on the area broker implementation, please refer to the taoTests extension.

Helpers

Some helpers are provided to centralize the common features.

Map

The Map Helper brings several API to manipulate the Map of the test assessment. Each of these API need the map to manipulate as their first parameter.

Note: Each time a component needs to manipulate the map, it should use this helper.

Messages

The Messages Helper is intended to complete the message displayed when trying to leave the test.

    var exitMessage = getExitMessage(message, scope, runner);

For a particular scope the helper will complete the provided message, if needed. The runner's instance must also be provided.

An option allows to prevent this completion: test-taker-unanswered-items-message.

Each time the runner must display a message related to an exit, it must use this helper to complete the message.

Plugins

A chapter is dedicated to the standard available plugins.

To get more info on the plugin implementation, please refer to the taoTests extension.

Options

A chapter is dedicated to the available options and how to manage them.