Advanced Assessment for Multiple Evaluators
On This Page:
Allocate Evaluators to Learners
Coordinate Multiple Evaluators for a Single Learner
Evaluating Assignments: One Evaluator Per Student
Publishing Assignments: One Evaluator Per Student
Evaluating and Publishing Assignments: Multiple Evaluators Per Student
Coordinating Multiple Evaluations: One Shared Evaluation
Coordinating Multiple Evaluations: Multiple Individual Evaluations
What is Advanced Assessment?
Advanced Assessment is an evaluation tool in the Evaluation & Feedback panel for editing Assignments that enables Instructors to:
- Assign multiple evaluators to evaluate assignments.
- Determine which evaluators can publish and which can evaluate (and who can do both).
- Match evaluators to specific learners.
- Select One Shared Evaluation (co-marking) or Multiple Individual Evaluations as a workflow.
Select Evaluators - Start Here!
Important
Advanced Assessment is an optional workflow.
The following three steps MUST be taken in a new assignment in order to start the workflow for Advanced Assessment:
- Open the Select Evaluators hyperlink.
- Deselect any potential evaluators you do not want as evaluators (by default all are checked).
- Click Done (simply closing without clicking done will NOT start Advanced Assessment).
If submissions are received before any evaluators are selected, the option to use advanced assessment will no longer be available.
Step 1
If you want to use Advanced Assessment, Choose Select Evaluators. The number of evaluators available in the course site is visible in the panel. All users with evaluator roles - e.g. Instructor, TA, Grader - can be selected.
Step 2
All potential evaluators will have check marks to the left of their names by default. De-select anyone you don't want to evaluate this assignment or leave them all selected and choose Done. The selected Evaluators will now be displayed in the Evaluation & Feedback panel. Evaluators can be changed at any time.
Note
To stop using Advanced Assessment, even after submissions have been graded, remove all evaluators from the Select Evaluators menu and save and close the assignment edit page. You will not be able to re-enter Advanced Assessment for that assignment.
Select Publishers
Note
Selecting Evaluators initiates the Select Publishers workflow. Once it appears, all evaluator roles have the potential to be selected as just Evaluator, just Publisher, or both.
Step 1
Choose Select Publishers.
Step 2
Select the relationship between evaluators and publishers from the following three options:
- All evaluators can publish any evaluations (regardless of whether they have done the evaluating)
- Evaluators can only publish their evaluation
- Manually select publishers
If you manually select publishers, a Select Publishers hyperlink will appear, allowing you to choose any publisher from the initial evaluator list.
Allocate Evaluators to Learners
Note
Selecting two or more evaluators initiates the Manage Allocations workflow.
The number of evaluators you will have a choice to allocate evenly depends on how many evaluators are selected and how many learners are in the Classlist.
Step 1
Select Manage Allocations.
Step 2
Choose between the following options:
- Allocate all evaluators to every learner (default)
- Randomly allocate 1 (or more) evaluator to every learner evenly
Note
You have the ability to sort allocations by Sections, Groups or Learners.
If you remove allocated evaluators, D2L will automatically re-allocate.
Step 3
Select Apply Allocation Method, then select Done when it is complete.
Step 4
If you are done editing, Save and Close the assignment.
Coordinate Multiple Evaluators for a Single Learner
Important
Once you receive submissions, this setting will lock. You can change evaluators after receiving submissions, but you will not be able to change from shared to individual (aggregated) evaluations (or vice-versa).
Step 1
If you wish to have more than one evaluator per student, select one of the following two workflows under Coordinate Multiple Evaluators:
- One shared evaluation
- Multiple individual evaluations
Note
The Multiple individual evaluations setting enables each evaluator to conduct their own assessment of a single learner, and only one final aggregated evaluation is published after all evaluations are completed. The final evaluation does not have to be an average of the others.
Step 2
Save and Close the assignment.
Evaluating Assignments: One Evaluator Per Student
Note
Evaluators should learn which students they will evaluate. The assignment View Submissions screen will show all submitted assignments with no differentiation.
Step 1a
To set Advanced Assessments for one evaluator per student, under Manage Allocations, select Randomly allocate one evaluator to every learner evenly. When this setting is selected, the option to Coordinate Multiple Evaluators will disappear.
Step 1b
Evaluators can also be manually allocated by selecting or de-selecting individual students before they have been saved in a random allocation. A message appears in the Evaluation & Feedback panel to notify that custom allocations are used.
Step 2a
If an evaluator enters the evaluation page of a student for whom they have been allocated, but they are not selected as a publisher for that student, they will be able to choose Ready for Review, or Draft. Ready for Review gives these evaluators a signal for publishers that they have moved past the draft stage of their evaluation.
Step 2b
If an evaluator enters the evaluation page of a student for whom they have not been allocated, they will not have a place to enter a grade or feedback. The evaluator for that student is displayed.
Publishing Assignments: One Evaluator Per Student
Publishing assignments allocated to one evaluator per student varies according to each publishing setting.
- All evaluators can publish any evaluation
If this setting is selected, anyone on the evaluator list for this assignment will be able to publish any submission that has been evaluated and set to draft status. A publisher cannot change an evaluation for which they are not an evaluator.
- Evaluators can only publish their evaluation
If this setting is selected, evaluators will only be able to publish the submissions that they have been allocated to evaluate.
- Manually select publishers
If this setting is selected, anyone who is manually chosen can publish any evaluation, but cannot evaluate or make any changes to submissions for which they are not allocated.
Evaluating and Publishing Assignments: Multiple Evaluators Per Student
- The default in the Advanced Assessment workflow is for all evaluators to evaluate every student (Allocate all evaluators to every learner in Manage Allocations).
- Multiple evaluators can be allocated per student through Manage Allocations. If more than one are allocated, the Coordinate Multiple Evaluators options remain available until submissions are received.
- There are two options for coordinating multiple evaluators for one learner: one shared evaluation and multiple individual evaluations. The way that evaluations work differs according the setting.
Coordinating Multiple Evaluations: One Shared Evaluation
- Anyone allocated to evaluate a student's submission can do so.
- Any evaluator can override another's evaluation and feedback while it is still a draft.
- Anyone allocated to publish a student's evaluation can do so.
- Any publisher can retract a published evaluation. If they are also an evaluator, they can change it.
Coordinating Multiple Evaluations: Multiple Individual Evaluations
Important
Be aware that for multiple individual evaluations, any grades and feedback on evaluator tabs WILL NOT be seen by students. ONLY the information on the aggregated tab will display to students after publication.
Note
When this setting is selected, all allocated evaluators for a student receive an evaluation tab at the top of the evaluation, followed by a tab labeled Aggregated.
All evaluations must be completed in order for the Aggregated tab to open.
The process is different depending on the following three evaluator and publisher role combinations:
Evaluators but not publishers
- If an evaluator enters the evaluation page of a student for whom they have been allocated, they will find a tab with their name where they can enter an Overall Grade and Feedback, followed by a Ready to Aggregate button in place of Publish or Ready to Review.
- Once the evaluator has selected Ready to Aggregate, they receive a check mark by their name to indicate the completion of their evaluation.
- A warning appears that only grades and feedback on the Aggregated tab will be displayed to the student.
Publishers but not evaluators
- Those who are selected as publishers but not evaluators will encounter a message letting them know if evaluations remain to be completed, in which case it is not possible to publish.
- Once all the evaluations are complete, the publisher has the ability to enter the Overall Grade and Overall Feedback into the Aggregated tab.
Note
D2L Brightspace does not automatically aggregate the scores. The person who publishes the aggregated score can enter what they deem best as an Overall Grade and Feedback. It is expected that they will review the other scores and feedback to make their decision.
Evaluators and Publishers
- Those who hold both roles have the ability to enter an Overall Score and Feedback in their evaluator tab.
- Only when all the assigned evaluators have completed their respective evaluations will the Aggregated tab be open for publishing.