Designing an online assessment is a blueprint problem first. The question "what questions should I write?" is premature until you have answered "what decision does this assessment need to support?" Software comes second. Most assessment failures — questions that confuse instead of measure, scores that do not predict performance, candidates who dispute results — trace back to design gaps, not technology limitations. This step-by-step guide walks through the entire design process, from defining the purpose to reviewing item quality after the first live run.
Every assessment should answer a specific question: Can this candidate perform the job? Did this employee retain the training? Is this student ready to advance? Writing that question explicitly before building anything else forces clarity. If you cannot write the decision in one sentence, the assessment scope is not yet defined. A vague purpose produces a scattered question set that measures nothing cleanly.
The blueprint maps topics to question counts and marks. If the assessment covers three areas — data analysis (40%), statistical methods (35%), and business communication (25%) — the question distribution and marks should reflect those weights. Without a blueprint, question writers default to their own knowledge areas and the assessment ends up measuring one topic at three times its intended weight.
Topic areas and weights: SQL fundamentals (30%) — 9 questions; statistical concepts (25%) — 7 questions; data interpretation and charts (25%) — 7 questions; business context and communication (20%) — 6 questions. Total: 29 questions, 35 minutes, one correct answer per question, no negative marking. Pass mark: 65% to advance to interview stage. The blueprint forces agreement on what the role actually requires before anyone writes a single question.
Each question should test one concept. Compound questions — those that require two correct pieces of knowledge to answer — make it impossible to diagnose why a candidate failed. Write more questions than you need for the final assessment. A bank of 60 questions lets you draw a randomised 30 per attempt, reducing answer-sharing between candidates and giving you data on individual question difficulty.
A reasonable timing rule for MCQ assessments is 60–90 seconds per question for straightforward recall items, 90–120 seconds for application questions, and 2–3 minutes for scenario or calculation items. Give a small buffer — a 30-question assessment with mostly application questions runs well at 45 minutes rather than 30. Test the timing yourself by doing the assessment without any cheating advantage, then add 30% for candidates who read more carefully.
The pass mark should be set based on what it means to be minimally competent, not on what most candidates are likely to score. If 95% of candidates pass, the bar may be too low to be useful. If 5% pass, either the bar is too high or the questions are too hard. Set the initial pass mark based on your best professional judgment, then review it after the first live run. For negative marking: only use it when the assessment is trying to differentiate between candidates who know versus those who guess.
Before candidates see the first question, they should see a clear instructions screen: total marks, duration, question count, marking scheme (including whether negative marking applies), and a note on any integrity controls in effect. Surprises during an exam create anxiety and support tickets. Candidates who know exactly what to expect perform more consistently and raise fewer post-exam disputes.
Run the assessment with a small group — colleagues, a few willing candidates, or a sample class — before the first live session. Review two things: which questions most candidates got wrong (ambiguous wording or wrong answer key), and which questions everyone got right (may be too easy to be useful). Revise or retire items that fail on either count.
"All of the above" and "none of the above" as options look like legitimate choices but reduce answer quality. A candidate who knows any single option is wrong can eliminate "all of the above" without knowing the actual answer — the question tests elimination strategy, not knowledge. Similarly, questions where one option is noticeably longer than the others reliably signal the correct answer to test-savvy candidates. Both patterns reduce discrimination and waste question slots.
Set the access window in the platform rather than relying on candidates to start at the right time. Send invites from the dashboard if your platform supports it — EasyEvaluate lets you email candidates in batch directly, which eliminates the manual step of copying and forwarding a link. Confirm the auto-submit setting is active so the timer enforces itself without manual intervention.
A score of 70% means very different things depending on the assessment design. On a well-calibrated 30-question test with a solid difficulty distribution, 70% probably means competent. On a 30-question test where all questions were easy, 70% tells you very little. On a test where the pass mark was set without a pilot, 70% may be above or below the minimum competency threshold by accident. Results are only as meaningful as the quality of the assessment they come from.
