Contributing legal tasks

We are collecting realistic legal work tasks written by practising lawyers. Each task is a piece of work a junior lawyer might be given by a partner or client: a set of matter documents, a short instruction, a model answer, and a marking scheme. Frontier AI models attempt every task, and an automated grader marks each answer against the marking scheme. We accept tasks that a careful lawyer can answer well but strong AI models still get materially wrong.

The sample in albion-house-lease-break-sample.zip is a complete task in the required format: an English-law commercial lease break. Unzip it to see every part.

What we are asking for

We would like 20 tasks from you.

Spread: cover at least four practice areas, for example corporate, real estate, employment, disputes and finance, with no more than six tasks in any one area.
Jurisdiction: Malaysian law is welcome, as is any common-law jurisdiction you practise in. State it in each task.json.
Several tasks per matter: up to three tasks may share one set of matter documents, each with a different instruction and deliverable. For example, the same lease file can carry an advice task, a notice-drafting task and a review task.
Each task: 40 to 70 criteria, a reference answer that passes at least 95% on the submission site, and documents that all open.
How to submit: upload each task as its own zip. Use a clear, distinct task_id for each one, so we can track all 20.

What a task contains

your-task-name/
  instruction.md      the short instruction the lawyer receives
  documents/          the matter documents (emails, agreements, letters, spreadsheets, notes)
  reference/          your model answer, one file per requested output
  task.json           title, practice area, jurisdiction, outputs and the marking criteria

Zip the folder and upload it on the submission site (see How to submit).

instruction.md

A short, natural instruction of the kind a partner or client actually gives. The detailed brief usually belongs in an email inside documents/, as it would in practice. Name the output file the lawyer should save, for example "Save your advice as Northmere_break_advice.md".

documents/

5 to 30 documents in their native formats: .eml, .docx, .pdf, .xlsx or .pptx. Every file must open.
Realistic file names, as they would appear in a matter file: Lease - Albion House (executed) 25.04.2022.pdf, not doc_03.pdf.
Include the legal materials needed to answer: short extracts or summaries of the relevant statutes, rules or cases, in your own words with brief quotations. The task must be answerable from the documents alone, without outside research.
Do not include confidential client information or real personal data. Anonymise or replace names, companies, addresses and figures so that no real matter can be identified.

reference/

The answer you would expect from a strong senior associate. It must satisfy at least 95% of your own criteria. Write it as real work product: for example an advice memo with a heading, a summary of advice, the analysis with clause and statutory references, and next steps with dates.

task.json

Copy the structure from the sample. The fields are:

Field	Content
`task_id`	A short lowercase name, such as `kl-commercial-lease-renewal`
`title`	A one-line title
`practice_area`	For example "Corporate", "Real estate", "Employment", "Construction", "Disputes"
`jurisdiction`	For example "Malaysia" or "England and Wales"
`task_type`	`analyze`, `draft`, `review` or `research`
`instructions`	The same text as `instruction.md`
`outputs`	Each requested output, with a `name` (the file name without extension) and a one-line `description`
`reference_files`	The file names in `documents/`
`criteria`	The marking scheme (below)

Writing the criteria

Each criterion is one pass or fail check on the answer. Aim for 40 to 70 criteria.

{
  "id": "C07",
  "rule": {
    "question": "Does the advice state that the break notice must be served on the current landlord, Harrowgate Estates Ltd, at the address in its letter of 3 September 2026, because clause 23.2 requires service at the landlord's last notified address?",
    "pass": "States that service must be on Harrowgate at the notified address, or equivalent. Equivalent wording and reasoning are acceptable.",
    "fail": "Advises service on the original landlord or at the address in the lease, or does not address where the notice must be served."
  },
  "deliverable": ["Northmere_break_advice.md"]
}

Rules that every criterion must follow:

One point per criterion. Test a single fact, conclusion, step or drafting requirement.
Objective. Two lawyers would agree on the result. Do not test style, tone or overall quality.
Legally correct and supported by the documents. Cite the clause, section or document the point rests on.
Fair. Accept equivalent wording and any other reasonable correct approach. Do not fail an answer for a small slip or because a point appears in a different place.
Positive. Write each criterion as something the answer should do.
No overlap. Two criteria must not test the same point.

What makes a good task

The tasks we value most are the ones where strong AI models make the mistakes a careful lawyer would not. In our experience these come from:

Several instruments read together. A later variation, side letter or consent changes the effect of the main agreement, and the answer differs item by item.
Exact deadlines and amounts, where the documents contain a tempting but wrong trigger date or figure.
Consequences the instruction does not mention, such as an adjacent statutory regime, what survives termination, set-off, interest, tax or insolvency effects.
Limits on authority. The client says what may and may not be done today, and the answer must still be useful within those limits.
A plausible but wrong position taken by the other side or by the client's own team, which the answer must deal with correctly.
A draft to correct. An unagreed working draft of a notice, letter or agreement that must be turned into clean, correct operative text.

Avoid making a task hard through ambiguity, missing facts or trick wording. A qualified lawyer must agree that every criterion is fair.

How to submit

Upload each task's zip at https://legal.standardharness.com with the access code we give you. You will get a reference number for each upload. Send it to your point of contact, and we will review the task and get back to you. A task is accepted when it is fair, realistic and still hard for current AI models. We may suggest revisions to criteria that need clarifying.

Originality and confidentiality

Submit only your own original work. Do not copy published precedents, textbooks or other copyrighted material beyond brief quotations, and do not include any real client or personal data. By submitting, you confirm that the task is your own work, that it contains no confidential client information or real personal data, and that you may license it to us.

Back to the upload form · README.md