Creating Tasks on SprawdzAI

Our platform supports many forms of collaboration. Whether you want to expand our collection with a new task created by you or found elsewhere, organize your own contest or contribute to one that is already planned, or use our system for your own development or a study group you run - we will be happy to help!

If you are considering creating your own task, or do not have an idea yet but simply want to become familiar with the system, all you need to do is message one of the organizers of the AI study group at MIKO (for example, afedafed), the official SprawdzAI Discord account, or email sprawdzai@mikomath.org. You will then receive permission to edit your own tasks.

The rest of this page will show you how to edit tasks correctly (assuming you already have a link to your first task). Reading this entire document may take a dozen or so minutes, but compared with the time required to create a complete task (writing the statement, preparing datasets, and so on), that is not very long. We recommend reading it carefully before you begin creating a task - once you know the system well and have prepared the statement and datasets, integrating a task with SprawdzAI will often take only a few minutes!

Using the CLI

When creating tasks, our Python command-line interface will be very helpful. It lets you easily maintain a local copy of a task. You can edit all files locally and then synchronize them with the server using a single command. Installing and using the CLI is optional, but this tutorial is intended mainly for editing through the CLI. The same operations can always be performed intuitively in the browser, so we will not discuss them here in detail. You can install it with:

pip install --upgrade sprawdzai-cli

Verify the installation:

sai

Then sign in to your account:

sai login

You can quickly check your account information with:

sai me

You can find a description of every command with:

sai [COMMAND] --help

sai

also supports using multiple accounts at once (for example, if you want a separate account for editing tasks). To do so, use the

sai backend

command and the

--backend

flag (for more information:

sai login --help

). To sign out of an account (or every account if you signed in to several), use

sai logout --all

You can now easily create a local copy of the task:

sai pull [id | slug | url]

For example,

sai pull https://sprawdzai.org/task/task-slug/...

sai pull task-slug

. After running this command, you will see a message telling you where the task folder was created. By default, the folder has the same name as the task unless the

--location

flag was used. Enter the task folder:

cd task-folder

You can now edit the task locally and then synchronize it with the server using:

sai sync

sai

also handles remote changes and conflicts. In other words, you can edit a task locally and in the browser at the same time. When you run

sai sync

, all changes made elsewhere will automatically be transferred to the local copy, and all changes made locally will be transferred to the server. If conflicts occur - that is, the same field was changed both remotely and locally - you will be notified about the conflict and can resolve it in any way you choose.

Introduction

To create secure, high-quality tasks effectively, you must first understand how the system works and why it was designed this way. In SprawdzAI, we had to reconcile several qualities that a good evaluation system should have. Tasks should be secure: regardless of the intentions of editors or users, no other user or system should be harmed. At the same time, it should not be possible to change your score by hacking the system, and in particular it should not be possible to extract data from the test dataset. Finally, we also want to give users as much information as possible about their submission so that they can fix errors in their solution.

For this reason, we normally divide evaluation into two phases. In the first phase - running the notebook - it has access only to training data (which users most often already have direct access to), so we can expose the entire output produced during that phase (the notebook run's STDOUT and STDERR). At the end, the notebook generates a cloudpickle file that will be run during evaluation. It then has access to test data, although it should of course not have access to labels or answers. Even the data alone, however, would make solving the task substantially easier if extracted (someone could, for example, classify it manually). We therefore usually expose only minimal logs from this phase. This is precisely why the split exists: it allows us to provide full logs for as long as possible and restrict them only once test data can influence the algorithm in any way. In short, the standard task-evaluation process is as follows:

We run the notebook, which generates a cloudpickle file.
The checker (
checker.py
) is run:
- First, it loads all test labels (answers) into RAM.
- It removes every answer file and leaves only the data that will be passed to the cloudpickle function. Otherwise, the cloudpickle file could simply read all the answers.
- It runs the cloudpickle file in a separate process (running it directly inside
  checker.py
  would make it trivial to read the answers or assign any score).
- That separate process saves the function output to one or more files.
- The checker calculates the score from the answers loaded into variables and the saved predictions.

Of course, this is only the most common structure - SprawdzAI supports many other task styles. One possibility is an interactive task, where code needs to run repeatedly and continuously affect the evaluation process (for example, “label noise” from the second edition of OAI). Such tasks are so varied and unusual that they are not covered in this documentation. If you want to create one, contact the administration for help.

Importantly, you can also go in the opposite direction and give up some of the assumptions stated at the beginning of this section in exchange for simpler task creation. For example, you can provide validation or test data directly when the notebook runs and have it generate a predictions file. The checker can then skip most of the earlier steps (it does not need to run anything in a separate process or remove all answer files), and when writing it you can go straight to the final step: calculating the task metric. This naturally creates a very high risk of data extraction, because someone could simply print the data to STDOUT or STDERR. If simplicity matters more to you than whether someone can cheat the system, you may consider this option. Alternatively, you can simply reduce the STDOUT and STDERR character limits (see the next section). That preserves the inability to extract test data, but sacrifices another original assumption: giving users as many debugging logs as possible.

Nevertheless, we generally encourage writing tasks around cloudpickle files whenever possible. Although this introduction was fairly long, once you understand it, creating tasks that launch a separate process is only marginally harder and more time-consuming. We have worked to make task creation as quick as possible, including by providing a library that already implements almost everything you may need. Now that you understand the system better, we can finally move on to actually editing tasks!

Editing metadata

To edit a task in the browser, click the “Edit task” button above the submission button (it appears when you have permission to edit the task). All fields are located where they belong: in the green header and in the statement, submissions, and ranking tabs. For local editing, use the

metadata.json

file. Editing the title, description, tags, or custom link is easier on the website because you can immediately see the visual result. Most other fields - especially files and datasets - are easier to edit locally through

sai

. It is therefore usually best to keep both a development environment (for example, Visual Studio Code with a terminal open) and the website open while editing a task. This lets you make each change wherever it is most convenient and immediately see visual changes on the website.

title
- task title. It should generally contain no more than 4 words.
short_description
- task description. It may contain newline characters, but this is not recommended and they will not be visible in the task list.
tags_ids
/ tags - we recommend editing these in the browser.
slug
/ custom link - the link under which the task will be available. It may contain only lowercase Latin letters, digits, and
-
characters.
difficulty
/ difficulty - task difficulty in the range 1.0-7.0. If you do not want to specify a difficulty or it has not been determined, leave this field empty (or set it to
null
in
metadata.json
).
notebook_limits
,
checker_limits
/ the table in the submissions tab - limits during the
notebook
execution and
checker
evaluation phases. The values displayed in the green header are the notebook limits.
- time
  / time limit - in seconds.
- ram
  and
  vram
  / RAM and VRAM limits - in
  metadata.json
  , the value
  "ram": 1048576
  means 1 MB, but you can also enter, for example,
  "ram": "1m"
  or
  "ram": "1024k"
  . When using the form that runs a cloudpickle file, it is worth giving the checker more memory - specifically, as much additional memory as the checker itself uses (if it uses any), for example for keeping data in RAM or performing calculations on the GPU. For users' convenience, from their perspective they should have as much RAM as the task specifies, so the checker limit should be the sum of that amount and the memory the checker uses anyway.
- cpus
  / % of CPU cores - an additional limit on the task's total CPU-core usage. For example, 150% (
  "cpus": 150
  in the CLI) limits CPU usage to 150% of a single core's capacity. Unless you have an unusual need for an additional CPU speed restriction, we suggest setting it to 1000%, which allows the task to use all available CPU capacity (normally only one or four cores anyway - 1000% is only an extra limit that will not be reached).
- cpu_only
  / CPU only - enabling this option blocks access to the GPU.
- allow_internet
  / Internet access - unless there is a specific, carefully established need, please never grant Internet access, as it can lead to system abuse.
compute
/ compute type - determines which compute machine evaluates the task. You should receive information about this flag together with the task link.
submission_files
/ submission files - the files that must be uploaded with a submission and their maximum sizes. In the CLI, you can use the same memory-size abbreviations, for example
"max_file_size": "100k"
.
valid
,
test
/ the second table in the submissions tab - limits for submissions sent to the validation or test dataset.
- daily_submissions
  / Daily submission limit - set this to 0 to disable submissions to that dataset completely. The website will display this correctly: resubmission and submission to that dataset will no longer be available when its limit is 0. If both limits are 0, the submissions tab and submission button disappear. As a task editor, however, you bypass this limit - simply enter edit mode, and the tab will reappear so that you can submit to either dataset regardless of the limits set here. Set
  "daily_submissions": null
  to disable the daily limit (unlimited submissions).
- notes_char_limit
  ,
  stdout_char_limit
  ,
  stderr_char_limit
  / Notes, STDOUT, and STDERR character limits - the character limits for the corresponding fields shown in submission details (the checker return value is called the notes; it is the field at the top, above STDOUT). Please do not exceed reasonable values around 20,000, so that excessive logs do not fill our disk space. The standard values of about 300 for notes and 10,000 for both STDOUT and STDERR are usually entirely sufficient to convey all relevant information without truncating any errors.
display_ranking
/ ranking visible - whether the ranking is visible to all users. When disabled, it remains available to you and administrators after entering edit mode.
score_mode
/ ranking calculation type - this option is not finished yet; please always set it to “Metric” (
"score_mode": "raw"
), or strange things will start happening!
baseline_score
- relates to other ranking calculation settings. With “Metric” selected, this value changes nothing, so for now you should ignore it entirely (like the previous option) until these options are implemented.

Using the website and additional permissions

As a task editor, you have access to many useful capabilities. While viewing the page outside edit mode, you see the task exactly as everyone else will. After entering edit mode, you can access the submissions and ranking tabs even if they are disabled. You also bypass all submission limits, both daily limits and the global limit of one submission at a time. Queues are uncommon on SprawdzAI, but if an exceptional situation causes submissions to queue, please do not abuse the bypass and make someone wait an hour for a dozen of your submissions to finish. If the queue is empty anyway, you are welcome to send many test submissions. As an editor, you also receive all information needed to debug checkers; you can find it at the bottom of the submission details for every submission to that task. You can also rejudge submissions and view submissions from everyone in the task.

Managing datasets

Every dataset in SprawdzAI is simply one file, most often a

.zip

. You can find all datasets in the Files tab in the dropdown that appears when you click your username in the upper-right corner of the page (you must be an editor of some task to access this tab). Continually zipping folders, uploading them through the tab, and managing task assignments would be very tedious, so datasets are also integrated with the CLI. There are four possible evaluation phases (called “stages” in the CLI): you can assign separate datasets to submissions sent to validation and test, and separate datasets to notebook execution (the “running” phase) and checker execution (the “evaluation” phase), giving four possible combinations.

The most important command is:

sai ds

It shows which datasets are assigned to each phase. It also gives you links and IDs for your datasets. Be careful not to leak those links, because possessing one could, for example, allow someone to download the test dataset directly. Initially, the dataset list should be empty.

You can create a new dataset by running:

sai ds create [name]

We recommend avoiding spaces - and generally any characters other than letters and digits - in dataset names, because they make further work more difficult. If you run

sai ds

again, you will see that the new dataset was created and automatically linked to all phases. This means it will be visible both during notebook execution and to the checker (and any cloudpickle files it runs), for both validation and test submissions. You can unlink a dataset from a phase with:

sai ds unlink [name] --notebook/--checker (-n/-c) --valid/--test (-v/-t)

For example,

sai ds unlink [name] -n -v

unlinks the dataset from the notebook-execution phase for validation submissions, while

sai ds unlink [name] -c

unlinks it from the checker phase for both validation and test submissions. The corresponding command is:

sai ds link [name] --notebook/--checker (-n/-c) --valid/--test (-v/-t)

It links a dataset to a given phase. For example:

sai ds link [name] --test

links the dataset to test submissions, for both the notebook-execution and evaluation phases.

You can edit the contents of each dataset folder freely and then run:

sai ds send [name]

This automatically packages it as a .zip and uploads it to the server. Importantly, files appear directly in the notebook or checker folder. If you create

task-folder/dataset-name/foo.bar

, for example, it will be available to the checker directly as

./foo.bar

(not, for example,

./dataset-name/foo.bar

). Sometimes, instead of creating a dataset made up of an entire file-and-folder structure, you want the dataset to be a single file. Rather than zipping one file and later having to unpack it, you can upload it directly with the following command, which reports an error if the dataset folder does not contain exactly one file:

sai ds send [name] --no-zip (-n)

To download a dataset back from the server (it is automatically unpacked if it was a zip), run:

sai ds pull [name]

You can also use:

sai ds pull --missing (-m)

to download all datasets linked to the task that you do not have locally.

SprawdzAI also has several global datasets that can be useful in many tasks and that we did not want to duplicate each time. You can find all of them in the Files tab. You cannot edit them, but you can link them to your task and download them. To link such an external dataset, simply use:

sai ds link [id] --notebook/--checker (-n/-c) --valid/--test (-v/-t)

For example:

sai ds link 28d12307-3880-4bdc-bbf6-68fefb3a434d

links the ResNet-18 file to all four phases.

Note. Sai uses folder names to determine which folder represents which dataset. If you rename a folder, Sai will no longer know which dataset that folder represents. Instead, if you want to rename a dataset (and therefore its folder), use:

sai ds rename [name] [new name]

For example, the commands for creating a standard dataset layout could be:

sai ds create train
sai ds create valid --checker --valid
sai ds create test --checker --test

statement.ipynb

sai sync

The first cell should be a Markdown cell containing exactly:
# <div align="center">Title</div> ---
If you want to retain information about the task author, put the following in the next cell:
<h4> Task author: ... </h4>
Do not put the entire task statement in one cell. Wherever possible, splitting different sections of the statement into separate cells will be much more convenient for you and for users.
Images should be included as HTML tags, for example:
<img width="500" src="https://sprawdzai.org/static/files/image/..." alt="Optional image description" />
We encourage you not to use images hosted elsewhere on the Internet (because their links may expire). Instead, add an image in the “images” section of the Files tab and copy its link.
Use gender-neutral forms where applicable, as they are used throughout most of the website.
In Polish, we also capitalize second-person forms (such as “Ci” and “Twój”) across most of the website, but whether to use them is left to the editor's personal preference.

baseline

sai

statement.ipynb

baseline

statement.ipynb

sai

baseline

baseline.ipynb

baseline

statement.ipynb

statement.ipynb

sai sync

image

baseline

image

requirements.txt

This is a standard

pip

dependency file. It is installed with uv using

uv pip install -r requirements.txt

, but in practice it behaves just like ordinary

pip

. In the debugging information for each submission (which you can access as an editor of the task), you will find any errors and logs generated while installing packages. We recommend pinning package versions to avoid surprises in the future.

Note. The

torch

torchvision

torchaudio

, and

triton

packages are installed separately. You cannot add versions or anything else to them. For each of these four packages, the line in requirements.txt must contain exactly, for example,

torch

and nothing else. You cannot write, for example,

torch=...

initiator.py

This file is run during the notebook-execution phase. In fact, the notebook itself is not launched directly;

initiator.py

is. Its standard contents are simply:

import os
import subprocess
os.remove('initiator.py')
subprocess.run(['python', '-u', 'submission.py'])

It deletes itself (so it does not leave clutter in the notebook folder) and then simply launches the submission. In 90% of cases, you should leave this file as it is.

So what is it for? You can use it, for example, to prepare the environment before running the notebook. To use global ResNet datasets through PyTorch, you must first place them in the

.cache

folder. The file might then look like this:

import os
import subprocess
import shutil

os.makedirs('/home/runner/.cache/torch/hub/checkpoints', exist_ok=True)
shutil.move('./resnet18-f37072fd.pth', '/home/runner/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth')

os.remove('initiator.py')
subprocess.run(['python', '-u', 'submission.py'])

If you want to use ResNets and other global datasets, be sure to review their list (you must be an editor of some task to access it).

You can also, for example:

Edit
submission.py
before running it.
Run it in a different way, for example with
import submission
or through
exec()
. This makes it possible to create tasks in the style of the first and third OAI editions, where users only create variables rather than save them to cloudpickle (after reading or importing them, you can save them yourself from
initiator.py
).
By running it differently, you can also alter its behavior in various ways, such as blocking access to certain libraries.

The checker

We finally reach the most interesting part - and fortunately it is not very difficult. We will divide the explanation of checker development into two parts. The first covers the whole system for cases where we do not run any code inside the checker. These are situations where answers are submitted directly and the notebook-execution phase is skipped (for example, Submit your score or Reverse operation), or where the notebook runs but receives test data as its execution-phase dataset and generates an answer file rather than a cloudpickle file. The “Introduction” section already explains why this approach is not preferred. Either way, the ability to validate answers in general is essential. The second part is an additional explanation of how to run cloudpickle files safely.

First of all, the checker for a new task will always have the following imports:

from checklib import IS_VALID, NOTEBOOK_EXECUTION_TIME, NOTEBOOK_TIME_LIMIT, CHECKER_TIME_LIMIT
from checklib import check_file_exists, linear_grade, safe_score, debug, format_time, print_tree, run_pickle
from checklib import SubmissionError, TimeError

checklib.py

is a helper library containing numerous flags and functions for writing checkers. The matching version appears automatically in the task folder. Do not edit this file; it is provided only as a reference so that you can inspect exactly how the available functions work when needed.

Your task is to implement the

check()

function. This function should return a number representing the task score, or a tuple containing a number and a string, where the number is the score and the string contains the submission notes (see

notes_char_limit

under metadata editing for more information):

def check() -> float | tuple[float, str]:
    score = ...
    notes = 'This message will be visible to users'
    return score, notes

The checker can return any score, including a floating-point number or a value outside the 0-100 range (it's not a bug, it's a feature). Most often, however, you want to return an integer from 0 to 100. To avoid simple mistakes in that case, we provide the

checklib.safe_score

decorator. It ensures that the score is an integer; if it exceeds 100, it is clamped to 100, and likewise it can never be less than 0.

@safe_score
def check() -> float | tuple[float, str]:
    return score, notes

You can also report time-limit and submission errors (only from inside the

check()

function!) simply by raising

checklib.SubmissionError

checklib.TimeError

, for example:

def check() -> float | tuple[float, str]:
    raise SubmissionError('This message will be visible to users')

Or without any additional message:

def check() -> float | tuple[float, str]:
    raise SubmissionError()

Similarly, you can report a time-limit error with

raise TimeError(optional message)

All submission files and all datasets are available to the checker in the same folder. For example, this is the checker for the Submit your score task:

def check():
    with open('score.txt', 'r') as f:
        res = f.read()

    try:
        score = int(res)
    except Exception:
        raise SubmissionError('The submitted score.txt file is not an integer')

    if score < 0 or score > 100:
        raise SubmissionError('The score must be between 0 and 100')

    return score

The entire notebook-execution folder is available to the checker under

./nb

. Thus, if the notebook creates

./foo.bar

, the checker can read it at

./nb/foo.bar

It is also worth discussing the remaining functions and flags in

checklib.py

IS_VALID
-
True
for submissions sent to the validation dataset and
False
for test submissions.
NOTEBOOK_EXECUTION_TIME
- the time, in seconds, taken by notebook execution. This is useful for raising time-limit errors when you want to restrict the combined duration of the pickle and notebook runs.
NOTEBOOK_TIME_LIMIT
- the notebook time limit, exactly as specified in the metadata, provided here purely for convenience.
CHECKER_TIME_LIMIT
- like the previous value, but for the checker.
check_file_exists(filepath: str) -> bool
- given a path string, returns whether the file exists (that is, whether the path exists and actually points to a file rather than a folder).
linear_grade(score: float, min_score: float, max_score: float, use_better: bool = False) -> int
- the standard function for grading on a linear scale.
score
is the metric achieved by the user in the submission,
min_score
is the metric value awarded 0 points, and
max_score
is the metric value awarded 100 points (everything works correctly even if
max_score
is lower than
min_score
). It returns a score according to the linear mapping [
min_score
,
max_score
] -> [0 points, 100 points].
use_better
enables more accurate but less commonly used rounding.
debug()
- works exactly like the built-in
print()
function but writes to STDERR. It can help separate certain logs more clearly. Everything the checker writes through
print()
or
debug()
is visible only to administrators and task editors; users see only the notes you leave for them.
format_time(t: float) -> str
- a helper function that converts a number of seconds
t
into a readable time representation. For example,
format_time(3667.4)
returns
1:01:07
, while
format_time(-69)
returns
-1:09
. It is used, for example, when displaying how long it took to run something.
print_tree(path: str = '.')
- neatly displays the file tree in a given folder, by default the current folder - the checker's
/sandbox
folder. Note. This function is intended for debugging and checking whether unexpected files appeared in the folder (see the section on running pickle files for more information). We recommend removing it before finally publishing the task, because it may take unnecessary time if, for example, the notebook generated a very large number of small files.
run_pickle
- see the section on running pickle files for more information.

Beyond that, there is not much more to add. Every possible error should be handled and reported as a

SubmissionError

. If the checker raises any other error and therefore fails to save a result file, the message

System error

appears. There should be no system errors in production. We regularly check for new system errors, because each one indicates that a checker was written incorrectly. System errors must be fixed immediately and the affected submissions subsequently rejudged. There should be no system errors in production. They may of course occur while a task is being created, and that is completely normal, but the final checker should handle every error properly and clearly tell the user what is wrong. We leave the rest of the checker implementation logic to you, because at that point it is simply a matter of implementing the task's idea.

Finally, let us explain exactly how the checker is run. Just as the notebook was launched through

initiator.py

, the checker is always launched through the same

checkrun.py

file. It contains exactly:

from checklib import IS_VALID, NOTEBOOK_EXECUTION_TIME, NOTEBOOK_TIME_LIMIT, CHECKER_TIME_LIMIT
from checklib import write_ok, write_time_error, write_submission_error
from checklib import SubmissionError, TimeError
from checker import check
import os
import shutil

print('This message is automatically printed by checkrun.py before running check():')
print(f'IS_VALID: {IS_VALID}')
print(f'NOTEBOOK_EXECUTION_TIME: {NOTEBOOK_EXECUTION_TIME}')
print(f'NOTEBOOK_TIME_LIMIT: {NOTEBOOK_TIME_LIMIT}')
print(f'CHECKER_TIME_LIMIT: {CHECKER_TIME_LIMIT}')

print('checkrun.py will first remove: checker.py, checklib.py, checkrun.py and __pycache__')
os.remove('checker.py')
os.remove('checklib.py')
os.remove('checkrun.py')
shutil.rmtree('__pycache__')

print('checkrun.py will now run check()...')

try:
    result = check()
    if isinstance(result, tuple):
        score, notes = result
        write_ok(score, notes)
    else:
        write_ok(result)
except SubmissionError as e:
    write_submission_error(str(e))
except TimeError as e:
    write_time_error(str(e))

In short, it:

Imports the
check
function from the checker (which is why errors raised outside
check()
are treated as system errors - only the call to
check()
is caught).
Prints the values of the helper flags so they are visible in every submission's logs.
Removes temporary files and every other file that could be used to hack the system while running a pickle (see the next section for more information).
Runs
check()
and saves the score or any errors.

Running cloudpickle

If you carefully read and understood the previous sections, running cloudpickle is now only a simple intermediate step. To recall the introduction, we must run a cloudpickle file in a separate process; otherwise, it could trivially do things such as assign itself a million points or cause highly unpredictable system errors. This also means that all answer files must first be loaded into variables and then removed before the pickle runs, since otherwise it could simply read them. In short, you need to complete these simple steps:

Load and remove the answer files.
Check that the cloudpickle file and every other expected file were actually created by the notebook.
Run the pickle in a separate process where its predictions are saved to a file, and report any errors to the user in the appropriate form.
Finally, check those files exactly as in the previous section.

The first two points are fairly obvious (the example below should resolve any remaining doubts), so we will move directly to the third. Run the pickle with the

run_pickle()

function. If you inspect

checklib.py

, you will see that this function does three things:

Runs
subprocess_script.py
- simply
python subprocess_script.py
.
Prints the exit code, STDOUT, and STDERR of
subprocess_script.py
.
Checks whether an
error.txt
file was created. If so, it raises
SubmissionError
with that file's contents; otherwise, it finishes and returns the time taken to execute the script.

Thus, all you need to do is write a

subprocess_script.py

that loads the cloudpickle file, loads the test data, runs the function on that data, and saves the result (and writes

error.txt

if an error occurs). The contents of

error.txt

can be completely arbitrary, so use a reasonably small

notes_char_limit

to protect against data leakage.

The

subprocess_script.py

file in the Simple classification task looks like this:

try:
    import cloudpickle
    import numpy as np

    with open('nb/predict.pkl', 'rb') as f:
        try:
            predict = cloudpickle.load(f)
        except Exception as e:
            raise Exception('Failed to load the cloudpickle file (is the file corrupted?) ' + repr(e))
    
    try:
        predicted = predict(np.load('snails_test.npy'))
    except Exception as e:
        raise Exception('An error occurred while running the cloudpickle file: ' + repr(e))

    try:
        if hasattr(predicted, 'cpu'):
            predicted = predicted.cpu()
        np.save('predictions.npy', predicted)
    except Exception as e:
        raise Exception('Failed to save the answers to a file (are you returning a NumPy array or tensor?) ' + repr(e))

except Exception as e:
    with open('error.txt', 'w', encoding='UTF-8') as f:
        f.write(str(e) if type(e).__name__ == 'Exception' else repr(e))
    raise  # raise so full logs will be visible on stderr

It is not particularly complicated: it does exactly what we described, running the cloudpickle function on test data and saving the results while handling every possible error. In this case, only the error type and its contents are saved. You can consider other options as well (they use more characters in the notes, so use them only if the notes character limit has been increased for some reason), such as also printing the line that caused the error:

try:
    predicted = pickle_function()
except Exception as e:
    import traceback
    tb = traceback.extract_tb(e.__traceback__)[-1]
    raise Exception('An error occurred while running the cloudpickle file: ' + repr(e) + ('' if tb.line is None) else '
' + str(tb.line))

or the entire traceback:

try:
    predicted = pickle_function()
except Exception as e:
    import traceback
    raise Exception(traceback.format_exc())

Next, inside the checker, simply follow the four steps listed earlier: verify the generated files, load and remove the answer files, run the pickle, and then check the generated files. Here is an example checker from the Simple classification task that you can use as a starting point:

from checklib import IS_VALID, NOTEBOOK_EXECUTION_TIME, NOTEBOOK_TIME_LIMIT, CHECKER_TIME_LIMIT
from checklib import check_file_exists, linear_grade, safe_score, debug, format_time, print_tree, run_pickle
from checklib import SubmissionError, TimeError
import numpy as np
import os

@safe_score
def check() -> float | tuple[float, str]:
    pickle_file = 'predict.pkl'
    
    # Load all test labels (files you don't want pickle function to access) into RAM and REMOVE THOSE FILES!
    labels = np.load('labels_test.npy')
    os.remove('labels_test.npy')
    print_tree()  # This prints all files, useful for checking if there are only files that the subprocess will need
    
    # Check if all files that the notebook should generate are actually here
    if not check_file_exists(f'nb/{pickle_file}'):
        raise SubmissionError(f'Could not find the {pickle_file} file')
    
    
    # Run pickle by calling subprocess_script.py
    # If it creates error.txt file, it will be raised as SubmissionError()
    # If running pickle exceeds timeout, it will raise TimeError()
    # You can handle these yourself to give additional error messages like in this example
    total_execution_time = NOTEBOOK_EXECUTION_TIME  # This will be the 'Total execution time'
    total_time_limit = NOTEBOOK_TIME_LIMIT  # This will be the time limit for total_execution_time
    total_execution_time += run_pickle(timeout = total_time_limit - total_execution_time)
    
    
    # Now just check the generated files.
    # Do not assume anything about the files, handle every exception!
    # If you don't run any pickles inside checker, just start from this section and remove all sections above.
    try:
        preds = np.load('predictions.npy')
    except Exception as e:
        print(e)
        raise SubmissionError('The predict() function did not return a NumPy array or tensor')
    
    if preds.shape != labels.shape:
        raise SubmissionError(f'Expected the output of {pickle_file} to have shape {labels.shape}, but received {preds.shape}')
    
    acc = (preds == labels).astype(float).mean()
    return linear_grade(acc, 0.5, 0.95, True), f'Classification accuracy: {acc * 100 :.1f}%, Total execution time: {format_time(total_execution_time)}'

The

run_pickle()

function automatically reports

SubmissionError

and

TimeError

errors when they occur. If you want to add information to them, you can handle them separately:

try:
    total_execution_time += run_pickle(timeout = total_time_limit - total_execution_time)
except SubmissionError as e:
    raise SubmissionError('You can add an extra message here: ' + str(e))
except TimeError as e:
    raise TimeError('Message shown when the time limit is exceeded')

Conclusion

Remember that you cannot assume anything about files generated by
subprocess_script.py
. Someone could maliciously save those files and then execute
sys.exit(0)
, allowing them to put absolutely anything in the files. If you believe that a particular error, should it occur, can only result from deliberate user abuse, you can even raise
SubmissionError
without a message. Just make sure there are no unexpected system errors.
Do not eliminate all system errors either. For example, do not wrap the entire
check()
function in a huge try...catch that always raises
SubmissionError
. System errors make it easy to monitor whether something happened that should never have occurred. You can therefore leave errors unhandled when you are certain they should not occur. If you turn out to be wrong, the system error will notify you clearly and quickly.
If your task runs a pickle inside the checker, we strongly recommend linking every training dataset used during notebook execution to the checker as well, in case someone tries to read it only from inside the pickle.

Bonus

Sai can also send submissions, which is very convenient because you do not have to switch to the browser and click through the submission flow. Submit with
sai sub PATHS...
, where PATHS... are the paths to the files you want to send, in the order specified by
submission_files
in the metadata. For example, if the task has only one submission file, you must provide exactly one path. Paths may be abbreviated (see
sai sub --help
for more information). For example, you can create a folder
./submissions
and send submissions from it using
sai sub s/[first letter of the file]
(or the first few letters if several files begin with the same letter). The submission is sent to the default dataset: validation if the daily validation-submission limit is not 0, and test otherwise. You can also add the flags:

sai sub PATHS... --valid/--test (-v/-t)

--all (-a)

--open (-o)