Technical details
What system are tasks evaluated on?
- All tasks are evaluated on Linux using the Ubuntu distribution. Unless stated otherwise for a particular task, we use Python 3.12.3.
How does task evaluation work?
- Your code runs in a restricted system isolated from the rest of SprawdzAI. Inside it, you can freely create and edit files (for example, temporary files in
/tmp
or answer files in the notebook directory). Task evaluation consists of two phases: running every notebook cell and then running the checker. These appear in a submission as "running" and "evaluating", respectively. The two phases are completely isolated from each other, which means that files created while running the notebook and environment changes are not preserved afterward. Depending on the task, the checker may calculate metrics from generated files or run cloudpickle
files created by the notebook.Why are there two phases? Why must I save a cloudpickle file that is run separately?
- By default, the notebook execution phase has access only to training data (the data users can access directly while solving the task). Validation or test data is provided only to the pickle that has already been prepared or trained in the notebook. This lets us provide all logs from notebook execution (
stdout
and stderr
); their length is limited only to prevent excessive output from filling our storage. Logs produced while running the cloudpickle
file are much more restricted. This gives users as much useful debugging information as possible without risking the exposure of our test data, which is especially important during contests. As a result, debugging errors that occur during evaluation (inside a function from cloudpickle
) can be considerably harder because the available information may not be as precise as it is during notebook execution.Why does the preparation phase sometimes take so long?
- During this phase, the system prepares all files and the entire environment so that nothing needs to interrupt execution later. This keeps the duration of subsequent phases reliable and fair. The duration of preparation generally does not depend on your submission, so there is no need to worry if it occasionally takes longer than usual. We continuously work to keep it as short as possible.
Why did my submission receive points even though notebook execution failed?
- After execution of all notebook cells begins, our system no longer considers exactly what happens during that execution. In particular, if notebook execution ends with an error but all evaluated files have already been saved, points will still be awarded for the task.
Can the same submission have different execution times?
- Due to numerous factors, even a submission with no randomness may report execution times differing by approximately 3 seconds when submitted several times, particularly for long-running submissions.
What processor are tasks run on?
- We currently use an AMD EPYC 7402 processor. Each task has access to 4 CPU cores.
Why is the GPU's total VRAM greater than the amount specified in the task?
- The VRAM limit is enforced by allocating memory in a separate process equal to the difference between the GPU's total memory and the required limit. For example, if a card has 40GB of VRAM and the limit should be 24GB, 16GB is allocated in advance to enforce that limit. Therefore, code still sees 40GB of total memory rather than 24GB.
What is the SprawdzAI file structure? How is the notebook run?
- All your submission files and the dataset files mentioned in a task are located in
/sandbox
. The python -u /sandbox/submission.py
command is also run from there (the -u
flag ensures notebook output is displayed correctly when a memory or time limit is exceeded). The Python environment is located in /venv
and added to PATH
. Lines in your notebook of the form {indentation}!{command}
are converted to:
{indentation}__import__('subprocess').run({command}, shell=True, check=True)