openml/openml-python

We recently switched the CI to run tests against local services using Docker instead of the remote test server. This change was introduced in: #1629 #1697 The documentation should now be updated to explain how contributors can run the test suite using these local services. Proposed changes: Update local service setup In the documentation for starting the local services, add the additional permission setup steps required before running docker compose, as implemented in #1697. Update the testing guide Extend the test guide with instructions on how to run pytest against the local services. This requires setting the environment variable: OPENML_USE_LOCAL_SERVICES

Several tests currently rely on direct connections to the live server (production/test), which can cause race conditions, server overload, and sporadic failures (see #1586). To ensure these remain true unit tests, they should use mocks/patches instead of the remote server. Steps: Identify tests using the remote server, e.g., marked with "production_server" or "test_server". Mock or patch requests.Session in tests relying on openml._api_calls._send_request. Alternatively, higher-level mocking can be used to simplify implementation. Affected test files (can be separate sub-tasks/PRs): test_datasets/test_dataset_functions.py test_datasets/test_dataset.py test_evaluations/test_evaluation_functions.py #1672 test_flows/test_flow_functions.py #1670 test_flows/test_flow.py test_openml/test_api_calls.py test_runs/test_run_functions.py test_runs/test_run.py test_runs/test_trace.py test_setups/test_setup_functions.py #1673 #1674 test_study/test_study_functions.py #1668

Currently the sdk doesnot allow setting up tags for setups, even though this is supported by the back-end. This is because it doesnot inherit the OpenMLBase class. Therefore this issue depends on #1647 Steps in utils/_openml.py: update _get_rest_api_type_alias to return valid alias for OpenMLSetup "setup" is already in legal_entities of _tag_entity so hopefully nothing changes there check that the result_xml in _tag_entity is applicable with the current parser Add tests for tagging/untagging setups.

OpenMLSetup should inherit and use methods from OpenMLBase, like every other resource, since it's an independent resource with dedicated schema in backend. Also this will enable tagging which is currently not supported. Steps: Inherit OpenMLBase in OpenMLSetup implement the abstract methods like id, _get_repr_body_fields, ... OpenMLSetup doesn't support publish, so handle that method accordingly Add a test file tests/test_setups/test_setup.py and write unitests for OpenMLSetup there Update docstrings. CC: @PGijsbers for approval

Several tests in test_run_functions.py have TODO comments indicating missing assertions (e.g., lines 170, 287, 291, 341, 514, 555). These include checks for holdout tasks, parameter presence, and runtime values, which would improve test reliability. Steps: Review each TODO comment and the surrounding test code to understand what's being tested. Add appropriate assertions, such as: assert task.task_type == "holdout" for holdout task checks. assert "runtime" in run.attributes for runtime presence. assert len(run.parameters) > 0 for parameter checks. Run the tests to ensure they pass and cover the intended behavior. If needed, mock or set up test data to make assertions deterministic. Tests will be more comprehensive, catching regressions in run handling and parameter validation.