uxlfoundation/scikit-learn-intelex

Split out of the review discussion on #3362, so the constraint has a tracking item rather than living only in a comment thread. What the code does today PyInit_mpi_transceiver exports a pointer to a std::shared_ptr<mpi_transceiver> (src/mpi/mpi_transceiver.cpp), and create_transceiver() reinterprets it as a std::shared_ptr<transceiver_iface> * (src/transceiver.cpp). Those are distinct types and the reinterpret_cast is not blessed in general. It works here only because the hierarchy happens to be a single non-virtual public inheritance chain with transceiver_iface as the primary base, so the stored pointer values coincide and the two shared_ptr instantiations have the same layout. That is a real constraint on the class hierarchy which nothing in the type system states. What has already been done #3362 pins the assumption rather than fixing it, because changing the module's exported contract does not belong in a PR about MPI lifecycle: two static_asserts, for the parts the type system c

In KNeighborsClassifier, method .predict_proba() is implemented in Python, as the underlying oneDAL class doesn't offer this method. When running in distributed mode, the nearest neighbors for a given point might not necessarily be in the same data fold that the rank is holding, which makes the implementation of such methods more challenging. As such, .predict_proba() throws a NotImplementedError when called in an SPMD KNeighborsClassifier object: #2700 For class predictions (calls to .predict()), oneDAL C++ would handle this in distributed mode by moving data among ranks: https://github.com/uxlfoundation/oneDAL/blob/main/cpp/oneapi/dal/algo/knn/backend/gpu/infer_kernel_impl_dpc_distr.hpp#L669 A similar logic could be implemented in Python through mpi4py, although it's unclear how efficient it would be. For example, it could use this function: https://mpi4py.readthedocs.io/en/latest/reference/mpi4py.MPI.Comm.html#mpi4py.MPI.Comm.Isendrecv_replace Or perhaps this: https://mpi4py.readthe