Hyperopt with SparkTrials will automatically track trials in MLflow. To view the MLflow experiment associated with the notebook, click the 'Runs' icon in the notebook context bar on the upper right. There, you can view all runs.
To view logs from trials, please check the Spark executor logs. To view executor logs, expand 'Spark Jobs' above until you see the (i) icon next to the stage from the trial job. Click it and find the list of tasks. Click the 'stderr' link for a task to view trial logs.
/databricks/spark/python/pyspark/rdd.py:980: FutureWarning: Deprecated in 3.1, Use pyspark.InheritableThread with the pinned thread mode enabled.
warnings.warn(
100%|██████████| 25/25 [23:56<00:00, 57.47s/trial, best loss: 0.3335099208282874]
Total Trials: 25: 25 succeeded, 0 failed, 0 cancelled.
Hyperparameter tuning
Since all experiments will be tried in parallel, we make our training and testing set accessible from each of our executors. This prevents our driver to send same information prior to any model experiment.