The current implementation of SelectTool is such that if you have a shape(AnnotatedRectangle) selected and you want to select another shape(AnnotatedRectangle) on the same BasePlot(), you have to click outside the shape, on the canvas, and into the another shape that you want to select to. This is the smoothest the SelectTool's flow can get. I double-checked the test script at plotpy\tests\tools\test_image_plot_tools.py and it has the same problem.
Does the SelectTool provide the ability to directly select another shape without having to unselect the initial selected shape?