Long-press (or right click with mouse) on the annotation you want to delete WITHOUT first highlighting the text. Press/click on the blue arrow next to highlight/underline and then press/click remove highlight.
Right now this only works if the annotation you want to remove is not currently highlighted. I believe this is on the list to be fixed in the future.