Somehow I missed this earlier. Had to remove the rest of the ball joint from the lower control arm. Thankfully it was easy to do.
This is the thing to remove:
Just hit it with a hammer around the edges…
…until it starts to break free…
…then hit it some more until it’s completely out.
Some cleaning to do then it’s ready for a new ball joint to be installed…
See my replacing ball joints how-to article here.