From the entry of the cave swim to the ledge ahead and climb up. Follow the cave and climb up the ledge again. Drop down and turn right and jump up the the ledge:

Jump Up to the Ledge to find Another Way into the Castle in Bloody Baron in Witcher 3 Wild Hunt.

Follow the path until you get to another ledge. Follow the path until you get to a ladder.