Explore the cave to Find Another Way into the Castle – Bloody Baron – The Witcher 3 Wild Hunt
From the entry of the cave swim to the ledge ahead and climb up. Follow the cave and climb up the ledge again. Drop down and turn right and jump up the the ledge:
Follow the path until you get to another ledge. Follow the path until you get to a ladder.