To find another way into the castle ride out of the town. On the other side of the bridge turn right. At the next intersection turn right again. Follow the path until you get to a shrine:

Find Another Way into the Castle in Bloody Baron in The Witcher 3 Wild Hunt

Get off the horse and go into the river. Go into the cave ahead:

Find another way into the castle in bloody baron in the Witcher 3 Wild Hunt