We often add the words 'down' or 'up' to a location, simply to show that it's a different place from where we are. Often, these words have no literal meaning : the other place is not necessarily higher up or lower down - it's just somewhere else. For example: 'I'm going down to the shops. Do you need anything?'

"down" is not necessary for the sentence to make sense. This usage of "down" is informal and adds a sense of familiarity and proximity with a place, here the Rialto.