I think the first one means I had gone somewhere before I did sth else, while the latter one just express the place I went. I hope this will help you.