It could mean two things based on the context. 1. The boat is swaying back and forth <--- I think it is this one or 2. The boat keeps falling over and he has to get back in. I have to know more about the story in order to tell you exactly. Sorry!

The grammar in this sentence is not natural spoken English. It sounds old-fashioned. However, the two dashes on either side of the phrase "a short, stout figure" is a literary device to provide extra information that is not essential to the main point. It indicate two things. First that the "short, stout figure" is describing the rower. Second, this kind of punctuation (the dashes) indicates that if you remove the phrase in between the dashes from the sentence, the sentence will still make sense and retain it's meaning (minus the extra information between the dashes). If you remove the phrase "a short, stout figure," the sentence reads "...the rower, splashing badly and rolling a good deal, but trying his hardest." means that the rower is performing the actions of "splashing badly," and "rolling a good deal," but he's not "splashing badly" and "rolling a good deal" with his body, but since this phrase is attached to the phrase "a wager-boat flashed into view," it is implied that it is the boat that he is causing to "splash badly" and "roll around a good deal" in the water. This explanation was probably more convoluted than necessary, but I hope it was helpful.