Let's sit in the shade. (Any shade.)
Let's sit in the shadow of that tree. (The shadow of a particular object.)
If you say, "Let's sit in the shadow", you are referring to a particular shadow cast by a particular object. If you say, "Let's sit in the shade", all you care about is that you're not in the sun anymore.
Shade is used as a general term for a zone where there is no light.
Shadow, on the other hand, is used when refering to the obstruccion of light created by an especific object.