The gray area is probably the distant relatives that you no longer feel as family. Technically they are family, but the feeling isn't there, Thus the area is gray.

"in the family,the gray area is expanding" i think it's means that , in the family , something strange ( that it unclear ) is spreading as if the family members are not acting not like they usually do A grey area: is a term for a border in-between two or more things that is unclearly defined, a border that is hard to define or even impossible to define, or a definition where the distinction border tends to move. There are several flavors of grey areas: Grey areas are widely accepted in democratic societies and have a clear connection to the notion of tolerance, whereas in societies of totalitarianism, grey areas are typically not accepted on any level. Many people accept grey areas as a natural part of the human experience, whereas others may react with suspicion and a feeling of defectness or uncompleteness of any thought-system (or paradigm) accepting grey areas. http://www.wordiq.com/definition/Gray_area

This sounds like a general comment on how a family is structured. So I guess a modern family unit is no longer based on the "2 parents+children" and there are more examples of different situations, which makes the unclear (grey/gray) part larger. So a clear definition of "the family" is not as easy to explain. The "grey area" of anything is the area where things are not sure or definite. As opposed to seeing things clearly: "in black and white".

a gray area is an area that is not clear or certain or something that is open to interpretation. We would need more information to be more specific.