[MINOR] make use of set_host_port() and get_host_port() to get rid of family mismatches

This also simplifies the code and makes it more auditable.
7 files changed