Fix Server socket is Client dirty hack

[jotuel]

I think it is finally time to make this work properly. Even though it was suprisingly decent workaround.