Why using relu before softmax?

[wang975679801]

In the line 196,
gate_network = tf.contrib.layers.fully_connected( inputs=input, num_outputs=subexpert_nums, activation_fn=tf.nn.relu, \ weights_regularizer=l2_reg)
and then gate_network = tf.nn.softmax(gate_network, axis=1)
It is confusing.

[JJplane]

这个地方确实写错，如果domain的embedding 矩阵是0初始化的话，relu出来的值是0，relu的梯度也会是0，会导致domain的embedding永远都不会更新.....太坑了

In the line 196, gate_network = tf.contrib.layers.fully_connected( inputs=input, num_outputs=subexpert_nums, activation_fn=tf.nn.relu, \ weights_regularizer=l2_reg) and then gate_network = tf.nn.softmax(gate_network, axis=1) It is confusing.