why beta=0.1 zeta=0.01?

[iron9light]

I use this recommend params for my own model, the mean of noise scale is keeping about 1e-9 which is too small.

My model this something like below:

layer1 = multi_layer_complex_mess(input)  # layer1 shape: [batch_size, 512]
layer1_bn = batch_norm(layer1)
layer1_act = relu(layer1_bn)
layer2 = dense_with_sdr(layer1_act, unit=1024)
logit = dense(layer2, unit=3)

My origin modal is:

layer1 = multi_layer_complex_mess(input)  # layer1 shape: [batch_size, 512]
layer1_bn = batch_norm(layer1)
layer1_noise = layer1_bn + gaussian_noise(mean=0, stddev=noise_scale)
layer1_act = relu(layer1_noise)
layer2 = dense(layer1_act, unit=1024)
logit = dense(layer2, unit=3)

For the origin model, I tried noise_scale=0.5 and 1.0.
1.0 is better.

If beta and, especially zeta, need to be tune, then there's no much advantage against fix scale gaussian noise.