python – theano hard_sigmoid()打破了梯度下降

对于突出问题的意图让我们按照这个tutorial.

theano有3种方法来计算张量的S形,即sigmoid,ultra_fast_sigmoid和hard_sidmoid.似乎使用后两种方法打破了梯度下降算法.

传统的sigmoid应该收敛,但其他sigmoid具有奇怪的不一致行为. ultra_fast_sigmoid,只是在尝试计算渐变’方法未定义(‘grad’,ultra_fast_sigmoid)’时抛出一个直接错误,而hard_sigmoid编译得很好,但无法收敛解决方案.

有谁知道这种行为的来源？在文档中没有强调这应该发生,它似乎反直觉.

码：

import theano
import theano.tensor as T
import theano.tensor.nnet as nnet
import numpy as np

x = T.dvector()
y = T.dscalar()

def layer(x,w):
    b = np.array([1],dtype=theano.config.floatX)
    new_x = T.concatenate([x,b])
    m = T.dot(w.T,new_x) #theta1: 3x3 * x: 3x1 = 3x1 ;;; theta2: 1x4 * 4x1

    h = nnet.sigmoid(m) ## THIS SIGMOID RIGHT HERE

    return h

def grad_desc(cost,theta):
    alpha = 0.1 #learning rate
    return theta - (alpha * T.grad(cost,wrt=theta))

theta1 = theano.shared(np.array(np.random.rand(3,3),dtype=theano.config.floatX))
theta2 = theano.shared(np.array(np.random.rand(4,1),dtype=theano.config.floatX))

hid1 = layer(x,theta1) #hidden layer

out1 = T.sum(layer(hid1,theta2)) #output layer
fc = (out1 - y)**2 #cost expression

cost = theano.function(inputs=[x,y],outputs=fc,updates=[
        (theta1,grad_desc(fc,theta1)),(theta2,theta2))])
run_forward = theano.function(inputs=[x],outputs=out1)

inputs = np.array([[0,1],[1,0],[0,0]]).reshape(4,2) #training data X
exp_y = np.array([1,1,0]) #training data Y
cur_cost = 0
for i in range(2000):
    for k in range(len(inputs)):
        cur_cost = cost(inputs[k],exp_y[k]) #call our Theano-compiled cost function,it will auto update weights
    if i % 500 == 0: #only print the cost every 500 epochs/iterations (to save space)
        print('Cost: %s' % (cur_cost,))

print(run_forward([0,1]))
print(run_forward([1,0]))
print(run_forward([0,0]))

我从代码中更改了以下行,以使该帖子的输出更短(它们与教程不同,但已包含在上面的代码中)：

from theano.tensor.nnet import binary_crossentropy as cross_entropy #imports
fc = cross_entropy(out1,y) #cost expression
for i in range(4000): #training iteration

乙状结肠

Cost: 1.62724279493
Cost: 0.545966632545
Cost: 0.156764560912
Cost: 0.0534911098234
Cost: 0.0280394147992
Cost: 0.0184933786794
Cost: 0.0136444190935
Cost: 0.0107482836159
0.993652087577
0.00848194143055
0.990829396285
0.00878482655791

ultra_fast_sigmoid

  File "test.py",line 30,in 

Cost: 1.19810193303
Cost: 0.684360309062
Cost: 0.692614056124
Cost: 0.697902474354
Cost: 0.701540531661
Cost: 0.703807604483
Cost: 0.70470238116
Cost: 0.704385738831
0.4901260624
0.486248177053
0.489490785078
0.493368670425