Stochastic gradient descent implementation with Python's numpy
我必须使用 python numpy 库来实现随机梯度下降。为此,我给出了以下函数定义:
1 2 3 4 5 6 | def compute_stoch_gradient(y, tx, w): """Compute a stochastic gradient for batch data.""" def stochastic_gradient_descent( y, tx, initial_w, batch_size, max_epochs, gamma): """Stochastic gradient descent algorithm.""" |
我还获得了以下帮助功能:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | def batch_iter(y, tx, batch_size, num_batches=1, shuffle=True): """ Generate a minibatch iterator for a dataset. Takes as input two iterables (here the output desired values 'y' and the input data 'tx') Outputs an iterator which gives mini-batches of `batch_size` matching elements from `y` and `tx`. Data can be randomly shuffled to avoid ordering in the original data messing with the randomness of the minibatches. Example of use : for minibatch_y, minibatch_tx in batch_iter(y, tx, 32): <DO-SOMETHING> """ data_size = len(y) if shuffle: shuffle_indices = np.random.permutation(np.arange(data_size)) shuffled_y = y[shuffle_indices] shuffled_tx = tx[shuffle_indices] else: shuffled_y = y shuffled_tx = tx for batch_num in range(num_batches): start_index = batch_num * batch_size end_index = min((batch_num + 1) * batch_size, data_size) if start_index != end_index: yield shuffled_y[start_index:end_index], shuffled_tx[start_index:end_index] |
我实现了以下两个功能:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | def compute_stoch_gradient(y, tx, w): """Compute a stochastic gradient for batch data.""" e = y - tx.dot(w) return (-1/y.shape[0])*tx.transpose().dot(e) def stochastic_gradient_descent(y, tx, initial_w, batch_size, max_epochs, gamma): """Stochastic gradient descent algorithm.""" ws = [initial_w] losses = [] w = initial_w for n_iter in range(max_epochs): for minibatch_y,minibatch_x in batch_iter(y,tx,batch_size): w = ws[n_iter] - gamma * compute_stoch_gradient(minibatch_y,minibatch_x,ws[n_iter]) ws.append(np.copy(w)) loss = y - tx.dot(w) losses.append(loss) return losses, ws |
我不确定迭代应该在 range(max_epochs) 还是更大的范围内完成。我这样说是因为我读到一个纪元是"每次我们遍历整个数据集"。所以我认为一个时代包含多个迭代......
在典型的实现中,批量大小为 B 的小批量梯度下降应该从数据集中随机选择 B 个数据点,并根据该子集上计算的梯度更新权重。这个过程本身将持续很多次,直到收敛或某个阈值最大迭代。 B=1 的 Mini-batch 是 SGD,有时会很吵。
除了上述评论之外,您可能还想尝试一下批量大小和学习率(步长),因为它们对随机和小批量梯度下降的收敛速度有显着影响。
下图显示了在对亚马逊产品评论数据集进行情感分析时,这两个参数对
有关这方面的更多详细信息,您可以参考 https://sandipanweb.wordpress.com/2017/03/31/online-learning-sentiment-analysis-with-logistic-regression-via-stochastic-gradient-ascent /?frame-nonce=987e584e16