如何强制caffe读取所有训练数据？

我正在使用caffe和STILL输入有问题 .

这是我的solver.prototxt：

train_net: "auto_train.prototxt"
test_net: "auto_test.prototxt"
test_iter: 800
test_interval: 20
base_lr: 0.01
momentum: 0.9
weight_decay: 0.0005
lr_policy: "inv"
gamma: 0.0001
power: 0.75
display: 100
max_iter: 10000
snapshot: 5000
snapshot_prefix: "sed"
solver_mode: GPU

这是正在运行的python脚本：

import os

PROJECT_HOME = '/home/romulus/code/project/'
CAFFE_HOME = '/home/romulus/code/caffe/'
os.chdir(PROJECT_HOME)

import sys
sys.path.insert(0, CAFFE_HOME + 'caffe/python')
import caffe, h5py
from pylab import *
from caffe import layers as L, params as P

OUTPUT_DIM = 8

def net(db, batch_size):
    n = caffe.NetSpec()
    n.data, n.label = L.Data(batch_size=batch_size, backend=P.Data.LEVELDB, source=db,
                             transform_param=dict(scale=1./255), ntop=2)
    n.ip1 = L.InnerProduct(n.data, num_output=500, weight_filler=dict(type='xavier'))
    n.relu1 = L.ReLU(n.ip1, in_place=True)
    n.ip2 = L.InnerProduct(n.relu1, num_output=500, weight_filler=dict(type='xavier'))
    n.relu2 = L.ReLU(n.ip2, in_place=True)
    n.ip3 = L.InnerProduct(n.relu2, num_output=OUTPUT_DIM, weight_filler=dict(type='xavier'))
    n.loss = L.SoftmaxWithLoss(n.ip3, n.label)

    return n.to_proto()


with open('/home/romulus/code/project/auto_train.prototxt', 'w') as f:
    f.write(str(net('/home/romulus/code/project/traindb', 64)))
with open('/home/romulus/code/project/auto_test.prototxt', 'w') as f:
    f.write(str(net('/home/romulus/code/project/testdb', 100)))

caffe.set_device(0)
caffe.set_mode_gpu()
solver = caffe.SGDSolver(PROJECT_HOME + 'auto_solver.prototxt')

solver.net.forward()  # train net
solver.test_nets[0].forward()  # test net (there can be more than one)

niter = 500
test_interval = 15
train_loss = zeros(niter)
test_acc = zeros(int(np.ceil(niter * 1.0 / test_interval)))
output = zeros((niter, 8, OUTPUT_DIM))

for it in range(niter):
    solver.step(1)  # SGD by Caffe

    train_loss[it] = solver.net.blobs['loss'].data
    solver.test_nets[0].forward(start='ip1')
    output[it] = solver.test_nets[0].blobs['ip3'].data[:8]

    if it % test_interval == 0:
        print 'Iteration', it, 'testing...'
        correct = 0

        for test_it in range(1):
            solver.test_nets[0].forward()
            correct += sum(solver.test_nets[0].blobs['ip3'].data.argmax(1)
                           == solver.test_nets[0].blobs['label'].data)
        test_acc[it // test_interval] = correct * 1.0 / len(data)

_, ax1 = subplots()
ax2 = ax1.twinx()
ax1.plot(arange(niter), train_loss)
ax2.plot(test_interval * arange(len(test_acc)), test_acc, 'r')
ax1.set_xlabel('iteration')
ax1.set_ylabel('train loss')
ax2.set_ylabel('test accuracy')
_.savefig('converge.png')

手动生成数据，每个数据为1x256向量，所有相同的比例值为8 * label value . 这意味着，标签3的数据是[24,24,24,24,24 ......，24,24] . 我有8个标签和80000个数据 .

我的问题是，如果我将数据放入带有 0,1,2,3,4,5,6,7,8,0,1,2,3,4,5... 等标签顺序的leveldb，那么caffe可以很好地训练网络 . 但如果按照 0,0,...,0,0,1,1,1,...,1,1,2,2,... 命令，caffe就无法学习 . 如果我将solver.prototxt中的 test_iter 减少为100，则caffe将始终说输出标签为0 .

似乎caffe不会读取所有的训练数据，而只会读取前面的内容 . 但除了培训批次之外，我找不到任何描述它的内容 .

事实上，如果我将训练批量增加到80000，那么caffe会再次训练 . 虽然它很慢而且不是所谓的批次 .

有人可以帮忙吗？谢谢！