首页 文章

MyBatis散货运营商的表现

提问于
浏览
1

我使用过MyBatis-spring Java . 我需要在一个事务中将> 10000条记录插入表中 . 为此,我使用了一个映射器:

<insert id="saveBulk" parameterType="List">
    INSERT INTO "quote" ("id", "mi_id", "timestamp", "open", "close", "low", "high", "volume", "period")
    VALUES
    <foreach collection="list" item="item" separator=",">
        ( #{item.key}, #{item.marketInstrumentKey}, #{item.timestamp}, #{item.open}, #{item.close}, #{item.low},
        #{item.high}, #{item.volume}, #{item.period}::quote_period)
    </foreach>
</insert>

并将List传递给此语句 . 对于2000-3000条记录,它的工作速度非常慢,但10000条记录的插入时间超过4分钟(我应该增加超时间隔)!通过PgAdmin将相同的10000条记录插入同一个DB中的时间少于10秒 . 我试图跟踪这个操作的处理并发现了一个瓶颈

public int doUpdate(MappedStatement ms, Object parameter) throws SQLException {
    Statement stmt = null;
    try {
      Configuration configuration = ms.getConfiguration();
      StatementHandler handler = configuration.newStatementHandler(this, ms, parameter, RowBounds.DEFAULT, null, null);
      stmt = prepareStatement(handler, ms.getStatementLog());
      return handler.update(stmt);
    } finally {
      closeStatement(stmt);
    }
  }

StatementHandler的计算时间为几分钟,而prepareStatement的计算时间为几分钟 . 我明白了,为什么会这样:10000条记录,每条记录有9个字段 . 所有这些100k字段都应作为参数插入到语句中 . 我怎样才能加速这个过程?

更新:

我使用sqlFactory和@Transactional的“BATCH”模式实现批量保存 . 这是mybatis-spring XML配置的配置:

<bean id="sqlBatchTemplate" class="org.mybatis.spring.SqlSessionTemplate">
        <constructor-arg index="0" ref="sqlSessionFactory"/>
        <constructor-arg index="1" value="BATCH"/>
    </bean>

    <bean id="quoteBatchMapper" class="org.mybatis.spring.mapper.MapperFactoryBean">
        <property name="mapperInterface" value="tafm.dataaccess.mybatis.mapper.QuoteMapper"/>
        <property name="sqlSessionFactory" ref="sqlSessionFactory"/>
        <property name="sqlSessionTemplate" ref="sqlBatchTemplate"/>
    </bean>
   <bean id="dataAccessBatch" class="tafm.dataaccess.DataAccess">
        <property name="quoteMapper" ref="quoteBatchMapper"/>
    </bean>

然后,我实现了一个“批处理”方法:

@Transactional
    public void saveBulk(List<Quote> quotes) {


        for(Quote q:quotes) {
            mapper.save(q);
        }
    }

mapper - 是实体Quote的XML映射器:

<insert id="saveBulk" parameterType="List">
    INSERT INTO "quote" ("id", "mi_id", "timestamp", "open", "close", "low", "high", "volume", "period")
    VALUES
    <foreach collection="list" item="item" index="index" separator=",">
        ( #{item.key}, #{item.marketInstrumentKey}, #{item.timestamp}, #{item.open}, #{item.close}, #{item.low},
        #{item.high}, #{item.volume}, #{item.period}::quote_period)
    </foreach>
</insert>

它工作得很快

1 回答

  • 1

    你可以做这样的事情 .
    Mapper.xml

    <insert id="saveBulk" parameterType="List">
        INSERT INTO "quote" ("id", "mi_id", "timestamp", "open", "close", "low", "high", "volume", "period")
        VALUES
        <foreach collection="list" item="item" separator=",">
            ( #{item.key}, #{item.marketInstrumentKey}, #{item.timestamp}, #{item.open}, #{item.close}, #{item.low},
            #{item.high}, #{item.volume}, #{item.period}::quote_period)
        </foreach>
    </insert>
    

    In Java File

    public void insertBulkData(List<Item> list)
        {
            int total = list.size();        
            int interval = 1000;
            int from = 0;
            int to = 0;
            while (to <= total)
            {
                from = to == 0 ? 0 : to;
                to = (to + interval) <= total ? (to + interval) : total;
                saveBulk(list.subList(from, to));
                if (to == total)
                    break;
            }
        }
    

    上面的代码使用1000的间隔 . 一次插入1000条记录 . 你也可以修改这个间隔 .
    你必须拨打 insertBulkData(list) .
    我希望这对你有所帮助 .

相关问题