首页 文章

使用Spring Boot和JPA批处理 - 对与批处理相关的表使用内存数据源

提问于
浏览
2

Context

我正在尝试使用JPA Repository开发使用Spring Boot的批处理服务 . 使用两个不同的数据源,我想要在内存数据库中创建与批处理相关的表,这样它就不会污染我的业务数据库 .

关于网络上的多个主题,我想出了我的两个数据源的配置:

@Configuration
public class DataSourceConfiguration {

    @Bean(name = "mainDataSource")
    @Primary
    @ConfigurationProperties(prefix="spring.datasource")
    public DataSource mainDataSource(){
        return DataSourceBuilder.create().build();
    }

    @Bean(name = "batchDataSource")
    public DataSource batchDataSource( @Value("${batch.datasource.url}") String url ){
        return DataSourceBuilder.create().url( url ).build();
    }   
}

第一个 mainDataSource 使用默认的Spring数据库配置 . batchDataSource 定义了一个嵌入式HSQL数据库,我希望在其中创建批处理和步骤表 .

# DATASOURCE (DataSourceAutoConfiguration & DataSourceProperties)
spring.datasource.url=jdbc:mariadb://localhost:3306/batch_poc
spring.datasource.username=root
spring.datasource.password=
spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
spring.datasource.max-age=10000

spring.datasource.initialize=false

# JPA (JpaBaseConfiguration, HibernateJpaAutoConfiguration)
spring.jpa.generate-ddl=false
spring.jpa.show-sql=true
spring.jpa.database=MYSQL

# SPRING BATCH (BatchDatabaseInitializer)
spring.batch.initializer.enabled=false

# ----------------------------------------
# PROJECT SPECIFIC PROPERTIES
# ----------------------------------------

# BATCH DATASOURCE
batch.datasource.url=jdbc:hsqldb:file:C:/tmp/hsqldb/batchdb

这是我的批量配置:

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

    private static final Logger LOG = Logger.getLogger( BatchConfiguration.class );

    @Bean
    public BatchConfigurer configurer(){
        return new CustomBatchConfigurer();
    }

    @Bean
    public Job importElementsJob( JobBuilderFactory jobs, Step step1 ){
        return jobs.get("importElementsJob")
                .incrementer( new RunIdIncrementer() )
                .flow( step1 )
                .end()
                .build();               
    }

    @Bean
    public Step step1( StepBuilderFactory stepBuilderFactory, ItemReader<InputElement> reader,
            ItemWriter<List<Entity>> writer, ItemProcessor<InputElement, List<Entity>> processor ){

        return stepBuilderFactory.get("step1")
                .<InputElement, List<Entity>> chunk(100)
                .reader( reader )
                .processor( processor )
                .writer( writer )
                .build();
    }

    @Bean
    public ItemReader<InputElement> reader() throws IOException {       
        return new CustomItemReader();
    }

    @Bean
    public ItemProcessor<InputElement, List<Entity>> processor(){
        return new CutsomItemProcessor();
    }

    @Bean
    public ItemWriter<List<Entity>> writer(){
        return new CustomItemWriter();
    }

}

BatchConfigurer,使用内存数据库:

public class CustomBatchConfigurer extends DefaultBatchConfigurer {

    @Override
    @Autowired
    public void setDataSource( @Qualifier("batchDataSource") DataSource dataSource) {
        super.setDataSource(dataSource);
    }

}

最后,我的作家:

public class CustomItemWriter implements ItemWriter<List<Entity>> {

    private static final Logger LOG = Logger.getLogger( EntityWriter.class );

    @Autowired
    private EntityRepository entityRepository;

    @Override
    public void write(List<? extends List<Entity>> items)
            throws Exception {
        if( items != null && !items.isEmpty() ){

            for( List<Entity> entities : items ){
                for( Entity entity : entities ){                        
                    Entity fromDb = entityRepository.findById( entity.getId() );

                    // Insert
                    if( fromDb == null ){
                        entityRepository.save( entity );
                    }

                    // Update
                    else {
                        // TODO : entityManager.merge()
                    }
                }
            }

        }
    }

}

EntityRepository接口扩展了JpaRepository .

Problem

当我以这种方式分离数据源时,当我调用存储库的save方法时没有任何反应 . 我在日志中看到来自findById()调用的select查询 . 但没有什么可以拯救 . 我的输出数据库最后是空的 .

当我回到一个独特的数据源配置(删除配置文件bean并让Spring Boot单独管理数据源)时,插入查询工作正常 .

也许主数据源配置不足以让JPA正确执行插入 . 但缺少什么?

3 回答

  • 9

    我终于解决了在Spring类BasicBatchConfigurer的基础上实现我自己的BatchConfigurer的问题,并强制使用基于Map的jobRepository和jobExplorer . 没有更多的自定义数据源配置,只有一个我让Spring Boot管理的数据源:这样更容易 .

    我的自定义BatchConfigurer:

    public class CustomBatchConfigurer implements BatchConfigurer {
    
        private static final Logger LOG = Logger.getLogger( CustomBatchConfigurer.class );
    
        private final EntityManagerFactory entityManagerFactory;
    
        private PlatformTransactionManager transactionManager;
    
        private JobRepository jobRepository;
    
        private JobLauncher jobLauncher;
    
        private JobExplorer jobExplorer;
    
        /**
         * Create a new {@link CustomBatchConfigurer} instance.
         * @param entityManagerFactory the entity manager factory
         */
        public CustomBatchConfigurer( EntityManagerFactory entityManagerFactory ) {
            this.entityManagerFactory = entityManagerFactory;
        }
    
        @Override
        public JobRepository getJobRepository() {
            return this.jobRepository;
        }
    
        @Override
        public PlatformTransactionManager getTransactionManager() {
            return this.transactionManager;
        }
    
        @Override
        public JobLauncher getJobLauncher() {
            return this.jobLauncher;
        }
    
        @Override
        public JobExplorer getJobExplorer() throws Exception {
            return this.jobExplorer;
        }
    
        @PostConstruct
        public void initialize() {
            try {
                // transactionManager:
                LOG.info("Forcing the use of a JPA transactionManager");
                if( this.entityManagerFactory == null ){
                    throw new Exception("Unable to initialize batch configurer : entityManagerFactory must not be null");
                }
                this.transactionManager = new JpaTransactionManager( this.entityManagerFactory );
    
                // jobRepository:
                LOG.info("Forcing the use of a Map based JobRepository");
                MapJobRepositoryFactoryBean jobRepositoryFactory = new MapJobRepositoryFactoryBean( this.transactionManager );
                jobRepositoryFactory.afterPropertiesSet();
                this.jobRepository = jobRepositoryFactory.getObject();
    
                // jobLauncher:
                SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
                jobLauncher.setJobRepository(getJobRepository());
                jobLauncher.afterPropertiesSet();
                this.jobLauncher = jobLauncher;
    
                // jobExplorer:
                MapJobExplorerFactoryBean jobExplorerFactory = new MapJobExplorerFactoryBean(jobRepositoryFactory);
                jobExplorerFactory.afterPropertiesSet();
                this.jobExplorer = jobExplorerFactory.getObject();
            }
            catch (Exception ex) {
                throw new IllegalStateException("Unable to initialize Spring Batch", ex);
            }
        }
    
    }
    

    我的配置类现在看起来像这样:

    @Configuration
    @EnableBatchProcessing
    public class BatchConfiguration {
    
        @Bean
        public BatchConfigurer configurer( EntityManagerFactory entityManagerFactory ){
            return new CustomBatchConfigurer( entityManagerFactory );
        }
    
        [...]
    
    }
    

    我的属性文件:

    # DATASOURCE (DataSourceAutoConfiguration & DataSourceProperties)
    spring.datasource.url=jdbc:mariadb://localhost:3306/inotr_poc
    spring.datasource.username=root
    spring.datasource.password=root
    spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
    spring.datasource.max-age=10000
    
    spring.datasource.initialize=true
    
    # JPA (JpaBaseConfiguration, HibernateJpaAutoConfiguration)
    spring.jpa.generate-ddl=false
    spring.jpa.show-sql=true
    spring.jpa.database=MYSQL
    
    # SPRING BATCH (BatchDatabaseInitializer)
    spring.batch.initializer.enabled=false
    
  • 0

    感谢上面的帖子!在过去的几天里,我一直在努力让我的Spring Boot with Batch应用程序与基于内存映射的作业存储库和无资源的事务管理器一起工作 . (我不能让Spring Batch使用我的应用程序数据源用于Batch元数据表,因为我没有DDL访问权限来创建BATCH_表)最后在查看上面的帖子后得到了以下配置并且它工作得很好! !

    public class CustomBatchConfigurer implements BatchConfigurer {
    
        private static final Logger LOG = LoggerFactory.getLogger(CustomBatchConfigurer.class);
    
       // private final EntityManagerFactory entityManagerFactory;
    
        private PlatformTransactionManager transactionManager;
    
        private JobRepository jobRepository;
    
        private JobLauncher jobLauncher;
    
        private JobExplorer jobExplorer;
    
        /**
         * Create a new {@link CustomBatchConfigurer} instance.
         * @param entityManagerFactory the entity manager factory
    
        public CustomBatchConfigurer( EntityManagerFactory entityManagerFactory ) {
            this.entityManagerFactory = entityManagerFactory;
        }
    */
        @Override
        public JobRepository getJobRepository() {
            return this.jobRepository;
        }
    
        @Override
        public PlatformTransactionManager getTransactionManager() {
            return this.transactionManager;
        }
    
        @Override
        public JobLauncher getJobLauncher() {
            return this.jobLauncher;
        }
    
        @Override
        public JobExplorer getJobExplorer() throws Exception {
            return this.jobExplorer;
        }
    
        @PostConstruct
        public void initialize() {
            try {
                // transactionManager:
                LOG.info("Forcing the use of a Resourceless transactionManager");
    
                this.transactionManager = new ResourcelessTransactionManager();
    
                // jobRepository:
                LOG.info("Forcing the use of a Map based JobRepository");
                MapJobRepositoryFactoryBean jobRepositoryFactory = new MapJobRepositoryFactoryBean( this.transactionManager );
                jobRepositoryFactory.afterPropertiesSet();
                this.jobRepository = jobRepositoryFactory.getObject();
    
                // jobLauncher:
                SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
                jobLauncher.setJobRepository(getJobRepository());
                jobLauncher.afterPropertiesSet();
                this.jobLauncher = jobLauncher;
    
                // jobExplorer:
                MapJobExplorerFactoryBean jobExplorerFactory = new MapJobExplorerFactoryBean(jobRepositoryFactory);
                jobExplorerFactory.afterPropertiesSet();
                this.jobExplorer = jobExplorerFactory.getObject();
            }
            catch (Exception ex) {
                throw new IllegalStateException("Unable to initialize Spring Batch", ex);
            }
        }
    
    }
    

    以下是我在作业配置类中添加的bean

    @Bean
    public BatchConfigurer configurer(){
        return new CustomBatchConfigurer();
    }
    
  • 1

    Eria的回答有效!但是,我已将其修改为使用:

    org.springframework.batch.support.transaction.ResourcelessTransactionManager

    来自CustomBatchConfigurer:

    @PostConstruct
        public void initialize() {
            try {
                // transactionManager:
                LOGGER.info("Forcing the use of ResourcelessTransactionManager for batch db");
    
                this.transactionManager = new ResourcelessTransactionManager(); 
                //the rest of the code follows... 
        }
    

相关问题