首页 文章

Spring Batch分区步骤未运行

提问于
浏览
1

我试图配置我的第一个多线程作业 . 我们有一个大约200,000条记录的主文件,我们需要处理这些记录 . 我想将文件分解为10个文件并处理它们 . 拆分文件tasklet工作正常,它将错误文件拆分为10个较小的文件 . 我将文件的路径传递给作业参数键中的作业"urlFilesPath",值为"file:/scrapper/spliturl*" . 我验证了 MultiResourcePartitioner bean中正确读取的文件 .

主步骤在我的配置中运行,但从属步骤不运行 . 以下是我的配置 .

Partitioner:

<bean id="partitioner"
    class="org.springframework.batch.core.partition.support.MultiResourcePartitioner"
    scope="step">
    <property name="resources" value="#{jobParameters['urlFilesPath']}" />
</bean>

MultiResourceItemReader:

<bean id="multiResourceItemReader"
    class="org.springframework.batch.item.file.MultiResourceItemReader"
    scope="step">
    <property name="resources" value="#{jobParameters['urlFilesPath']}" />
    <property name="delegate" ref="urlFileItemReader" />
    <property name="strict" value="true" />
    <property name="saveState" value="false" />
</bean>

FlatFileItemWriter:

<bean id="urlFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader"
scope="step">
    <property name="lineMapper" ref="passThroughLineMapper" />      
    <property name="resource" value="#{stepExecutionContext['fileName']}" /> 
    <property name="saveState" value="true" /> 
</bean>

Job Configuration:

<batch:job id="importJob" job-repository="jobRepository">
    <batch:step id="fileSplitter" next="readURLFileRunner">
        <batch:tasklet ref="fileSplittingTasklet"
            transaction-manager="transactionManager" />
    </batch:step>

    <batch:step id="readURLFileRunner">
        <batch:partition step="readURLFile" partitioner="partitioner">
            <batch:handler grid-size="10" task-executor="taskExecutor" />
        </batch:partition>
    </batch:step>
</batch:job>

Slave Step Configuration:

<batch:step id="readURLFile">
    <batch:tasklet transaction-manager="transactionManager"
        task-executor="taskExecutor" throttle-limit="10">
        <batch:chunk reader="multiResourceItemReader" processor="urlFileItemProcessor"
            writer="validURLItemWriter" commit-interval="200" skip-limit="100">
            <batch:skippable-exception-classes>
                <batch:include class="java.net.MalformedURLException" />
                <batch:include class="java.net.URISyntaxException" />
                <batch:include class="java.net.UnknownHostException" />
            </batch:skippable-exception-classes>
        </batch:chunk>
        <batch:listeners>
            <batch:listener ref="malformedURLExceptionListener" />
            <batch:listener ref="uriSyntaxExceptionListener" />
            <batch:listener ref="unknownHostExceptionListener" />
        </batch:listeners>
    </batch:tasklet>
    <batch:end on="COMPLETED" />
</batch:step>

请告诉我我做错了什么 . 我没有看到处理器 urlFileItemProcessor 和作者 validURLItemWriter 正在执行 .

Update 我按照@dimzak给出的答案 . 但我仍然没有看到正在执行的步骤 readURLFile 并且 urlFileItemProcessorvalidURLItemWriter 的 Logger 未打印到控制台 . 该作业在以下 Logger 之后挂起

org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate - Starting repeat context.

我对我配置步骤范围的方式有疑问 .

StepScope Configuration

<bean class="org.springframework.batch.core.scope.StepScope">
    <property name="autoProxy" value="true"/>
    <property name="proxyTargetClass" value="true"/>
</bean>

在Spring论坛上,我已经读过 multiResourceItemReaderdelegate 属性不需要是步骤范围的 . 当我从 urlFileItemReader 中删除 scope="step" 时,我得到以下异常 .

INFO : 26 Sep 2014 00:10:58,811 - org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Initializing ExecutorService  'taskExecutor'
INFO : 26 Sep 2014 00:10:59,066 - org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Shutting down ExecutorService 'taskExecutor'
Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'urlFileItemReader' defined in class path resource [beansBatchService.xml]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Field or property 'stepExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:547)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:703)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
    at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
    at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
    at com.chw.hma.service.batch.jobrunner.MainJobRunner.run(MainJobRunner.java:29)
    at com.chw.hma.service.batch.jobrunner.MainJobRunner.main(MainJobRunner.java:21)
Caused by: org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Field or property 'stepExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
    at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:146)
    at org.springframework.beans.factory.support.AbstractBeanFactory.evaluateBeanDefinitionString(AbstractBeanFactory.java:1364)
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.evaluate(BeanDefinitionValueResolver.java:214)
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:186)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1456)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1197)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
    ... 12 more
Caused by: org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Field or property 'stepExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
    at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:217)
    at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:85)
    at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:78)
    at org.springframework.expression.spel.ast.CompoundExpression.getValueRef(CompoundExpression.java:49)
    at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:85)
    at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:102)
    at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:94)
    at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:143)
    ... 18 more

请指教 .

Update Sep 27, 2014

根据@dimzak的建议发布任务执行程序等

<bean id="taskExecutor"
    class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
    <property name="corePoolSize" value="10" />
    <property name="maxPoolSize" value="10" />
</bean>

<bean id="jobRepository"
    class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
    <property name="transactionManager" ref="transactionManager" />
    <property name="dataSource" ref="dataSource" />
    <property name="databaseType" value="mySQL" />
</bean>

<bean id="jobLauncher"
    class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
    <property name="jobRepository" ref="jobRepository" />
    <property name="taskExecutor" ref="taskExecutor" />
</bean>

1 回答

  • 3

    MultiResourcePartitioner接受 file:/scrapper/spliturl* 并在不同的步骤中拆分每个文件,如spliturl1,spliturl2 .

    然后,在从属步骤中,您必须只读取此步骤的特定文件 . 所以你的配置应该没有 MultiResourceItemReader .

    <batch:step id="readURLFile">
        <batch:tasklet transaction-manager="transactionManager"
            task-executor="taskExecutor" throttle-limit="10">
            <batch:chunk reader="urlFileItemReader" processor="urlFileItemProcessor"
                writer="validURLItemWriter" commit-interval="200" skip-limit="100">
                <batch:skippable-exception-classes>
                    <batch:include class="java.net.MalformedURLException" />
                    <batch:include class="java.net.URISyntaxException" />
                    <batch:include class="java.net.UnknownHostException" />
                </batch:skippable-exception-classes>
            </batch:chunk>
            <batch:listeners>
                <batch:listener ref="malformedURLExceptionListener" />
                <batch:listener ref="uriSyntaxExceptionListener" />
                <batch:listener ref="unknownHostExceptionListener" />
            </batch:listeners>
        </batch:tasklet>
        <batch:end on="COMPLETED" />
    </batch:step>
    

相关问题