Friday, September 11, 2009

Retry Spring Batch job due to database deadlock

I encountered the problem with Spring Batch where 2 jobs are started at the exact same time and a database deadlock occurs in Spring Batch database. It causes one job to fail as it was chosen as a deadlock victim.

com.rhd.ams.batch.AmsCommandLineBatchRunner: Job Terminated in error:
org.springframework.dao.DataAccessResourceFailureException: Could not increment identity; nested exception is java.sql.SQLException: Transaction (Process ID 114) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at org.springframework.jdbc.support.incrementer.SqlServerMaxValueIncrementer.getNextKey(SqlServerMaxValueIncrementer.java:107) [spring-jdbc-2.5.6.jar:2.5.6]
at org.springframework.jdbc.support.incrementer.AbstractDataFieldMaxValueIncrementer.nextLongValue(AbstractDataFieldMaxValueIncrementer.java:125) [spring-jdbc-2.5.6.jar:2.5.6]
at org.springframework.batch.core.repository.dao.JdbcJobInstanceDao.createJobInstance(JdbcJobInstanceDao.java:66) [spring-batch-core-1.1.0.RELEASE.jar:na]
at org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:183) [spring-batch-core-1.1.0.RELEASE.jar:na]


There are other people encountered the same problem as well.
http://forum.springsource.org/archive/index.php/t-54954.html

I took suggestions from this link and changed the isolation level in a configuration file to be ISOLATION_READ_UNCOMMITTED or REPEATABLE_READ but still it doesn't solve the problem, although we got fewer deadlock occurrences.

So, I took a different approach by retrying a failed job instead. Spring provides RetryOperationsInterceptor. See http://static.springsource.org/spring-batch/trunk/reference/html-single/index.html#retry.

It can be done easily by just declaring retry operations in a configuration file:

   

org.springframework.dao.DeadlockLoserDataAccessException
org.springframework.dao.DataAccessResourceFailureException


















3 comments:

Trey said...

Thank you for this. I am having the exact same problem, and there are no other solutions posted besides fiddling w/ the transaction isolation level. The Spring Batch developers should really make this a high priority issue.

Unknown said...

Nice solution. Thank you very much.

Anonymous said...

This is a good one, I ran into the same problem and learned from your post, thanks!

An article I want to share too:

Deadlock in database