[Hibernate-issues] [Hibernate-JIRA] Created: (HB-386) Hibernate crashes while populate large tables

[<leg...@at...>]

Message:

  A new issue has been created in JIRA.

---------------------------------------------------------------------
View the issue:

  http://opensource.atlassian.com/projects/hibernate/secure/ViewIssue.jspa?key=HB-386


Here is an overview of the issue:
---------------------------------------------------------------------
        Key: HB-386
    Summary: Hibernate crashes while populate large tables
       Type: Bug

     Status: Unassigned
   Priority: Major

    Project: Hibernate2
 Components: 
             core
   Versions:
             2.1
             2.0.3

   Assignee: 
   Reporter: Jeff Gager

    Created: Wed, 8 Oct 2003 6:18 AM
    Updated: Wed, 8 Oct 2003 6:18 AM
Environment: Dell 2Ghz Pentium 4 512MB RAM Running Windows XP.
Java 1.4.2
Eclipse 2.1.1

Description:
I am trying to write a simple java application that reads a large number of records from a Oracle database (using JDBC) and loads them into a mySQL database through hibernate.

The application crashes with java.lang.OutOfMemoryError.

As the application runs I can see it steadilly using up memory until it eventually falls over. I am commiting the Transaction, closing the Session and creating a new Session and Transaction every 1000 records. I think hibernate is either failing to release some kind of resource, or it is performing some action on the existing data before creating each record which is consuming more memory as more records are created.

I am not sure how to proceed, your help would be much appreciated.

Source follows;

package X
lots of import's

public class XLoader {

	private static Logger log = Logger.getLogger("X.XLoader");
	
	public static void main(String[] args) throws Throwable {

		log.info("Processing starting at " + new Date());

		Connection conn = null; 
		try {
			
			// Create the JDBC Driver
			final OracleDriver driver = new OracleDriver();
				
			// Load the Oracle JDBC driver
			DriverManager.registerDriver(driver);
		
			// Establish Read-only JDBC Connection to the PAS database
			conn = (Connection)DriverManager.getConnection("jdbc:oracle:oci8:@XXX", "XXX", "XXX");
			conn.setAutoCommit(false);

			// Loading...
			loadPerson(conn);

			log.info("Processing completed at " + new Date());

		} catch (Throwable e) {
			log.fatal("Unhandled Exception - processing halted\n" + e.toString());
			throw e;
		} finally {
			conn.close();
			System.exit(1);
			log.info("Finished at " + new Date());
		}
	}
	
	private static void loadPerson(Connection conn) throws Throwable {

		log.info("Processing people at " + new Date());

		// Declare statment and result set
		PreparedStatement stmt = null;
		ResultSet rset = null; 
		Session session = null;
		Transaction tx = null;
		SessionFactory sf = new Configuration().configure("/XX.xml").buildSessionFactory();
		
		try {

			// Load people 
			String sql = "BIG SELECT STATEMENT HERE";
					   
			log.info("Reading people at " + new Date());
			stmt = conn.prepareStatement(sql,
				ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
			rset = stmt.executeQuery();

			if (rset.getFetchSize() < 1)
				throw new SQLException("No People found");

			session = sf.openSession();
			tx = session.beginTransaction();
			log.info("Loading people at " + new Date());
			int counter = 0;
			String lastPayroll = "";
			Payroll payroll = null;

			while (rset.next()) {

				// Create new payrolls as we find them
            	String thisPayroll = rset.getString(1);
				if (!thisPayroll.equals(lastPayroll)) {
					payroll = new Payroll();
					payroll.setName(rset.getString(1));
					payroll.setFrequency(rset.getString(2));
					payroll.setStatus(rset.getString(3));
					payroll.setFirstPaydate(rset.getDate(4));
					payroll.setNextPaydate(rset.getDate(5));
					payroll.setPrevPaydate(rset.getDate(6));
					payroll.setRunDate(rset.getDate(7));
					payroll.setAccountName(rset.getString(8));
					payroll.setAccountNumber(rset.getInt(9));
					payroll.setSortcode(rset.getInt(10));
					payroll.setBacsName(rset.getString(11));
					payroll.setBacsNumber(rset.getInt(12));
					ArrayList payrollMemberships = new ArrayList();
					payroll.setMemberships(payrollMemberships);
					session.save(payroll);
            	}
            	lastPayroll = thisPayroll;

				// Create people
				Person person = new Person();
				person.setReference(rset.getString(13));
				person.setSurname(rset.getString(14));
				person.setSex(rset.getString(15).charAt(0));
				person.setForenames(rset.getString(16));
				person.setInitials(rset.getString(17));
				person.setTitle(rset.getString(18));
				person.setNiNumber(rset.getString(19));
				person.setBirth(rset.getDate(20));
				ArrayList personMemberships = new ArrayList();
				person.setMemberships(personMemberships);
				ArrayList personAddresses = new ArrayList();
				person.setAddresses(personAddresses);
				session.save(person);

				// Create memberships
				Membership membership = new Membership();
				membership.setStart(rset.getDate(21));
				membership.setStatus("Paying");
				membership.setPerson(person);
				membership.setPayroll(payroll);
				person.addMembership(membership);
				payroll.addMembership(membership);
				session.save(membership);
				
				// Create addresses
				Address address = new Address();
				address.setStart(new Date());
				address.setLine1(rset.getString(22));
				address.setLine2(rset.getString(23));
				address.setTown(rset.getString(24));
				address.setCounty(rset.getString(25));
				address.setPostcode(rset.getString(26));
				address.setPerson(person);
				person.addAddress(address);
				session.save(address);

				if ((++counter % 1000) == 0) {
					tx.commit();
					long payid = payroll.getId().longValue();
					session.close();
					session = sf.openSession();
					tx = session.beginTransaction();
					payroll = (Payroll)session.load(Payroll.class, new Long(payid));
					log.info("Loaded " + counter);
				}

			}
			
			tx.commit();
			session.close();
			sf.close();
			log.info("Total People Loaded " + counter);

			rset.close();
			stmt.close();

		// Handle errors
		} catch (Throwable e) {
			log.fatal(e.getMessage(), e);
			rset.close();
			stmt.close();
			if (sf != null) sf.close();
			if (session != null) session.close();
			if (tx != null) tx.rollback();
			throw e;
		}
		log.info("Finished processing people at " + new Date());
	}
}



---------------------------------------------------------------------
JIRA INFORMATION:
This message is automatically generated by JIRA.

If you think it was sent incorrectly contact one of the administrators:
   http://opensource.atlassian.com/projects/hibernate/secure/Administrators.jspa

If you want more information on JIRA, or have a bug to report see:
   http://www.atlassian.com/software/jira