[Sqlalchemy-commits] [3993] sqlalchemy/trunk: - added very rudimentary yielding iterator behavior t
Brought to you by:
zzzeek
Menu
▾
▴
[Sqlalchemy-commits] [3993] sqlalchemy/trunk: - added very rudimentary yielding iterator behavior to Query.
|
From: <co...@sq...> - 2008-01-02 23:13:07
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head><meta http-equiv="content-type" content="text/html; charset=utf-8" /><style type="text/css"><!-- #msg dl { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; } #msg dt { float: left; width: 6em; font-weight: bold; } #msg dt:after { content:':';} #msg dl, #msg dt, #msg ul, #msg li, #header, #footer { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; } #msg dl a { font-weight: bold} #msg dl a:link { color:#fc3; } #msg dl a:active { color:#ff0; } #msg dl a:visited { color:#cc6; } h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; } #msg pre { overflow: auto; background: #ffc; border: 1px #fc0 solid; padding: 6px; } #msg ul, pre { overflow: auto; } #header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; } #patch { width: 100%; } #patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;} #patch .propset h4, #patch .binary h4 {margin:0;} #patch pre {padding:0;line-height:1.2em;margin:0;} #patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;} #patch .propset .diff, #patch .binary .diff {padding:10px 0;} #patch span {display:block;padding:0 10px;} #patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;} #patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;} #patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;} #patch .lines, .info {color:#888;background:#fff;} --></style> <title>[3993] sqlalchemy/trunk: - added very rudimentary yielding iterator behavior to Query.</title> </head> <body> <div id="msg"> <dl> <dt>Revision</dt> <dd>3993</dd> <dt>Author</dt> <dd>zzzeek</dd> <dt>Date</dt> <dd>2008-01-02 18:13:02 -0500 (Wed, 02 Jan 2008)</dd> </dl> <h3>Log Message</h3> <pre>- added very rudimentary yielding iterator behavior to Query. Call query.yield_per(<number of rows>) and evaluate the Query in an iterative context; every collection of N rows will be packaged up and yielded. Use this method with extreme caution since it does not attempt to reconcile eagerly loaded collections across result batch boundaries, nor will it behave nicely if the same instance occurs in more than one batch. This means that an eagerly loaded collection will get cleared out if it's referenced in more than one batch, and in all cases attributes will be overwritten on instances that occur in more than one batch.</pre> <h3>Modified Paths</h3> <ul> <li><a href="#sqlalchemytrunkCHANGES">sqlalchemy/trunk/CHANGES</a></li> <li><a href="#sqlalchemytrunklibsqlalchemyormquerypy">sqlalchemy/trunk/lib/sqlalchemy/orm/query.py</a></li> <li><a href="#sqlalchemytrunktestormquerypy">sqlalchemy/trunk/test/orm/query.py</a></li> </ul> </div> <div id="patch"> <h3>Diff</h3> <a id="sqlalchemytrunkCHANGES"></a> <div class="modfile"><h4>Modified: sqlalchemy/trunk/CHANGES (3992 => 3993)</h4> <pre class="diff"><span> <span class="info">--- sqlalchemy/trunk/CHANGES 2008-01-02 01:57:44 UTC (rev 3992) +++ sqlalchemy/trunk/CHANGES 2008-01-02 23:13:02 UTC (rev 3993) </span><span class="lines">@@ -1,6 +1,20 @@ </span><span class="cx"> ======= </span><span class="cx"> CHANGES </span><span class="cx"> ======= </span><ins>+0.4.3 +----- +- orm + - added very rudimentary yielding iterator behavior to Query. Call + query.yield_per(<number of rows>) and evaluate the Query in an + iterative context; every collection of N rows will be packaged up + and yielded. Use this method with extreme caution since it does + not attempt to reconcile eagerly loaded collections across + result batch boundaries, nor will it behave nicely if the same + instance occurs in more than one batch. This means that an eagerly + loaded collection will get cleared out if it's referenced in more than + one batch, and in all cases attributes will be overwritten on instances + that occur in more than one batch. + </ins><span class="cx"> 0.4.2 </span><span class="cx"> ----- </span><span class="cx"> - sql </span></span></pre></div> <a id="sqlalchemytrunklibsqlalchemyormquerypy"></a> <div class="modfile"><h4>Modified: sqlalchemy/trunk/lib/sqlalchemy/orm/query.py (3992 => 3993)</h4> <pre class="diff"><span> <span class="info">--- sqlalchemy/trunk/lib/sqlalchemy/orm/query.py 2008-01-02 01:57:44 UTC (rev 3992) +++ sqlalchemy/trunk/lib/sqlalchemy/orm/query.py 2008-01-02 23:13:02 UTC (rev 3993) </span><span class="lines">@@ -50,6 +50,7 @@ </span><span class="cx"> self._limit = None </span><span class="cx"> self._statement = None </span><span class="cx"> self._params = {} </span><ins>+ self._yield_per = None </ins><span class="cx"> self._criterion = None </span><span class="cx"> self._having = None </span><span class="cx"> self._column_aggregate = None </span><span class="lines">@@ -106,7 +107,22 @@ </span><span class="cx"> q = self._clone() </span><span class="cx"> q._current_path = path </span><span class="cx"> return q </span><ins>+ + def yield_per(self, count): + """yield only ``count`` rows at a time. </ins><span class="cx"> </span><ins>+ WARNING: use this method with caution; if the same instance + is present in more than one batch of rows, end-user changes + to attributes will be overwritten. + In particular, it's usually impossible to use this setting with + eagerly loaded collections (i.e. any lazy=False) since those + collections will be cleared for a new load when encountered + in a subsequent result batch. + """ + q = self._clone() + q._yield_per = count + return q + </ins><span class="cx"> def get(self, ident, **kwargs): </span><span class="cx"> """Return an instance of the object based on the given </span><span class="cx"> identifier, or None if not found. </span><span class="lines">@@ -749,12 +765,12 @@ </span><span class="cx"> </span><span class="cx"> def _execute_and_instances(self, querycontext): </span><span class="cx"> result = self.session.execute(querycontext.statement, params=self._params, mapper=self.mapper, instance=self._refresh_instance) </span><del>- try: - return iter(self.instances(result, querycontext=querycontext)) - finally: - result.close() </del><ins>+ return self.iterate_instances(result, querycontext=querycontext) </ins><span class="cx"> </span><span class="cx"> def instances(self, cursor, *mappers_or_columns, **kwargs): </span><ins>+ return list(self.iterate_instances(cursor, *mappers_or_columns, **kwargs)) + + def iterate_instances(self, cursor, *mappers_or_columns, **kwargs): </ins><span class="cx"> session = self.session </span><span class="cx"> </span><span class="cx"> context = kwargs.pop('querycontext', None) </span><span class="lines">@@ -808,26 +824,39 @@ </span><span class="cx"> else: </span><span class="cx"> raise exceptions.InvalidRequestError("Invalid column expression '%r'" % m) </span><span class="cx"> </span><del>- context.progress = util.Set() - if tuples: - rows = util.OrderedSet() - for row in cursor.fetchall(): - rows.add(tuple(proc(context, row) for proc in process)) - else: - rows = util.UniqueAppender([]) - for row in cursor.fetchall(): - rows.append(main(context, row)) </del><ins>+ while True: + context.progress = util.Set() + + if self._yield_per: + fetch = cursor.fetchmany(self._yield_per) + if not fetch: + return + else: + fetch = cursor.fetchall() + + if tuples: + rows = util.OrderedSet() + for row in fetch: + rows.add(tuple(proc(context, row) for proc in process)) + else: + rows = util.UniqueAppender([]) + for row in fetch: + rows.append(main(context, row)) </ins><span class="cx"> </span><del>- if context.refresh_instance and context.only_load_props and context.refresh_instance in context.progress: - context.refresh_instance.commit(context.only_load_props) - context.progress.remove(context.refresh_instance) </del><ins>+ if context.refresh_instance and context.only_load_props and context.refresh_instance in context.progress: + context.refresh_instance.commit(context.only_load_props) + context.progress.remove(context.refresh_instance) </ins><span class="cx"> </span><del>- for ii in context.progress: - context.attributes.get(('populating_mapper', ii), _state_mapper(ii))._post_instance(context, ii) - ii.commit_all() - - return list(rows) - </del><ins>+ for ii in context.progress: + context.attributes.get(('populating_mapper', ii), _state_mapper(ii))._post_instance(context, ii) + ii.commit_all() + + for row in rows: + yield row + + if not self._yield_per: + break + </ins><span class="cx"> def _get(self, key=None, ident=None, refresh_instance=None, lockmode=None, only_load_props=None): </span><span class="cx"> lockmode = lockmode or self._lockmode </span><span class="cx"> if not self._populate_existing and not refresh_instance and not self.mapper.always_refresh and lockmode is None: </span></span></pre></div> <a id="sqlalchemytrunktestormquerypy"></a> <div class="modfile"><h4>Modified: sqlalchemy/trunk/test/orm/query.py (3992 => 3993)</h4> <pre class="diff"><span> <span class="info">--- sqlalchemy/trunk/test/orm/query.py 2008-01-02 01:57:44 UTC (rev 3992) +++ sqlalchemy/trunk/test/orm/query.py 2008-01-02 23:13:02 UTC (rev 3993) </span><span class="lines">@@ -443,7 +443,27 @@ </span><span class="cx"> ] == q.all() </span><span class="cx"> self.assert_sql_count(testbase.db, go, 1) </span><span class="cx"> </span><del>- </del><ins>+ +class YieldTest(QueryTest): + def test_basic(self): + import gc + sess = create_session() + q = iter(sess.query(User).yield_per(1).from_statement("select * from users")) + + ret = [] + self.assertEquals(len(sess.identity_map), 0) + ret.append(q.next()) + ret.append(q.next()) + self.assertEquals(len(sess.identity_map), 2) + ret.append(q.next()) + ret.append(q.next()) + self.assertEquals(len(sess.identity_map), 4) + try: + q.next() + assert False + except StopIteration: + pass + </ins><span class="cx"> class TextTest(QueryTest): </span><span class="cx"> def test_fulltext(self): </span><span class="cx"> assert [User(id=7), User(id=8), User(id=9),User(id=10)] == create_session().query(User).from_statement("select * from users").all() </span></span></pre> </div> </div> </body> </html> |