Re: [Modeling-users] ec.insert() timing

[Sebastien B. <sbi...@us...>]

        Hi Yannick,

Yannick Gingras <yan...@sa...> wrote:
> Hi Everyone,=20
>   I noticed a weird slowdown recently when creating records with the
> framework.  I quickly scattered print statements in my code to identify
> the bottle next and I fond ec.insert() to be the main culprit.  This
> is probably a locking problem since I have 0.1% CPU Load on MySQL,
> almost 0 traffic on the network and the system is 60% idle in the peek
> of a 100 record insert profiling session.  The 100 record insert takes
> a few minutes with a noticeable pause on the call to ec.insert() (but
> no pause on the call to ec.saveChanges()).
>=20
> On the profiler side everything seems OK with ec.insert() :=20
>=20
>   101    0.010    0.000    0.440    0.004=20
> /usr/lib/python2.2/site-packages/Modeling/EditingContext.py:866(insertObj=
ect)
>=20
> ec.saveChanges() seems to eat much more CPU:
>=20
>   102    0.090    0.001    8.290    0.081=20
> /usr/lib/python2.2/site-packages/Modeling/EditingContext.py:718(saveChang=
es)


That seems correct. However I can't see how a locking problem can occur
given your code (below)... Strange that you're 60% idle during the loop,
however.

> But as said previously, I don't notice any pause on the call to
> ec.saveChanges() as I do for ec.insertObject() .
>=20
> The timing code looks like this :=20
>=20
>       for i in range(100):
>           sys.stderr.write("%d\n" % i)
>           newRec =3D self._dbClass()
>=20
>           sys.stderr.write("a\n")
>           for key, value in self._defaults.items():
>               newRec.takeValueForKey(value, key)
>=20
>           sys.stderr.write("b\n")=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
>           self._ec.insert(newRec)
>           sys.stderr.write("c\n")=20=20=20=20=20=20=20=20=20=20=20=20=20=
=20=20=20
>           self._ec.saveChanges()
>           sys.stderr.write("d\n")
>=20
> I'm using MySQL 3.23.56, Python 2.2.2 and Modeling 0.9-pre-14.

Maybe sys.stderr is buffered, which may explain why you do not see the
"c\n" after insert, and why you do not notice any pause for
saveChanges() (which is very strange). Do you experience the same thing
with sys.stderr.flush() after write()?

Last, you probably know this already, but just in case: if you insert()
a bunch of objects in a batch, you'd probably want to saveChanges()
after having inserted every objects.

  Reassure me: when you say that <<The 100 record insert takes a few
  minutes>>, that's under profiling, isn't it?

-- S=E9bastien.

>=20
> Does anyone experienced such a slowdown on inserts ?
>=20
> Thanks in advance for any feed-back.
>=20
> --=20
> Yannick Gingras
> Byte Gardener, Savoir-faire Linux inc.
> http://www.savoirfairelinux.com/