Menu
▾
▴
Re: [simspark-devel] SimSpark boosting project
|
From: Hedayat V. <hed...@gm...> - 2011-03-05 19:41:40
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Sander,<br>
<br>
<span>
<style type="text/css">blockquote {color: navy !important; background-color: RGB(245,245,245) !important; padding: 0 15 10 15 !important; margin: 15 0 0 0; border-left: #1010ff 2px solid;} blockquote blockquote {color: maroon !important; background-color: RGB(235,235,235) !important; border-left-color:maroon !important} blockquote blockquote blockquote {color: green !important; background-color: RGB(225,225,225) !important; border-left-color:teal !important} blockquote blockquote blockquote blockquote {color: purple !important; background-color: RGB(215,215,215) !important; border-left-color: purple !important} blockquote blockquote blockquote blockquote blockquote {color: teal !important; background-color: RGB(205,205,205) !important; border-left-color: green !important}</style><i><b>Sander
van Dijk <a class="moz-txt-link-rfc2396E"
href="mailto:sgv...@gm..."><sgv...@gm...></a></b></i>
wrote on 03/02/2011 9:11:10 PM +0350:</span><br>
<blockquote style="color: navy; background-color: rgb(245, 245,
245); padding-left: 15px; border-left: 2px solid rgb(16, 16,
255);"
cite="mid:AAN...@ma..."
type="cite">Hey guys,
<div><br>
</div>
<div>Another update: currently, with the ODE version I put now at
my page [1], and current SVN head of rcssserver3d, I am able to
run 9 vs 9 in real time on my i7 740QM laptop, with the agents
and monitor also running on the same machine. Having said that:</div>
</blockquote>
First, thanks a lot for making us informed about your progress.
Unfortunately I was a little busy these days and unable to follow
you fast enough :)<br>
It's great to hear about the results so far, and I'm really happy to
hear that the work of Hesham was not wasted. <br>
<br>
<br>
<blockquote style="color: navy; background-color: rgb(245, 245,
245); padding-left: 15px; border-left: 2px solid rgb(16, 16,
255);"
cite="mid:AAN...@ma..."
type="cite">
<div><br>
</div>
<div>- This is when the number of collisions is not too high.
Because of the extra constraints and the fact that they result
in agents being put into 1 one island, collisions are even worse
in multithreaded version. The select-and-move functionality of
RoboViz comes in quite handy here, but it's still a pain.</div>
</blockquote>
Is this a considerable issue even with the current automatic
referee? Are all agents able to collide with each other even with
this referee?<br>
<br>
<blockquote style="color: navy; background-color: rgb(245, 245,
245); padding-left: 15px; border-left: 2px solid rgb(16, 16,
255);"
cite="mid:AAN...@ma..."
type="cite">
<div>- Even worse: with many collisions ODE crashes. This seems to
be a memory issue, probably because ODE allocates everything on
the stack and runs out of space there. Indeed, configuring ODE
with -DdUSE_MALLOC_FOR_ALLOCA seems to prevent this issue by
using the heap instead. However, using the stack frequently is
suboptimal for multithreading, it might be worth it looking into
TBB's options to increase thread stack size.</div>
</blockquote>
We might be able to use a custom memory allocator too. So that we
can allocate a big block of memory from heap and assign it to ode in
an efficient manner. However, I don't know how ODE uses memory so my
suggestion might not be very practical.<br>
<br>
<blockquote style="color: navy; background-color: rgb(245, 245,
245); padding-left: 15px; border-left: 2px solid rgb(16, 16,
255);"
cite="mid:AAN...@ma..."
type="cite">
<div>- Multithreading in Simspark is still sometimes unstable, s
with 18 agents, eemingly on the communication/predicate
generation/parsing side. Looking into that (after the rather
enjoying 18 robot wrestling match currently going on on my
screen ;)</div>
</blockquote>
About the communication part, I'm going to take this issue very
seriously soon. <br>
<br>
Thanks,<br>
Hedayat<br>
<br>
<blockquote style="color: navy; background-color: rgb(245, 245,
245); padding-left: 15px; border-left: 2px solid rgb(16, 16,
255);"
cite="mid:AAN...@ma..."
type="cite">
<div><br>
</div>
<div>Sander</div>
<div><br>
</div>
<div><br>
<div class="gmail_quote">On Wed, Feb 23, 2011 at 7:43 PM, Sander
van Dijk <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:sgv...@gm...">sgv...@gm...</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
<div class="im">Hey all,<br>
<br>
I have worked on multithreading ODE. First of all thanks
for sending me Hesham Ebrahimi's work. He took basically
the same approach as I had, so it confirmed my ideas and
it helped identifying and verifying problematic places. I
also adopted his use if Intel's Thread Building Blocks
library. It is quite helpful; simple as OpenMP, but
doesn't respawn threads, which I ran into at my first
attempt. My implementation is slightly different, to make
sure that all tasks are created before spawning any of
them, which can be an issue when small tasks (like
stepping the ball) finish fast.<br>
<br>
</div>
I have uploaded the current result at[1]. Using it I got
similar results as Hesham with the same test he, i.e.
significantly less cpu cycles spent doing the world
stepping. However, although I double checked each part
multiple times, the physics seem a little less stable.
During most runs it is fine, but at some, especially with a
lot of agents, one blows up and the server crashes. I am
still digging into it, It is hard to reproduce, and not sure
yet if the multi threading makes it worse, but if any of you
want to do some test runs and see how it works for you, that
may help. Especially if it does blow up and you see a
pattern in when it happens.<br>
<br>
Thanks all,<br>
<font color="#888888"><br>
Sander<br>
</font><br>
[1] <a moz-do-not-send="true"
href="http://homepages.feis.herts.ac.uk/%7Esv08aav/ode-0.11.1-tbb.tar.gz"
target="_blank">http://homepages.feis.herts.ac.uk/~sv08aav/ode-0.11.1-tbb.tar.gz</a>
<div class="im"> <br>
<br>
PS don't forget to install tbb dev packages (though
configure should warn you about that too) and remake and
install simspark<br>
PS2 gmail won't let me tar.gz, that's why it's targz<br>
<br>
</div>
<div>
<div> </div>
<div class="h5">
<div class="gmail_quote"> On Wed, Feb 23, 2011 at 7:20
PM, Sander van Dijk <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:sgv...@gm..." target="_blank">sgv...@gm...</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt
0pt 0.8ex; border-left: 1px solid rgb(204, 204,
204); padding-left: 1ex;"> Hey all,<br>
<br>
I have worked on multithreading ODE. First of all
thanks for sending me Hesham Ebrahimi's work. He
took basically the same approach as I had, so it
confirmed my ideas and it helped identifying and
verifying problematic places. I also adopted his use
if Intel's Thread Building Blocks library. It is
quite helpful; simple as OpenMP, but doesn't respawn
threads, which I ran into at my first attempt. My
implementation is slightly different, to make sure
that all tasks are created before spawning any of
them, which can be an issue when small tasks (like
stepping the ball) finish fast.<br>
<br>
I have attached the current result here. Using it I
got similar results as Hesham with the same test he,
i.e. significantly less cpu cycles spent doing the
world stepping. However, although I double checked
each part multiple times, the physics seem a little
less stable. During most runs it is fine, but at
some, especially with a lot of agents, one blows up
and the server crashes. I am still digging into it,
It is hard to reproduce, and not sure yet if the
multi threading makes it worse, but if any of you
want to do some test runs and see how it works for
you, that may help. Especially if it does blow up
and you see a pattern in when it happens.<br>
<br>
Thanks all,<br>
<font color="#888888"><br>
Sander<br>
</font><br>
PS don't forget to install tbb dev packages (though
configure should warn you about that too) and remake
and install simspark<br>
PS2 gmail won't let me tar.gz, that's why it's targz
<div>
<div><br>
<br>
<div class="gmail_quote">On Sun, Feb 20, 2011 at
3:28 AM, Hedayat Vatankhah <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:hed...@gm..."
target="_blank">hed...@gm...</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:
0pt 0pt 0pt 0.8ex; border-left: 1px solid
rgb(204, 204, 204); padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000"> Hi,<br>
<br>
On ۱۱/۰۲/۲۰ 09:06, Sander van Dijk wrote:
<blockquote type="cite">....
<div>
<div class="gmail_quote">
<div><br>
</div>
<div>Yes, good point. I will make
sure to record all test details.
For now: I am mostly using
valgrind (with the callgrind and
helgrind tools in specific). I
first tried gprof, but then
everything was very unstable, but
maybe that's helped with the
current fixes.</div>
</div>
</div>
</blockquote>
Thanks.<br>
<br>
<br>
<blockquote type="cite">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin: 0pt 0pt 0pt 0.8ex;
border-left: 1px solid rgb(204, 204,
204); padding-left: 1ex;">
<div bgcolor="#ffffff"
text="#000000">...Also, Andreas
Seekircher has reported his
experience with multi-threaded
which also confirms that
multi-threaded mode is faster, but
also he has faced a problem which
deserves some attention:
<div><br>
<blockquote type="cite">However
there was a strange behavior,
that the simulation was
running quite fast on my
laptop with up to 9 agents and
the simulator was using more
than one core. When I started
the 10th agent it was getting
much slower and it seemed that
the simulator was then using
only one core (it was then
again the same speed like
without multi-threading). This
happened with 4 cores. On a
dual core system already the
5th agent slowed down the
simulation... Is this a known
issue? </blockquote>
I guess that in this situation,
ODE is the main bottleneck. But
that's just a guess.</div>
</div>
</blockquote>
<div>
<div><br>
</div>
<div>Yes, Andreas notified me of
that, too. This happens when
agents and server are run on the
same machine with multiple cores.I
have found that at some point, the
system's scheduler assigns an
agent to the same core as the
server (even though in practice
there is still room on another
core), so the server can't run at
full speed. With taskset(1) it is
possible to explicitly set a
process' CPU affinity, and by
starting the agents with e.g.
'taskset 2 ./start.sh localhost'
the server is able to take up 100%
again.</div>
</div>
</div>
</blockquote>
Great, thanks for the info. We might be
able to design a more general framework
for running agents using Linux CGroups and
maybe taking advantage of perf tools. But
I should investigate more before being
able to comment more on this issue.<br>
<br>
<br>
<blockquote type="cite">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote"
style="margin: 0pt 0pt 0pt 0.8ex;
border-left: 1px solid rgb(204, 204,
204); padding-left: 1ex;">
<div bgcolor="#ffffff"
text="#000000">
<div>
<blockquote type="cite">...<br>
</blockquote>
</div>
<div> Great. But it'd be probably
nice if we parallelize the
collision detection too.
Specially, it's computation time
will increase considerably when
two or more robots collide
(fortunately the new referee
doesn't allow many robots to
collide at the same place, but
with more players it is more
likely that we'll have
collisions in different part of
the field).<br>
</div>
</div>
</blockquote>
<div>
<div><br>
</div>
<div>From what I can tell so far it
seems that the main part of the
speed reduction due to collisions
is not caused by the collision
detection, but by the fact that it
adds many new constraints to the
LCP problem that ODE solves to
step the physics. However, I still
have to make a team of robots that
just run into each other to be
able to say that with certainty.
And you are right, it would be
nice in any case ;) <br>
</div>
</div>
</div>
</blockquote>
Thanks for the clarification. Yes IIRC
solving the LCP problem was a real
bottleneck. If it doesn't worth the
effort, we might skip this part for now. <br>
<div> <br>
<br>
<blockquote type="cite">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin: 0pt 0pt 0pt 0.8ex;
border-left: 1px solid rgb(204,
204, 204); padding-left: 1ex;">
<div bgcolor="#ffffff"
text="#000000">
<div>
<blockquote type="cite">So far
about what I am doing. Now,
I would also like something
from you guys ;-) First of
all, give the new stuff I
committed a good test.
Behaviour of the simulator
should still be the same,
but it could be that I
missed something and that
timing of messages is
slightly different, breaking
agents. Also, give the multi
threaded mode a good test,
see if you can make it
crash. And, finally, I will
be working full time on the
simulator for 1 1/2 months
more, if you think there is
anything that I may be able
to squeeze in there, do let
me know!<br>
</blockquote>
</div>
Certainly! :) <br>
I noticed something in your
recent commit: you've removed
the ugly busy-waiting loop in
SimControlThread, but wouldn't
it result in a faster simulation
when ODE has not much work to
do? The loop was there to make
sure that a cycle will last no
less than 0.02 (mSimStep). If
I'm not mistaken, it is now
possible for a cycle to finish
too soon. <br>
</div>
</blockquote>
<div><br>
</div>
<div>I think you refer to:</div>
<div>
<div><br>
</div>
<div> if
(isInputControl)</div>
<div> {</div>
<div> while
(int(mSumDeltaTime*100) <
int(mSimStep*100))</div>
<div>
controlNode->StartCycle();
// advance the time</div>
<div> }</div>
</div>
<div><br>
</div>
<div>? The only use I saw of that
was to keep updating the
InputControl to get messages from
the monitor while the physics are
updated. The time check is there
to stop doing this when the
physics are done. Without this
check, the InputControl (and the
other controls, AgentControl and
MonitorControl) wait for the
physics to be done anyway at the
next barrier, so the cycle can't
be finished too soon. And this
loop caused the most problems with
multi threading, because it
allowed the scene graph to be
changed while the physics were
still running on it.</div>
</div>
</blockquote>
</div>
Yes, I was referring to this loop.
Unfortunately InputControl doesn't exactly
do what it is supposed to. Beside handing
input, it also functions as the simulator
timer, which is very ambiguous (and I'm
going to replace it with another timer
implementation). This loop is not intended
to receive messages from the monitor
(you'll see the same loop in
SimulationServer::Cycle() which is run in
single threaded mode); it is a
busy-waiting loop to make sure that a
cycle will not last less than mSimStep. On
of the functions which
InputControl::StartCycle does is to
inspect SDL's timer and call
SimulationServer::AdvanceTime(), which in
turn updates mSumDeltaTime. Yes, doesn't
look good and was really problematic for
me to understand what happens completely
:P<br>
<br>
Personally, I'm planning to remove the use
of SDL timer altogether and switch to
using Boost's timing facilities, which is
much more cleaner and also doesn't need a
busy-waiting loop. <br>
<br>
Thanks,<br>
<font color="#888888"> Hedayat<br>
<br>
<br>
</font>
<blockquote type="cite">
<div>
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote"
style="margin: 0pt 0pt 0pt 0.8ex;
border-left: 1px solid rgb(204,
204, 204); padding-left: 1ex;">
<div bgcolor="#ffffff"
text="#000000"> <br>
There are some collaboration
opportunities with your work and
what I'm planning to do, but
I'll talk about them in a
separate email soon.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Looking forward to it :)</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin: 0pt 0pt 0pt 0.8ex;
border-left: 1px solid rgb(204,
204, 204); padding-left: 1ex;">
<div bgcolor="#ffffff"
text="#000000"> <br>
Thanks,<br>
<font color="#888888"> Hedayat</font><br>
</div>
</blockquote>
</div>
</div>
-- <br>
<div> Adaptive Systems Research Group<br>
Department of Computer Science<br>
University of Hertfordshire<br>
United Kingdom<br>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
Adaptive Systems Research Group<br>
Department of Computer Science<br>
University of Hertfordshire<br>
United Kingdom<br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
Adaptive Systems Research Group<br>
Department of Computer Science<br>
University of Hertfordshire<br>
United Kingdom<br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
Adaptive Systems Research Group<br>
Department of Computer Science<br>
University of Hertfordshire<br>
United Kingdom<br>
</div>
</blockquote>
</body>
</html>
|
