Re: [Dev-C++] STL Map question

[Shane B. <ast...@te...>]

Actually, the program is making use of threads now, but all the memory each
thread is accessing, is independant of the memory the main thread is
accessing.  To be painfully clear, the threads are used only to offload the
intensive process of dumping 60,000 records to a MySQL database.  What
happens, is the database is dumped to a text file within a couple seconds,
then a thread is created to open each of the sql dump files made, and
read/import into the MySQL database.  Functionally, it works perfectly and
the database has never had a problem dumping.  The ONLY thing shared between
the threads, is a call to a function "Log" which is used once per thread, to
output to the logs that it's finished.  Realizing the potential, a mutex
should be used on the Log class to prevent any accidental access at the same
time, but in the future I intend on pulling out the calls to "Log" and at
such a time there will be no use of shared memory, and thus no reason for
threads to be of concern.

    As is stands now, I am not willing to believe that the shared call to
Log is the culprit, as it would be a more random occurance of crashes, not
always at the exact same point in the code.  In fact, the point it's
crashing at, the threads never use.  The threads don't even iterate the
map's, they simply open an already closed file after it's been dumped, and
then reads the file, imports to the database, and delete's the sql dump
files to indicate it is done (rather than using shared memory, the file not
existing is just as well).

After speaking with a few people, it appears to me that the problem is not
at all with STL, at least not the fault of STL, but rather of original
implementation. The original implementation used some very obscure direct
manipulation of array's, such that it would zero out array's when they
weren't used, yet left them allocated. And in some other cases, would
actually call realloc to reallocate the memory.  This caused a little
confusion in the conversion to STL, with regards to performing the identical
operation with STL, and not changing the code calling on STL too
significantly.  The truely strange problem is that between the two changes
of the code, the UserList code was not changed, other code that calls it
was, and so therein lies the problem. However, I am not skilled enough to
know of any good tools that can help me determine why the pointer being
accessed is invalid.  More particularily, if there was a debugging tool for
checking memory, like the last time it was used before the crash occured, if
it could show me where it was free'd, or overwritten or whatever is
happening, I could most likely determine the cause, given a location as
general as a function it's occuring in.  Unfortunately, I have scanned over
the code numerous times and can only find 1 location where a User class is
delete'd.  And calls to that function appear to be called appropriately with
consideration for STL.

I am truely baffled, but I have given up blaming STL for the issue, there
has to be an overrun of memory writing, or somehow, something has
invalidated the memory that an STL map is still using.  There is no reason
for this, and another very seasoned programmer has looked at the code with
no immediate noticable problems.  If I wasn't in a position of
non-disclosure (actually, it's more like on my word that I won't release the
code, I can discuss the problems as I need to with other programmers), I
would tarball the code in question.  However, with as complicated as the
code is, even a vetran programmer wouldn't want to dig into it without a lot
of patience and time, of which I could not ask of anyone anyways.

I thank everyone for exhausting their ideas, even if this thread ended up
more of a flame war about who thinks what the best book is.  Unfortunately,
this problem is not resolved by any amount of books, regardless of how good
they are.  The problem is in implementation, not lack of understanding.  I
am almost certain that the map iterator has become invalid, and that's
causing the problem. But why, is the question regarding implementation.

A last note, just to get a couple opinions.  The original code that worked,
would delete users as they were disconnected. The newer code, using the same
concept for UserList, crashes.  What I have done (but not yet tested) is
changed the code where it delete's the User, instead of deleting them, it
sets a flag indicating they are ready to be processed for deletion. At the
top of every loop iteration, it generates a vector of Killed connections,
then steps through the vector and deletes them from the user list map.  The
change in implementation of how the User is deleted may lend itself to
fixing the problem in certain very strange conditions.  Does anyone see a
flaw with how I changed the code? I haven't tested it yet, beyond a
controlled environment where the crash wouldn't occur anyways.  I'm hoping
with little hope that it may simply be a matter of iterator invalidation,
and by not deleting the User and removing it right away, it might (??? I'm
really running out of ideas) fix the problem.

Sincerely,
    -Shane

----- Original Message ----- 
From: "Jaap de Wolff" <wo...@ne...>
To: "'Shane Bryldt'" <ast...@te...>; "Hossein Haeri"
<ult...@em...>
Cc: "C++" <dev...@li...>
Sent: Monday, July 14, 2003 6:36 AM
Subject: RE: [Dev-C++] STL Map question


> Hmm, I had problems like this also.
> Are you using a multithreaded environment?
> Is it possible that sometimes two threads at the same time are accessing
the
> map,
> one for removing a key, and one for finding a key?
>
> Jaap
>
> > -----Original Message-----
> > From: Shane Bryldt [mailto:ast...@te...]
> > Sent: Friday, July 11, 2003 10:53 PM
> > To: Hossein Haeri
> > Cc: C++
> > Subject: Re: [Dev-C++] STL Map question
> >
> ...
> >
> > Now, the effect of the issue, is as follows. A network of 6,000-10,000
> > active users at any given time is regulated by a security
> > service.  This
> > service runs and connects like a user, with special
> > priviledges.  Anytime a
> > user connects to the network, the services is informed, and
> > the User is
> > added to a global UserList.  The network also has smaller
> > units, known as
> > Channels.  Each Channel contains 3 UserList's as well.
> > First, depicts all
> > users in the Channel, the second depicts all the operators of
> > the channel,
> > and the third depicts the priviledged users who are not
> > operators.  99% of
> > the time, the code work fine.  Users constantly added and
> > removed from the
> > list.  Periodically, what seems like an issue without a
> > pattern, the code
> > will crash with a segmentation fault while trying to call
> > "std::map<>::find()", with a valid key.  Whether the key is
> > in the map, is
> > not definate, but the key passed in valid and should not have
> > a problem
> > being searched.
> >
> > This said, I have narrowed the issue down to one of 2 causes.
> > 1) Either
> > User's are being added to lists, and not being removed, and
> > the UserLists
> > are eventually gathering enough garbage that out of memory is being
> > encountered. The application has a signal hook for out of
> > memory, and it is
> > not being raised.  So I am leaning more towards option 2) The
> > User class is
> > being deleted, without being removed from the UserList.
> > However, I can find
> > no indication of this in the code.  The code is many
> > thousands of lines, it
> > is not a small project by any means.  Providing the source
> > for the issue,
> > would require the entire project as the issue is not
> > specifically a syntax
> > issue, but rather a situational issue.  I do believe something has
> > invalidated a User class, which a UserList has a pointer to,
> > and when trying
> > to use it, is causing a segmentation fault.  I'm sure most of
> > the experts
> > here will agree with my conclusion and realize tracing this
> > problem is no
> > easy task.
> >
> >
> ...
>