From: Fredrik T. <fr...@do...> - 2007-01-04 02:22:01
|
Hi list! A bit over a year ago, I mailed about problems with really large chat rooms in Gaim. I was writing a prpl where chat rooms can easily have over 5000 users, and the largest problem was that it takes a really long time to add all these users to the user list in the chat window. I was told that this would be fixed in Gaim 2, so I waited. However, I'm right now in the process of porting the prpl to Gaim 2, and it seems to me that it has gotten a lot worse instead. Back then, it used to take around half a minute for the largest rooms, which was merely annoying. Right now, Gaim 2 has been taking over 8 minutes of CPU time trying to add ~6300 users, with no sign of stopping any time soon. Needless to say, that goes beyond merely being annoying. Is there a way to fix this? Maybe I've done something wrong? Right now, I'm just adding users to the user list. Do I have to call some function to temporarily disable continuous updates or something similar? Fredrik Tolf |
From: Martin O. <pah...@gm...> - 2007-01-04 10:17:19
|
Just noting that in addition to all that - retrieving the full list of rooms available on some IRC server pretty much is a bad idea also. On 1/4/07, Fredrik Tolf <fr...@do...> wrote: > > Hi list! > > A bit over a year ago, I mailed about problems with really large chat > rooms in Gaim. I was writing a prpl where chat rooms can easily have > over 5000 users, and the largest problem was that it takes a really long > time to add all these users to the user list in the chat window. I was > told that this would be fixed in Gaim 2, so I waited. > > However, I'm right now in the process of porting the prpl to Gaim 2, and > it seems to me that it has gotten a lot worse instead. Back then, it > used to take around half a minute for the largest rooms, which was > merely annoying. Right now, Gaim 2 has been taking over 8 minutes of CPU > time trying to add ~6300 users, with no sign of stopping any time soon. > Needless to say, that goes beyond merely being annoying. Is there a way > to fix this? > > Maybe I've done something wrong? Right now, I'm just adding users to the > user list. Do I have to call some function to temporarily disable > continuous updates or something similar? > > Fredrik Tolf > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys - and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Gaim-devel mailing list > Gai...@li... > https://lists.sourceforge.net/lists/listinfo/gaim-devel > -- Martin |
From: Richard L. <rl...@wi...> - 2007-01-04 19:14:24
|
On Thu, 2007-01-04 at 03:21 +0100, Fredrik Tolf wrote: > However, I'm right now in the process of porting the prpl to Gaim 2, and > it seems to me that it has gotten a lot worse instead. Back then, it > used to take around half a minute for the largest rooms, which was > merely annoying. Right now, Gaim 2 has been taking over 8 minutes of CPU > time trying to add ~6300 users, with no sign of stopping any time soon. > Needless to say, that goes beyond merely being annoying. Is there a way > to fix this? Can you duplicate this with IRC? I haven't seen any problems since this was addressed. Are you calling gaim_conv_chat_add_user() or gaim_conv_chat_add_users()? Adding them one at a time is going to result in 3 * N lists being created and destroyed, which could be the source of your problem. You should use the plural version for room joining. Note that the users and flags lists are required, but extra_msgs can be NULL if your PRPL doesn't use that feature. I'm wondering if we should make flags optional. If your protocol doesn't use flags, you should hack libgaim/conversation.c in gaim_conv_chat_add_users (around line 1527), to make these changes: - while ((ul !=3D NULL) && (fl !=3D NULL)) { + while (ul !=3D NULL) { - GaimConvChatBuddyFlags flag =3D GPOINTER_TO_INT(fl->data); + GaimConvChatBuddyFlags flag =3D 0; Then, instead of building a list of size N filled with zeros, just pass NULL as "flags". Compare the before and after timings to see if there's a significant difference on large rooms. I'd be curious to know either way. > Maybe I've done something wrong? Right now, I'm just adding users to the > user list. Do I have to call some function to temporarily disable > continuous updates or something similar? I don't know off the top of my head. My advice would be to look at what the IRC prpl does. Richard |
From: Fredrik T. <fr...@do...> - 2007-01-05 01:52:52
|
On Thu, 2007-01-04 at 13:14 -0600, Richard Laager wrote: > On Thu, 2007-01-04 at 03:21 +0100, Fredrik Tolf wrote: > [...] > > merely annoying. Right now, Gaim 2 has been taking over 8 minutes of CPU > > time trying to add ~6300 users, with no sign of stopping any time soon. > > Needless to say, that goes beyond merely being annoying. Is there a way > > to fix this? > [...] > Are you calling gaim_conv_chat_add_user() or gaim_conv_chat_add_users()? > Adding them one at a time is going to result in 3 * N lists being > created and destroyed, which could be the source of your problem. Indeed, this was the problem. I had no idea that gaim_conv_chat_add_users existed. :) Is it new in Gaim 2, or did it exist previously as well? Anyways, thanks for enlightening me! While I'm at it with chat rooms, though: I also have a slight cosmetic problem where the room list window displays the listing as still being generated, even though I call gaim_roomlist_set_in_progress(..., FALSE). Could it be related to that I call it while still being in the roomlist_get_list callback? (I already have the list in memory, so there's no need for me to do I/O to generate the room list.) Fredrik Tolf |
From: Fredrik T. <fr...@do...> - 2007-01-10 17:27:33
|
On Fri, 2007-01-05 at 02:52 +0100, Fredrik Tolf wrote: > On Thu, 2007-01-04 at 13:14 -0600, Richard Laager wrote: > > On Thu, 2007-01-04 at 03:21 +0100, Fredrik Tolf wrote: > > [...] > > > merely annoying. Right now, Gaim 2 has been taking over 8 minutes of CPU > > > time trying to add ~6300 users, with no sign of stopping any time soon. > > > Needless to say, that goes beyond merely being annoying. Is there a way > > > to fix this? > > [...] > > Are you calling gaim_conv_chat_add_user() or gaim_conv_chat_add_users()? > > Adding them one at a time is going to result in 3 * N lists being > > created and destroyed, which could be the source of your problem. > > Indeed, this was the problem. I had no idea that > gaim_conv_chat_add_users existed. :) Is it new in Gaim 2, or did it > exist previously as well? Anyways, thanks for enlightening me! On another note about large chat rooms, it seems that in rooms where the number of users approach 10000, the operations of adding and removing single users the join and leave become painfully slow. It appears that it can take a couple of hundred millseconds to add or remove a single user. Since users join and leave frequently, the UI becomes almost unresponsive because of that. Is there some optimized API that I can use in these situations as well? Fredrik Tolf |
From: Adil <ad...@ya...> - 2007-01-11 09:40:24
|
--- Fredrik Tolf <fr...@do...> wrote: > On Fri, 2007-01-05 at 02:52 +0100, Fredrik Tolf wrote: > > On Thu, 2007-01-04 at 13:14 -0600, Richard Laager wrote: > > > On Thu, 2007-01-04 at 03:21 +0100, Fredrik Tolf wrote: > > > [...] > > > > merely annoying. Right now, Gaim 2 has been taking over 8 minutes > of CPU > > > > time trying to add ~6300 users, with no sign of stopping any time > soon. > > > > Needless to say, that goes beyond merely being annoying. Is there > a way > > > > to fix this? > > > [...] > > > Are you calling gaim_conv_chat_add_user() or > gaim_conv_chat_add_users()? > > > Adding them one at a time is going to result in 3 * N lists being > > > created and destroyed, which could be the source of your problem. > > > > Indeed, this was the problem. I had no idea that > > gaim_conv_chat_add_users existed. :) Is it new in Gaim 2, or did it > > exist previously as well? Anyways, thanks for enlightening me! > > On another note about large chat rooms, it seems that in rooms where the > number of users approach 10000, the operations of adding and removing > single users the join and leave become painfully slow. It appears that > it can take a couple of hundred millseconds to add or remove a single > user. > > Since users join and leave frequently, the UI becomes almost > unresponsive because of that. Is there some optimized API that I can use > in these situations as well? One workaround can be to update the user-list at most once every X seconds (whichever value of X you think is appropriate). So: 1. Once a user joins/leaves the room, you create a list with that user and start a timer (g_timeout_add). 2. If another user joins/leaves the room, you see if the timer for that room is still alive. If it is, you add the user in the list. If not, you do (1) 3. The timeout callback calls add-users and remove-users functions in libgaim and deactivates the timer. You could probably consider an idle-callback (g_idle_add) instead of a timeout-callback. I am not sure if that'd help. Sadrul __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com |
From: Sean E. <sea...@gm...> - 2007-01-11 22:38:21
|
On 1/11/07, Adil <ad...@ya...> wrote: > > On another note about large chat rooms, it seems that in rooms where the > > number of users approach 10000, the operations of adding and removing > > single users the join and leave become painfully slow. It appears that > > it can take a couple of hundred millseconds to add or remove a single > > user. I'm surprised nobody else has asked... what on earth are you doing that requires 10000 people in a chat? -s. |
From: <mo...@gm...> - 2007-01-11 22:44:07
Attachments:
smime.p7s
|
Sean Egan schrieb: > On 1/11/07, Adil <ad...@ya...> wrote: > >>> On another note about large chat rooms, it seems that in rooms where the >>> number of users approach 10000, the operations of adding and removing >>> single users the join and leave become painfully slow. It appears that >>> it can take a couple of hundred millseconds to add or remove a single >>> user. >>> > > I'm surprised nobody else has asked... what on earth are you doing > that requires 10000 people in a chat? > > Isn't trying out whether it works reason enaugh? ;-) Morty -- strübe.de <http://xn--strbe-mva.de> Diese Email ist signiert. Sollte Dein Email-Client keine Signaturen unterstützen wird eine smime.p7s-Datei im Anhang angezeigt. Meinen PGP/GPG-Key gibt es auf den üblichen Keyservern. |
From: Ethan B. <ebl...@cs...> - 2007-01-12 00:40:21
|
Moritz 'Morty' Str=FCbe spake unto us the following wisdom: > > I'm surprised nobody else has asked... what on earth are you doing > > that requires 10000 people in a chat? > > > Isn't trying out whether it works reason enaugh? ;-) Well, *every* algorithm has its breaking point, and there is a tension between writing clean and understandable code, and making things more efficient. Often both goals can be realized in parallel for some time, but at a certain point one must sacrifice one to gain the other (if indeed any progress can be made at all). Thus, one often writes code (and plans to use code) for the expected case, knowing that another load may be too much. In this case, virtually all of our structures are simple linked lists, because this is what GtkTreeView understands. Updates are processed as they arrive, rather than being batched in some fashion. Handling 10,000 buddies gracefully in real-time may be beyond a simple linked list. Either batching or a more complex data structure would serve to complicate the code (particularly in view of the fact that the more complicated structure would have to be reduced to a list for presentation to the GtkTreeView) and make understanding more difficult. So, no, just to see if it works isn't necessarily reason enough to *change* anything. On the other hand, a real use case probably is. Ethan --=20 The laws that forbid the carrying of arms are laws [that have no remedy for evils]. They disarm only those who are neither inclined nor determined to commit crimes. -- Cesare Beccaria, "On Crimes and Punishments", 1764 |
From: <mo...@gm...> - 2007-01-12 10:28:57
Attachments:
smime.p7s
|
Ethan Blanton schrieb: > Moritz 'Morty' Strübe spake unto us the following wisdom: > >>> I'm surprised nobody else has asked... what on earth are you doing >>> that requires 10000 people in a chat? >>> >>> >> Isn't trying out whether it works reason enaugh? ;-) >> > > Well, *every* algorithm has its breaking point, and there is a tension > between writing clean and understandable code, and making things more > efficient. Often both goals can be realized in parallel for some > time, but at a certain point one must sacrifice one to gain the other > (if indeed any progress can be made at all). Thus, one often writes > code (and plans to use code) for the expected case, knowing that > another load may be too much. > > In this case, virtually all of our structures are simple linked lists, > because this is what GtkTreeView understands. Updates are processed > as they arrive, rather than being batched in some fashion. Handling > 10,000 buddies gracefully in real-time may be beyond a simple linked > list. Either batching or a more complex data structure would serve to > complicate the code (particularly in view of the fact that the more > complicated structure would have to be reduced to a list for > presentation to the GtkTreeView) and make understanding more > difficult. > > So, no, just to see if it works isn't necessarily reason enough to > *change* anything. On the other hand, a real use case probably is. > > Ethan > > I'm totally with you, Ethan. The comment wasn't meant too serious (hoped the smiley would show this). I think there are much more important issues. Keep up the good work. Morty -- strübe.de <http://xn--strbe-mva.de> Diese Email ist signiert. Sollte Dein Email-Client keine Signaturen unterstützen wird eine smime.p7s-Datei im Anhang angezeigt. Meinen PGP/GPG-Key gibt es auf den üblichen Keyservern. |
From: Jon O. <jo...@ob...> - 2007-01-11 23:02:19
|
On Thu, 2007-01-11 at 14:38 -0800, Sean Egan wrote: > On 1/11/07, Adil <ad...@ya...> wrote: > > > On another note about large chat rooms, it seems that in rooms where = the > > > number of users approach 10000, the operations of adding and removing > > > single users the join and leave become painfully slow. It appears tha= t > > > it can take a couple of hundred millseconds to add or remove a single > > > user. >=20 > I'm surprised nobody else has asked... what on earth are you doing > that requires 10000 people in a chat? Are botnets not a legitimate use-case for Gaim? ;) Regards, Jon Oberheide --=20 Jon Oberheide <jo...@ob...> GnuPG Key: 1024D/F47C17FE Fingerprint: B716 DA66 8173 6EDD 28F6 F184 5842 1C89 F47C 17FE |
From: Greg H. <ghudson@MIT.EDU> - 2007-01-12 06:28:01
|
On Thu, 2007-01-11 at 14:38 -0800, Sean Egan wrote: > I'm surprised nobody else has asked... what on earth are you doing > that requires 10000 people in a chat? I can easily imagine wanting 10,000 people in a chat (probably not all with the ability to send), but if the chatroom is sending information to all 10,000 people about who else is in the room, that's a hundred million missives. The performance of Gaim's UI code seems like the least of your problems in that situation. |
From: Fredrik T. <fr...@do...> - 2007-01-12 13:03:37
|
On Thu, 2007-01-11 at 14:38 -0800, Sean Egan wrote: > On 1/11/07, Adil <ad...@ya...> wrote: > > > On another note about large chat rooms, it seems that in rooms where the > > > number of users approach 10000, the operations of adding and removing > > > single users the join and leave become painfully slow. It appears that > > > it can take a couple of hundred millseconds to add or remove a single > > > user. > > I'm surprised nobody else has asked... what on earth are you doing > that requires 10000 people in a chat? I'm writing a P2P client for the Direct Connect network (see <http://www.dolda2000.com/~fredrik/doldaconnect/>). Rather than writing my own half-assed chat interface for it, I'm writing a PRPL for gaim that connects to my program and chats through it, and thus get a real, time-tried chat interface. Hubs in Direct Connect frequently have 5000+ users, and each of them has the ability to chat. The vast majority of them don't, of course, but if I don't have them in the user list, it wouldn't be easy to PM them. A while ago, I tried running Gaim in gdb while doing this and stopped it while it was busy after adding or removing a user from a really large chat room, but stupid as I was, I didn't save the backtraces. I thought it seemed as though it was taking its time sorting the user list or something liek that. If you want to, I can do it again and send the backtraces back here. Fredrik Tolf |
From: Nathan W. <fac...@fa...> - 2007-01-05 18:25:45
|
Fredrik Tolf wrote: <snip> > While I'm at it with chat rooms, though: I also have a slight cosmetic > problem where the room list window displays the listing as still being > generated, even though I call gaim_roomlist_set_in_progress(..., FALSE). > Could it be related to that I call it while still being in the > roomlist_get_list callback? (I already have the list in memory, so > there's no need for me to do I/O to generate the room list.) I just made a commit that should fix this. Let me know if it doesn't. -Nathan |
From: Fredrik T. <fr...@do...> - 2007-01-09 00:06:15
|
On Fri, 2007-01-05 at 13:25 -0500, Nathan Walp wrote: > Fredrik Tolf wrote: > <snip> > > While I'm at it with chat rooms, though: I also have a slight cosmetic > > problem where the room list window displays the listing as still being > > generated, even though I call gaim_roomlist_set_in_progress(..., FALSE). > > Could it be related to that I call it while still being in the > > roomlist_get_list callback? (I already have the list in memory, so > > there's no need for me to do I/O to generate the room list.) > > I just made a commit that should fix this. Let me know if it doesn't. I don't run Gaim from SVN (or is it CVS?), so I'll take your word for it for now, and get back if it doesn't work when Gentoo pushes the next release on me. However, I've got another issue regarding large chat rooms. See, it just so happens that people enter and leave these rooms very often (which I'm guessing is a combination of the nature of people on this protocol and the effect of large chat rooms). When I say `very often', I mean about an arrival and a departure per second. Therefore, the messages that Gaim generates about arrivals and departures easily swamp the real chat. Is there a way for me to squelch these messages in just this prpl? Fredrik Tolf |
From: Fredrik T. <fr...@do...> - 2007-01-30 11:19:54
|
On Tue, 2007-01-09 at 01:06 +0100, Fredrik Tolf wrote: > On Fri, 2007-01-05 at 13:25 -0500, Nathan Walp wrote: > > Fredrik Tolf wrote: > > <snip> > > > While I'm at it with chat rooms, though: I also have a slight cosmetic > > > problem where the room list window displays the listing as still being > > > generated, even though I call gaim_roomlist_set_in_progress(..., FALSE). > > > Could it be related to that I call it while still being in the > > > roomlist_get_list callback? (I already have the list in memory, so > > > there's no need for me to do I/O to generate the room list.) > > > > I just made a commit that should fix this. Let me know if it doesn't. > > I don't run Gaim from SVN (or is it CVS?), so I'll take your word for it > for now, and get back if it doesn't work when Gentoo pushes the next > release on me. I just thought I'd report back on this. I got beta6 through Gentoo's portage, and it works now. Thanks! Fredrik Tolf |
From: Richard L. <rl...@wi...> - 2007-01-09 00:30:14
|
On Tue, 2007-01-09 at 01:06 +0100, Fredrik Tolf wrote: > I don't run Gaim from SVN (or is it CVS?), so I'll take your word for it > for now, and get back if it doesn't work when Gentoo pushes the next > release on me. >=20 > However, I've got another issue regarding large chat rooms. See, it just > so happens that people enter and leave these rooms very often (which I'm > guessing is a combination of the nature of people on this protocol and > the effect of large chat rooms). When I say `very often', I mean about > an arrival and a departure per second. Therefore, the messages that Gaim > generates about arrivals and departures easily swamp the real chat. Is > there a way for me to squelch these messages in just this prpl? You could just always set the new_arrivals argument to FALSE when calling the add_users function. In theory, we might want to do something else with this argument in the future, but you're probably safe doing this. If you wanted to be definitely safe, you could register a signal handler for chat-buddy-joining, check if the conversation is on your protocol, and return TRUE. Richard |
From: Fredrik T. <fr...@do...> - 2007-01-09 02:57:18
|
On Mon, 2007-01-08 at 18:30 -0600, Richard Laager wrote: > On Tue, 2007-01-09 at 01:06 +0100, Fredrik Tolf wrote: > > However, I've got another issue regarding large chat rooms. See, it just > > so happens that people enter and leave these rooms very often (which I'm > > guessing is a combination of the nature of people on this protocol and > > the effect of large chat rooms). When I say `very often', I mean about > > an arrival and a departure per second. Therefore, the messages that Gaim > > generates about arrivals and departures easily swamp the real chat. Is > > there a way for me to squelch these messages in just this prpl? > > You could just always set the new_arrivals argument to FALSE when > calling the add_users function. In theory, we might want to do something > else with this argument in the future, but you're probably safe doing > this. Correct me if I'm wrong, but that only works for arrivals and not for departures, right? > If you wanted to be definitely safe, you could register a signal > handler for chat-buddy-joining, check if the conversation is on your > protocol, and return TRUE. Without having checked the source in advance, I'm guessing there's something similar for chat-buddy-leaving, so that seems to be the way to go. Thanks! Fredrik Tolf |