Menu
▾
▴
fetchmail-devel
|
From: Rainer W. <rwe...@ms...> - 2010-05-28 02:35:05
|
The numbers contained in this mail come from two 24h 'observations' of
the old and new UID handling code done via oprofile. They are based on
querying 119 POP3 accounts. Some statistics about the .fetchids files
(everything in lines):
total: 35004
max: 8019
min: 1
average: 294.15
median: 46
std. dev: 823.87
Grouped by size with increments of 16:
0 36 (30.2%)
16 18 (15.1%)
32 7 (5.8%)
48 5 (4.2%)
64 3 (2.5%)
80 2 (1.6%)
96 4 (3.3%)
128 4 (3.3%)
144 1 (0.8%)
160 2 (1.6%)
176 5 (4.2%)
192 2 (1.6%)
208 1 (0.8%)
240 1 (0.8%)
256 4 (3.3%)
288 3 (2.5%)
304 1 (0.8%)
320 1 (0.8%)
384 1 (0.8%)
480 1 (0.8%)
512 2 (1.6%)
544 1 (0.8%)
560 1 (0.8%)
720 1 (0.8%)
736 1 (0.8%)
912 1 (0.8%)
976 1 (0.8%)
1104 1 (0.8%)
1200 1 (0.8%)
1264 1 (0.8%)
1344 2 (1.6%)
1408 1 (0.8%)
1744 1 (0.8%)
2000 1 (0.8%)
2064 1 (0.8%)
This is data about the presently existing files. All actual numbers
were 'somewhat smaller' for the 'new code' run and 'even more smaller'
for the old one.
UID handling is still the dominant operation (processor-time wise)
performed by fetchmail. The relative amount of time is down from
93.46% to 34.36%. During the measurement interval, 1,540,732 event
samples were collected for the old code and 63,092 for the new
code: The trie/ array-based UID db has used about 4.09% of the CPU
time the list-based one did use.
By function, list based (at least 0.1%):
929507 56.3856 fetchmail.dimap id_find
585333 35.5074 fetchmail.dimap str_in_list
25892 1.5707 fetchmail.dimap save_str [w/ tail pointers]
15089 0.9153 fetchmail.dimap yylex
10827 0.6568 fetchmail.dimap do_session
10265 0.6227 fetchmail.dimap initialize_saved_lists
5908 0.3584 fetchmail.dimap yyparse
4941 0.2997 fetchmail.dimap SockRead
3557 0.2158 fetchmail.dimap strlcpy
3094 0.1877 fetchmail.dimap gen_recv
2976 0.1805 fetchmail.dimap main
2838 0.1722 fetchmail.dimap pop3_getrange
2783 0.1688 fetchmail.dimap optmerge
2642 0.1603 fetchmail.dimap stuffline
2614 0.1586 fetchmail.dimap write_saved_lists
2519 0.1528 fetchmail.dimap imap_response
2167 0.1315 fetchmail.dimap set_timeout
2162 0.1312 fetchmail.dimap imap_is_old
2072 0.1257 fetchmail.dimap xstrdup
1791 0.1086 fetchmail.dimap xmalloc
By function, trie/ array:
36916 20.1071 fetchmail.uid-db uid_db_insert
26176 14.2573 fetchmail.uid-db find_uid_by_id
16937 9.2251 fetchmail.uid-db yylex
10700 5.8280 fetchmail.uid-db initialize_saved_lists
10248 5.5818 fetchmail.uid-db do_session
6041 3.2904 fetchmail.uid-db yyparse
4994 2.7201 fetchmail.uid-db SockRead
3728 2.0305 fetchmail.uid-db stuffline
3620 1.9717 fetchmail.uid-db strlcpy
3507 1.9102 fetchmail.uid-db free_uid_db
3212 1.7495 fetchmail.uid-db main
3174 1.7288 fetchmail.uid-db gen_recv
2791 1.5202 fetchmail.uid-db optmerge
2728 1.4859 fetchmail.uid-db imap_response
2641 1.4385 fetchmail.uid-db set_timeout
2428 1.3225 fetchmail.uid-db xmalloc
2356 1.2832 fetchmail.uid-db imap_is_old
2271 1.2369 fetchmail.uid-db write_uid_db_record
1901 1.0354 fetchmail.uid-db pop3_getrange
1803 0.9820 fetchmail.uid-db T.169
1786 0.9728 fetchmail.uid-db parsecmdline
1532 0.8344 fetchmail.uid-db traverse_uid_db
1413 0.7696 fetchmail.uid-db count_seen_deleted
1182 0.6438 fetchmail.uid-db free_uid_list
1112 0.6057 fetchmail.uid-db envquery
1112 0.6057 fetchmail.uid-db pop3_is_old
1098 0.5980 fetchmail.uid-db mark_as_expunged_if
936 0.5098 fetchmail.uid-db _fini
892 0.4858 fetchmail.uid-db imap_getrange
880 0.4793 fetchmail.uid-db gen_transact
812 0.4423 fetchmail.uid-db readbody
774 0.4216 fetchmail.uid-db SSL_verify_callback
747 0.4069 fetchmail.uid-db SockOpen
736 0.4009 fetchmail.uid-db readheaders
735 0.4003 fetchmail.uid-db parse_resp_code
687 0.3742 fetchmail.uid-db set_signal_handler
680 0.3704 fetchmail.uid-db pop3_ok
673 0.3666 fetchmail.uid-db prependdir
597 0.3252 fetchmail.uid-db SSLOpen
577 0.3143 fetchmail.uid-db xstrdup
575 0.3132 fetchmail.uid-db SockWrite
512 0.2789 fetchmail.uid-db write_saved_lists
499 0.2718 fetchmail.uid-db query_host
494 0.2691 fetchmail.uid-db in_drop_domain
481 0.2620 fetchmail.uid-db prc_parse_file
471 0.2565 fetchmail.uid-db do_protocol
463 0.2522 fetchmail.uid-db pop3_getauth
400 0.2179 fetchmail.uid-db pmg_stuffline
371 0.2021 fetchmail.uid-db parse_netrc
368 0.2004 fetchmail.uid-db imap_getauth
338 0.1841 fetchmail.uid-db MD5Update
314 0.1710 fetchmail.uid-db free_netrc
311 0.1694 fetchmail.uid-db init_uid_db
296 0.1612 fetchmail.uid-db yyensure_buffer_stack
292 0.1590 fetchmail.uid-db copy_str_list
291 0.1585 fetchmail.uid-db fm_getaddrinfo
291 0.1585 fetchmail.uid-db host_fqdn
283 0.1541 fetchmail.uid-db imap_ok
278 0.1514 fetchmail.uid-db capa_probe
270 0.1471 fetchmail.uid-db reply_hack
264 0.1438 fetchmail.uid-db __libc_csu_init
256 0.1394 fetchmail.uid-db _start
242 0.1318 fetchmail.uid-db report_init
240 0.1307 fetchmail.uid-db interface_approve
233 0.1269 fetchmail.uid-db T.136
232 0.1264 fetchmail.uid-db yy_flush_buffer
223 0.1215 fetchmail.uid-db list_merge
211 0.1149 fetchmail.uid-db save_str
209 0.1138 fetchmail.uid-db T.140
196 0.1068 fetchmail.uid-db terminate_poll
192 0.1046 fetchmail.uid-db interface_init
I hope to be able to produce something like a final patch at some time
late evening tomorrow.
|
|
From: Matthias A. <mat...@gm...> - 2010-05-28 04:24:49
|
Am 28.05.2010, 02:34 Uhr, schrieb Rainer Weikusat: > The numbers contained in this mail come from two 24h 'observations' of > the old and new UID handling code done via oprofile. They are based on > querying 119 POP3 accounts. Some statistics about the .fetchids files > (everything in lines): > > total: 35004 > max: 8019 > min: 1 > average: 294.15 > median: 46 > std. dev: 823.87 D'oh. A std dev' that's thrice the average. There's a considerable chance for a negative number. :-} The actual problem is that the model doesn't fit, you don't have a Normal distribution here, confirmed by the table I'm not quoting. > UID handling is still the dominant operation (processor-time wise) > performed by fetchmail. No it's not, see below -- you're not grouping other code properly. If oprofile samples call stacks so that it can cumulate per caller (like du(1) does) you'll see it's not. Otherwise, we'll resort to the preclusion reasoning below. > The relative amount of time is down from > 93.46% to 34.36%. During the measurement interval, 1,540,732 event > samples were collected for the old code and 63,092 for the new > code: The trie/ array-based UID db has used about 4.09% of the CPU > time the list-based one did use. The relative times of the functions among themselves are hard to interpret, particularly if you don't group the other functions properly. The total sample count is far more interesting, and there you're down from 1.52 million samples (out of 1.65 million) to 0.07 million (out of 0.18 million), where the other stuff remains more or less the same. IOW, there is a base amount of ~0.12 million samples NOT related to UID handling. Where the old code in your load scenario was more than 12x that of the other code (i. e. "dominated"), it's now roughly half the effort, which is no longer "dominating". > I hope to be able to produce something like a final patch at some time > late evening tomorrow. No urge here, I'm not sure if I'll have the time to merge that stuff over the week-end. May well turn mid to end of next week before I get to it. Thank you. -- Matthias Andree |
|
From: Rainer W. <rwe...@ms...> - 2010-05-28 11:30:10
|
"Matthias Andree" <mat...@gm...> writes: > Am 28.05.2010, 02:34 Uhr, schrieb Rainer Weikusat: > >> The numbers contained in this mail come from two 24h 'observations' of >> the old and new UID handling code done via oprofile. They are based on >> querying 119 POP3 accounts. Some statistics about the .fetchids files >> (everything in lines): >> >> total: 35004 >> max: 8019 >> min: 1 >> average: 294.15 >> median: 46 >> std. dev: 823.87 > > D'oh. A std dev' that's thrice the average. There's a considerable > chance for a negative number. :-} It is conceivable that I am miscalculating the number. But this really just communicates that the distribution is very asymmetric. >> UID handling is still the dominant operation (processor-time wise) >> performed by fetchmail. > > No it's not, see below -- you're not grouping other code properly. If > oprofile samples call stacks so that it can cumulate per caller (like > du(1) does) you'll see it's not. Otherwise, we'll resort to the > preclusion reasoning below. This seems to be a disagreement about the meaning of the term 'dominant'. It may well have some defined meaning I am not aware of. I am not a repurposed physicist or person with any other kind of degree , just a programmer and even this just because I spent something like twenty years learning about this stuff whatever I could gather from sources available to me. [...] >> I hope to be able to produce something like a final patch at some time >> late evening tomorrow. > > No urge here, I'm not sure if I'll have the time to merge that stuff > over the week-end. May well turn mid to end of next week before I get > to it. I will nevertheless do that since it is a fairly simple operation, which I can put into a few idle minutes in my (presently) usual 10h+ workdays and something I can absolutely not do is put much more effort into this, except maybe two o'clock in the morning after working for ten hours. Since the code exists, works and (as far as I could determine) mitigates the problem it was supposed to mitigate, any justification I could conceivable use to explain to my boss why I am even looking at this during the time I am neither eating nor sleeping is gone :-(. |
|
From: Matthias A. <mat...@gm...> - 2010-05-28 19:16:13
|
Rainer Weikusat wrote on 2010-05-28: > "Matthias Andree" <mat...@gm...> writes: >> D'oh. A std dev' that's thrice the average. There's a considerable >> chance for a negative number. :-} > > It is conceivable that I am miscalculating the number. But this really > just communicates that the distribution is very asymmetric. I'd rather trust your numerics than the underlying assumption it were normally distributed data :-) (Normalverteilung/Gauss'sche). Ok, there's always room to miss out on a "2" in an exponent and inadvertently confuse variance and standard deviation, but still why would the count of messages in the mailboxes be normally distributed? There isn't a single reason I could see, and this voids all chances that the arith. average were in 1-sigma, 2-sigma, 3-sigma confidence intervals, or anywhere near the expected value. >>> UID handling is still the dominant operation (processor-time wise) >>> performed by fetchmail. >> >> No it's not, see below -- you're not grouping other code properly. If >> oprofile samples call stacks so that it can cumulate per caller (like >> du(1) does) you'll see it's not. Otherwise, we'll resort to the >> preclusion reasoning below. > > This seems to be a disagreement about the meaning of the term > 'dominant'. It may well have some defined meaning I am not aware of. > I am not a repurposed physicist or person with any other kind of > degree , just a programmer and even this just because I spent > something like twenty years learning about this stuff whatever I could > gather from sources available to me. I'm not sure if there is a formal definiton, but I'd say it's "dominant" if you can neglect other terms in a sum, for instance. Or if you'd write the >> or << relation operators, often considered as possible with one order of magnitude in between. That we don't have after your fix. That doesn't make your fix less useful, however. As written for my 16,000 message inbox, fetchmail starts fetching immediately, whereas the unpatched fetchmail ponders UIDL more than a minute. > I will nevertheless do that since it is a fairly simple operation, > which I can put into a few idle minutes in my (presently) usual 10h+ > workdays and something I can absolutely not do is put much more effort > into this, except maybe two o'clock in the morning after working for > ten hours. Since the code exists, works and (as far as I could > determine) mitigates the problem it was supposed to mitigate, any > justification I could conceivable use to explain to my boss why I am > even looking at this during the time I am neither eating nor sleeping > is gone :-(. Which is good enough. I've always been meaning to do that, but there was always something else more important to do, so your final patch will be most welcome. I expect to make minor changes to fix compiler warnings and add side comments to NEWS and thereabouts (it's always interesting to see what you get when you compile on different Linux flavours, Solaris, NetBSD, FreeBSD, MacOS X), but from what I've seen so far, not much will be needed. Possibly the merge with my existing advance development branch (not yet published) for 6.4.0 or 6.5.0 may take another hour - I don't think I'll put this feature in a 6.3.X release. This 6.3 branch was originally just a drop-in compatible stopgap branch so nobody had an excuse to sidestep the >100 fixes between 6.2.5 and 6.3.0, but as we all know, makeshift arrangements often prove to be long-lived. I would never have expected it to spawn 18 nontrivial releases to date, and 6.3.18 would appear, too. The fetchmail code however really needs updating and leave compatibility with pre-2000 operating systems (with before C-89, before-POSIX2001) behind - especially the way it's sprayed all over the source with a gazillion of #ifdef makes maintaining the code a time-consuming and error-prone process, and my experience is if you don't have good improvements as selling points users will just stick to the old versions that you've left behind long ago. I know I may disappoint some users that way, but in the end I need to see to real life -- which means I'm not pampering up systems where vendor support ceased years ago. Efficiency improvements that users can see are one such selling point to motivate a migration... :) -- Matthias Andree |
|
From: Rainer W. <rwe...@ms...> - 2010-06-01 00:14:06
|
Rainer Weikusat <rwe...@ms...> writes: > "Matthias Andree" <mat...@gm...> writes: [...] >> No urge here, I'm not sure if I'll have the time to merge that stuff >> over the week-end. May well turn mid to end of next week before I get >> to it. > > I will nevertheless do that since it is a fairly simple operation, > which I can put into a few idle minutes in my (presently) usual 10h+ > workdays Update why this hasn't happened so far: I have a deadline to meet at 3pm 2010/06/01 and because of this, I've had no time to do anything except working towards this goal so far (been writing code since 2010/05/31 11am so far). |
|
From: Matthias A. <mat...@gm...> - 2010-06-01 10:25:39
|
Am 01.06.2010 00:13, schrieb Rainer Weikusat: > Rainer Weikusat <rwe...@ms...> writes: >> "Matthias Andree" <mat...@gm...> writes: > > [...] > >>> No urge here, I'm not sure if I'll have the time to merge that stuff >>> over the week-end. May well turn mid to end of next week before I get >>> to it. >> >> I will nevertheless do that since it is a fairly simple operation, >> which I can put into a few idle minutes in my (presently) usual 10h+ >> workdays > > Update why this hasn't happened so far: I have a deadline to meet at > 3pm 2010/06/01 and because of this, I've had no time to do anything > except working towards this goal so far (been writing code since > 2010/05/31 11am so far). Thanks for letting me know. This isn't a problem. I appreciate that real-life obligations precede fetchmail contributions. Focus on your RL project, and good speed with it. -- Matthias Andree |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X