From: Dave C. <dav...@gm...> - 2011-02-24 21:07:25
|
Hello, I have a question relating to the timing behavior of libusb under macos. We are using using libusb for linux/mac/windows cross platform USB access to our hardware, and have been researching a timing/performance difference between MacOS and Linux. In particular, on linux, when opening a device it typically takes 0 to 1 milliseconds to open the given device. However, on mac, it takes at minimum 30 ms. Of particular note, is that the time it takes to open a device by a given indexes seems to grow linearly with the i'th device index. I've copied and pasted the debug output below from the mac, and the corresponding code that generated the output. The number on the left is the timestamp in milliseconds that the log message was generated, and the number after 'DBG' is the number of milliseconds since the last debug line was logged. Thoughts? Thanks, -Dave 1298580883757:DBG: 0: Opending usb device... 1298580883786:DBG: 29: Successfully opened usb device... 1298580883789:DBG: 3: num_bytes = 8, idx = 10, serial='0000027', iSerialNumber=3 1298580883789:DBG: 0: Opending usb device... 1298580883825:DBG: 36: Successfully opened usb device... 1298580883827:DBG: 2: num_bytes = 8, idx = 13, serial='0000122', iSerialNumber=3 1298580883827:DBG: 0: Opending usb device... 1298580883870:DBG: 43: Successfully opened usb device... 1298580883873:DBG: 3: num_bytes = 8, idx = 16, serial='0000136', iSerialNumber=3 1298580883873:DBG: 0: Opending usb device... 1298580883920:DBG: 47: Successfully opened usb device... 1298580883923:DBG: 3: num_bytes = 8, idx = 18, serial='0000034', iSerialNumber=3 1298580883923:DBG: 0: Opending usb device... 1298580883978:DBG: 55: Successfully opened usb device... 1298580883980:DBG: 2: num_bytes = 8, idx = 21, serial='0000256', iSerialNumber=3 1298580883980:DBG: 0: Opending usb device... 1298580884044:DBG: 64: Successfully opened usb device... 1298580884047:DBG: 3: num_bytes = 8, idx = 25, serial='0000140', iSerialNumber=3 1298580884047:DBG: 0: Opending usb device... 1298580884119:DBG: 72: Successfully opened usb device... 1298580884121:DBG: 2: num_bytes = 8, idx = 28, serial='0000185', iSerialNumber=3 for (i = 0; i < numDevices; i++) { struct libusb_device_descriptor desc; int ret = libusb_get_device_descriptor(deviceListPtr[i], &desc); if( ret == LIBUSB_SUCCESS && desc.idVendor == USB_VID ) { log_debug("Opending usb device...\n"); const int openRet = libusb_open(deviceListPtr[i], devh); if( openRet == LIBUSB_SUCCESS ) { log_debug("Successfully opened usb device...\n"); unsigned char serial_buff[128]; strcpy((char*) serial_buff, ""); int num_bytes = libusb_get_string_descriptor_ascii(*devh, desc.iSerialNumber, serial_buff, sizeof(serial_buff)); libusb_close(*devh); *devh = NULL; if( num_bytes < 0 ) { log_error("Error retrieving device serial number (2)... r=%d %s\n", num_bytes, xxxx_strerror(abs(num_bytes) + XXXX_LIB_USB_ERROR)); } log_debug("num_bytes = %d, idx = %d, serial='%s', iSerialNumber=%u\n", num_bytes, i, serial_buff, desc.iSerialNumber); } } |
From: Nathan H. <hj...@me...> - 2011-02-24 22:05:00
|
On Feb 24, 2011, at 02:08 PM, Dave Camarillo <dav...@gm...> wrote: Hello, I have a question relating to the timing behavior of libusb under macos. We are using using libusb for linux/mac/windows cross platform USB access to our hardware, and have been researching a timing/performance difference between MacOS and Linux. In particular, on linux, when opening a device it typically takes 0 to 1 milliseconds to open the given device. However, on mac, it takes at minimum 30 ms. Of particular note, is that the time it takes to open a device by a given indexes seems to grow linearly with the i'th device index. I've copied and pasted the debug output below from the mac, and the corresponding code that generated the output. The number on the left is the timestamp in milliseconds that the log message was generated, and the number after 'DBG' is the number of milliseconds since the last debug line was logged. Thoughts? Thanks, -Dave This behavior is to be expected (for now). To be safe, every time a device is opened libusb iterates through the IORegistry to find the corresponding device. The further down the registry the device is, the longer the search takes. The only way I have come up with (so far) to avoid a linear search would be to cache and hold a reference to each device encountered. I am waiting until the hotplug interface is defined to check whether or not it would be a good idea. That said, 30ms is a very short period of time. What sort of application are you writing that needs faster open times? -Nathan |
From: Dave C. <dav...@gm...> - 2011-02-25 00:33:42
|
Hi Nathan, thanks for the quick response... That makes sense given your description. It boils down to the application opening and closing the usb device often and the assumption/expectation that it's a constant-time operation, in combination with many devices being attached to the host. The product of the two yields O(N^2) which I think is why it ends up being an observable time difference. However, I think there are alternate ways that our application could be utilizing the usb devices such that they do not need to be opened and closed so frequently. Thanks for the explanation, it gives us a clear direction to take to address the end-user issue.... Thanks, -Dave On Thu, Feb 24, 2011 at 2:04 PM, Nathan Hjelm <hj...@me...> wrote: > On Feb 24, 2011, at 02:08 PM, Dave Camarillo <dave.camarillo@gmailcom> > wrote: > > Hello, I have a question relating to the timing behavior of libusb > under macos. We are using using libusb for linux/mac/windows cross > platform USB access to our hardware, and have been researching a > timing/performance difference between MacOS and Linux. In particular, > on linux, when opening a device it typically takes 0 to 1 milliseconds > to open the given device. However, on mac, it takes at minimum 30 ms. > Of particular note, is that the time it takes to open a device by a > given indexes seems to grow linearly with the i'th device index. I've > copied and pasted the debug output below from the mac, and the > corresponding code that generated the output. The number on the left > is the timestamp in milliseconds that the log message was generated, > and the number after 'DBG' is the number of milliseconds since the > last debug line was logged. > > Thoughts? > > Thanks, > -Dave > > > This behavior is to be expected (for now). To be safe, every time a device > is opened libusb iterates through the IORegistry to find the corresponding > device. The further down the registry the device is, the longer the search > takes. The only way I have come up with (so far) to avoid a linear search > would be to cache and hold a reference to each device encountered. I am > waiting until the hotplug interface is defined to check whether or not it > would be a good idea. > That said, 30ms is a very short period of time. What sort of application are > you writing that needs faster open times? > -Nathan |
From: Pete B. <pb...@gm...> - 2011-02-25 11:01:26
|
> On Thu, Feb 24, 2011 at 2:04 PM, Nathan Hjelm<hj...@me...> wrote: >> The only way I have come up with (so far) to avoid a linear search >> would be to cache and hold a reference to each device encountered. I am >> waiting until the hotplug interface is defined to check whether or not it >> would be a good idea. If we go with the current proposal in hp, we will have a hash table in libusb that is meant to be used to produce a UID for each device, and (hopefully) speed up searches for a specific device. Even if we don't go with hp, I still need a hash table for new-enum on Windows anway, so that code could be factored in core if you think you could use it. It might actually be worth doing it when we integrate new-enum, so that you can have a go at improving search speed. Currently, the hash table code is already factored in core, in the hp branch. The only small limitation we have is that the hash function we use, while fast, seems to be fairly prone to collisions, so there's probably some room for improvement. Regards, /Pete |
From: Segher B. <se...@ke...> - 2011-02-25 19:19:35
|
>>> The only way I have come up with (so far) to avoid a linear search >>> would be to cache and hold a reference to each device encountered. I am >>> waiting until the hotplug interface is defined to check whether or not >>> it >>> would be a good idea. > > If we go with the current proposal in hp, we will have a hash table in > libusb that is meant to be used to produce a UID for each device, and > (hopefully) speed up searches for a specific device. Even if we don't go > with hp, I still need a hash table for new-enum on Windows anway, so > that code could be factored in core if you think you could use it. It > might actually be worth doing it when we integrate new-enum, so that you > can have a go at improving search speed. The important part (speed-wise) is not what data structure is used on the libusb side. The important part is keeping that data structure synched with the kernel's view, in an efficient way. What makes the current (no caching at all) scheme expensive is all the calls to the kernel; sure, O(N**2) isn't good anyway, but there is a HUGE constant factor in there, and that is what is hurting. If you want to do a cross-platform caching scheme, you need to make it work with asynchronous changes. Perhaps do a two-level cache, the first level is changed by the OS (via callbacks for example, can be asynchronous), and the second level is used by libusb; that way, only where you copy the first to the second level (which can also change data representation) you need to pay special attention to locking stuff, keeping things consistent. Or, you can just forget about making it cross-platform, have every platform implement everything itself. Segher |
From: Pete B. <pb...@gm...> - 2011-02-25 22:20:13
|
On 2011.02.25 19:19, Segher Boessenkool wrote: > Or, you can just forget about making it cross-platform, have every > platform implement everything itself. ...by calling into a generic hash_table API provide by core, which is my proposal, and using it in whichever way the backend thinks it should use it. Hashing would not be done on data that core has access to, and core would not use anything from that API. Instead, hashing would be done on backend data. The hash function in core would just be factorized code, if we find out that we have more than one backend "implementing everything itself" through a hash table. Regards, /Pete |
From: Peter S. <pe...@st...> - 2011-02-26 01:25:39
|
Pete Batard wrote: > generic hash_table API provide by core Sounds like overkill. Even with 127 devices on 16 busses that's only just over 2000 devices. Enumerating those should not take significant time. I guess the problem is that here too libusb doesn't quite fit the data model used in the kernel. We can improve that, but baby steps I guess. //Peter |
From: Segher B. <se...@ke...> - 2011-02-26 01:32:34
|
>> Or, you can just forget about making it cross-platform, have every >> platform implement everything itself. > > ...by calling into a generic hash_table API provide by core, which is my > proposal, and using it in whichever way the backend thinks it should use > it. I very much doubt this would be useful for more than one platform. If only one platform uses it, put it in the platform code; if later some other platform wants to use it, you can move it into some library kind of thing, but only then. Abstraction is the cause of all problems, not the solution. Segher |
From: Pete B. <pb...@gm...> - 2011-02-26 14:16:16
|
On 26 February 2011 01:32, Segher Boessenkool <se...@ke...> wrote: > I very much doubt this would be useful for more than one platform. Well, I currently suspect it is likely to be used by Linux once we do hotplug (because that's how it is in my hp branch), and I also understand that Nathan wants to do something about slow device matching, where using a hash table could help. > If only one platform uses it, put it in the platform code; if later > some other platform wants to use it, Which is what I asked Nathan about. > you can move it into some library > kind of thing, but only then. And the "then" part is precisely where we are, pending on Nathan's reply. > Abstraction is the cause of all problems, not the solution. Looking ahead (how we are going to implement hotplug) rather than just what's under your nose is a way to avoid problem and make future integration easier. If we go with the Nokia hp proposal, we _will_ need a hash function for Linux [1] (search for usbi_hash) and for Windows introduced in later commit. Then we have Nathan who also may find a hash table useful. And the hash function is also part of new enum. So we _will_ have a hash function after new enum is integrated. All I'm asking then is the location. I could very well present the thing as "Well, I'm going to factorize the hash function on new enum, because it shouldn't matter whether it is implemented in core or in the Windows backend, and there is a _reasonable prospect_ of it being used by other backends in the future". Does that make any kind of _logical_ sense as a proposal? I'd appreciate if, for once, you guys stopped thinking that I'm hell bent on pushing "disputable" things for Windows, for my own convenience, while happily ignoring any problems they might introduce elsewhere. I'm only applying logic here. For the sake of what other platforms are likely going to require, and for the sake of keeping our git tree clean (by not moving stuff around from backend to core), it does make logical preemptive sense to try to introduce the hash functions in core. Now, if you actually take the time to look at how you want to implement hotplug for Linux (I mean, we've only had that proposal up for 6 months now - all you guys need to do is take a look sometime), and indicate that it is likely that, as opposed to the proposal, our final implementation is not going to require a hash, and if Nathan comes back and says "I don't think I'll need hash functions for OS-X either", that's fine. Otherwise, what I am advocating for DOES make sense. So could we actually properly and logically try to consider looking ahead for once, and carefully evaluate the merits of a proposal? If Nathan comes back and says "Well, actually, I _could_ use hash table functions in OS-X", will you still pull the "Abstraction is the cause of all problems" absolute to dismiss a proposal? If you too seem to like like absolutes when judging the merits of a proposals, let me give you this meta one then. "Applying absolutes blindly, without looking at the specifics of a situation, is the source of many many issues". Regards, /Pete [1] http://git.libusb.org/?p=libusb-pbatard.git;a=commitdiff;h=6cb025b4c853f2c034f72e12b1ab243d9d6e2a59;js=1 |
From: Segher B. <se...@ke...> - 2011-02-26 15:15:56
|
>> I very much doubt this would be useful for more than one platform. > > Well, I currently suspect it is likely to be used by Linux once we do > hotplug (because that's how it is in my hp branch), and I also > understand that Nathan wants to do something about slow device > matching, where using a hash table could help. As I've explained already, the problem there (if you want to consider it a problem at all) is the OS calls, not the data structure used in libusb. Changing the datastructure would help a few microseconds; the problem is a few tens of milliseconds. >> you can move it into some library >> kind of thing, but only then. > > And the "then" part is precisely where we are, pending on Nathan's reply. No it's not. Not a *single* platform (in mainline) uses it yet. >> Abstraction is the cause of all problems, not the solution. > > Looking ahead (how we are going to implement hotplug) rather than just > what's under your nose is a way to avoid problem and make future > integration easier. For design, sure; but not for implementation. > Now, if you actually take the time to look at No, I'm not going to hunt down patches that may or may not be up to date. If someone posts a proposed patch for mainline, I'll look at it. That's how the game works. Patch is posted, patch is reviewed, patch is updated, loop a bit, patch is committed. If you don't do the earlier steps, you don't get to complain about the later steps. And if somehow it was dropped on the floor half a year ago, you better post it again, if you still want it included. > So could we actually properly and logically try to consider looking > ahead for once, and carefully evaluate the merits of a proposal? I didn't see a proposed patch. I'm not interested in discussing software development philosophies with you, sorry. Segher |
From: Pete B. <pb...@gm...> - 2011-02-27 01:19:42
|
On 2011.02.26 15:15, Segher Boessenkool wrote: >>> I very much doubt this would be useful for more than one platform. >> >> Well, I currently suspect it is likely to be used by Linux once we do >> hotplug (because that's how it is in my hp branch), and I also >> understand that Nathan wants to do something about slow device >> matching, where using a hash table could help. > > As I've explained already, the problem there (if you want to > consider it a problem at all) is the OS calls, not the data > structure used in libusb. Changing the datastructure would > help a few microseconds; the problem is a few tens of milliseconds.On You may be right (dunno, I'm not an OS-X backend developer), but I'd rather see Nathan, as the lead OS-X developer, comment on it. Lately, I've seen too many cases where people, who are not actually involved in a backend development, assume wrong. Plus, if I understand properly, you do seem to agree that using a hash table would be an improvement (since it would shave a few microseconds). Since when is _any_ performance improvement, that can be obtained at very little cost (through hash table functionality in core), not worthy of consideration then? Until he speaks, couldn't Nathan have a different opinion from yours there? Plus, let's say we solve what is supposed to be an OS problem, and make open extra quick. Wouldn't reducing the open time even further be worth something? Nathan clearly indicated "The only way I have come up with (so far) to avoid a linear search would be to cache and hold a reference to each device encountered". Mmmm, so Nathan explicitly says he wouldn't mind avoiding linear search through the use of cached references. To me that sounds an awful lot like something where a hash table might be of help. As such I wouldn't mind having an answer from the man himself, rather than a spokesperson who, AFAIK, comes from Linux, and may not be familiar with IORegistry operations on OS-X. > That's how the game works. Patch is posted, patch is reviewed, > patch is updated, loop a bit, patch is committed. If you don't > do the earlier steps, you don't get to complain about the later > steps. And if somehow it was dropped on the floor half a year > ago, you better post it again, if you still want it included. No, that's not how the game works when maintainers are 6 months to one year behind the curve, and don't follow the rules that everybody else follows (such as RERO on new development, or releases when there are enough commits to do so). I cannot post a hp patch without new enum, and I have been waiting _months_ for mainline to be in a state where new-enum can be proposed (i.e. clearing the backlog of all the other Windows and non-Windows stuff that was committed in my branch long before new-enum was even started). How the game works is: 1. existing backlog gets cleared, including backends/core bugfixes and bridging the gap with -pbatard, with regular releases occuring 2. _then_ proposed brand new developments (such as new-enum or hp) are posted, especially if it relies/is likely to rely on elements from the backlog. This way, the backlog is not increased further and maintainers don't have another extra item to consider. In case you haven't noticed, what bogs down libusb is that we have too many things up in the air for the libusb maintainers to handle them all (which Peter also just confirmed). If I could have posted a nice hotplug patch, I would have done so a long time ago. But of course, the real kicker is, Nokia _did_ post a patch, pretty much the way you want, against 1.0.8, back in August [1]. Now, isn't that interesting? Let us have a look see what happened to it... First Nathan and Ludovic reply, with a few remarks/notes for improvement, but overall enthusiastic (i.e. patch is not being dismissed as something they cannot work with). And then Peter speaks [2]: "I view the patch as a first communication, but I think it's a little too early (much Windows going on now) and judging from initial comments the implementation might not be so clean. But it's a start! Something we can discuss. I hope you'll participate." So, the libusb maintainers (though I think Peter only turned official maintainer a bit later) pretty much said "We'd rather look at it _after_ we're done with Windows". If that's the official libusb line, fair enough. But then, considering the above, plus the fact that somebody needed to compensate for the lack of RERO in libusb, so that Nokia's effort wouldn't be wasted (I very much doubt they want to resubmit an hp patch every few months), plus the fact that, considering libusb's history, I (correctly) anticipated that no further discussion would actually occur on this proposal, plus the fact that we would need a Windows part for hp (which the Nokia proposal was missing), plus the fact that the latter would require a reworking of Windows enum, plus the fact that in order to obtain feedback on this new implementation, so as to ensure good quality by the time it would be considered for mainline integration, it had to be tested within my semi-official branch (which would also mean that, once the previous Windows patches had been dealt with, I could propose new-enum in the continuity), it made NO SENSE WHATSOEVER to create a topic branch off mainline for hp. And for the record, I'm not refusing to create topic branch off mainline on principle. If it were any other project but libusb, that's what I'd be doing, and being in a position to create topic-branches against mainline is precisely what I was expecting to be able to do when I started contributing to libusb. It's only because libusb is unable to deal with topic branches in a timely manner, that creating them off mainline makes very little sense, as it directly reduces the time contributors can spend on development from having to handle months of potential conflicts when these branches are finally integrated and/or not being able to rely on one branch to be integrated quickly for new developments that depend on it. But sure, if you want to pretend that I should have ignored Peter's comments with regards to new development having to wait until existing Windows was cleared up, and, against his wishes, just proposed a new-enum patch against mainline, which would have been integrated in less than 1 month (whereas almost none of the Windows patches that I ever proposed have ever been integrated in less than 3 months at best), so that hotplug against mainline could happen and get tested, then I really want to visit the fantasy land you live in. So you can really take your (once more absolute) "That's how the game works". When the half the pieces are missing, and people don't play by the official rules (by blatantly ignoring RERO), that is NOT how the game works. >> So could we actually properly and logically try to consider looking >> ahead for once, and carefully evaluate the merits of a proposal? > > I didn't see a proposed patch. I'm not interested in discussing > software development philosophies with you, sorry. Then stop shoving absolute philosophies (or absolute predictions of the future) down our throat as a "justification" of why _you_ want or don't want to see something in libusb, and let people like Nathan answer questions for themselves. Regards, /Pete [1] http://marc.info/?t=128091602200002&r=1&w=2 [2] http://marc.info/?l=libusb-devel&m=128231941012624&w=2 |
From: Peter S. <pe...@st...> - 2011-02-27 01:53:10
|
Pete Batard wrote: > Since when is _any_ performance improvement .. not worthy of > consideration then? Since it's a micro optimization. > > That's how the game works. Patch is posted, patch is reviewed, > > patch is updated, loop a bit, patch is committed. > > No, that's not how the game works when maintainers are 6 months to > one year behind the curve, Hans' patches will e.g. need less effort and fewer iterations so are likely to jump that curve. > I cannot post a hp patch without new enum, and I have been waiting > _months_ for mainline to be in a state where new-enum can be proposed The only reason you have to wait is IMO that you do not want to use git features that would easily allow working around any wait states. Git really can be a great help when working in a bursty environment. > In case you haven't noticed, what bogs down libusb is that we have too > many things up in the air for the libusb maintainers to handle them all > (which Peter also just confirmed). I'm afraid you read too much into what I wrote. libusb is not the only thing under my nose, as I am sure is the case for you and everyone else. > being in a position to create topic-branches against mainline is > precisely what I was expecting to be able to do when I started > contributing to libusb. You're as able, if not more, to post patches and publish git repos as anyone else. And because I think it's a good idea to keep commits close together I set up libusb-pbatard. Trust me, you really do get to choose what goes in that repo. And you do. And that's good! > months of potential conflicts Only since you don't want to take advantage of git. > if you want to pretend that I should have ignored Peter's comments > with regards to new development having to wait until existing > Windows was cleared up, and, against his wishes, just proposed a > new-enum patch against mainline, That would have been great. I don't have an overview of the new code at all now, and a patch would allow review to start. There was a report on IRC of a regression with new enum recently, hopefully it'll make it to the mailing list also. Before that I was fairly eager to get new enum in, because of the issues that were identified with the current code in libusb.git. > which would have been integrated in less than 1 month Noone can say if or when the patch gets to libusb.git. If it's ready it is ready. Just posting the patch doesn't mean it's ready. > When the half the pieces are missing, and people don't play by the > official rules No rules. Please read the license. > stop shoving absolute philosophies (or absolute predictions of the > future) down our throat as a "justification" of why _you_ want or > don't want to see something in libusb, and let people like Nathan > answer questions for themselves. Pete, I'm sure you agree that we can only benefit from having as much good input as possible on any issue. Especially from competent developers, regardless of their particular system preference or background. Computer programming is all the same, it's a stupid machine, and good developers like everyone on this mailing list already think about software in system-agnostic ways. I value Segher's input into discussions as well as yours, Nathan's and everybody elses. I expect that goes both (well, all:) ways. It's all good. I don't think it's neccessary to state that what one writes is opinion, that's a given for me, and everyone is entitled to their opinion. Also to expressing it. Nathan will fill in on the discussion I'm sure. I don't mind at all discussing together with you and Segher and everyone else meanwhile. (Segher is also very experienced with Mac OS X, so he often has good input on that topic.) //Peter |
From: Segher B. <se...@ke...> - 2011-02-27 02:49:53
|
> Until he speaks, couldn't Nathan have a different > opinion from yours there? He is fully capable of speaking for himself, I believe. > Wouldn't reducing the open time even further be worth something? 0.1us on 30ms is worth at most 1 in 300000 of code complication. That's about 0.2 characters in core.c . > Nathan clearly indicated "The only way I have come up with (so far) to > avoid a linear search would be to cache and hold a reference to each > device encountered". Yes, and? The problem here is not to use a hash -- unless you *make* it a problem by insisting on some generic hash thing. The problem is figuring out how to keep your data synched with the OS's view of the data, without using much resources, and what interface for that to use at the user and core (i.e., non-platform) side, in such a way that that interface can be implemented on all supported platforms, and probably all future platforms, in an efficient way as well. Abstracting your hashtables helps exactly as much as #define ZERO (1-1) > I cannot post a hp patch without new enum, Of course you can. If a patch needs to be applied after some other patch, you either point that out explicitly (with a link to that patch on the ML), or just send that patch in the same series. > How the game works is: <snip> No, that's not the game you can play, unless you propose taking over maintainership from Peter? > In case you haven't noticed, what bogs down libusb is that we have too > many things up in the air for the libusb maintainers to handle them all Including various older stuff, and half-baked stuff. If there is something you want included that is not yet included, send a tested patch against mainline, it will be *much* easier to include it in mainline that way (just "|git am" in your mail reader, or whatever Peter uses for that; *much* less work than looking at some old patch, figuring out if anyone liked or disliked it, what tree it originally applied against, if it still applies cleanly, and it still works correctly. The only downside is slightly higher mail volume. > But of course, the real kicker is, Nokia _did_ post a patch, pretty much > the way you want, against 1.0.8, back in August [1]. You didn't read what I said, did you. Do they (or anyone else) post improved versions of that patch now? If not, why on earth should Peter do the work to merge it? > Then stop shoving absolute philosophies (or absolute predictions of the > future) down our throat as a "justification" of why _you_ want or don't > want to see something in libusb, and let people like Nathan answer > questions for themselves. Sheesh, relax. I was merely pointing out what _you_ can do to move things forward. Complaining he does not do his job does not help; making it easier for him to do his job should help. Segher |
From: Pete B. <pb...@gm...> - 2011-02-27 23:02:06
|
On 2011.02.27 02:49, Segher Boessenkool wrote: > Yes, and? The problem here is not to use a hash -- unless you *make* > it a problem by insisting on some generic hash thing. I'm not insisting. I asked a simple question, to Nathan, and by extension to Linux people (if they _wanted_ to look into what may become our Linux HP implementation), as to whether they thought having hash table functions in core (which I picked up from glibc by the way - they're not even homegrown) could prove helpful. "I have pipe wrench to spare. Anybody think they could use a pipe wrench? If not, I'll just keep it in my toolbox for now...". All I asked. Now, if you want to answer "Nobody wants your lousy pipe wrench - and you're a idiot for proposing it", that's your right, and point taken. But I'll still rather wait for the go around to be complete, to decide what I do with it. One person still hasn't spoken. > No, that's not the game you can play, unless you propose taking > over maintainership from Peter? Glad you asked, so that I can clarify this item, since I believe there might be a few people thinking that I want to take over things. As much as I'd like to go all Palpatine on libusb, and make it enter a new reign of terror, I do not, under any circumstances, want to become a libusb maintainer. Now, this being said, do I think I would do a much better job than Peter? Hell yes! I'd place RERO and timely releases, instead of ill-placed cleanliness or a vision of "pure" software that ignores users requests, as a priority, which I am confident would help provide quality software that both meets our users requirement as well as make libusb contributors feel welcome here (i.e. no having to wait weeks or months for their patches to be processed, which is what Hans, Nokia and others have had or are currently going through). But do I think I would be a good maintainer? Hell no! We need people experienced for this job, like yourself or Enrico, and a lot more impartial or less troublesome than I am. >> In case you haven't noticed, what bogs down libusb is that we have too >> many things up in the air for the libusb maintainers to handle them all > > Including various older stuff, and half-baked stuff. If there is > something you want included that is not yet included, send a tested > patch against mainline, I did. Repeatedly. And they got nowhere. So excuse me for being cautious about wanting to trust the line above once again. > it will be *much* easier to include it in > mainline that way (just "|git am" in your mail reader, or whatever > Peter uses for that; *much* less work than looking at some old > patch, figuring out if anyone liked or disliked it, what tree it > originally applied against, if it still applies cleanly, and it > still works correctly. Where did you get the idea that I was asking any maintainers to pick up old patches? That there were unresolved contentions points, linked to old patches, that needed further discussion does not mean that, once solved, libusb maintainers should go back and fetch these 3-4 month old patch. Heck, I've been resubmitting the Windows patchset over and over again when I felt there was a window to do so. Why would I stop? As I said elsewhere, if you want patches against mainline, I don't have much of an issue producing them at short notice. The only issue is that, from looking at all the unresolved stuff we still had, it really didn't make sense to me to try my luck again. > The only downside is slightly higher mail volume. > >> But of course, the real kicker is, Nokia _did_ post a patch, pretty much >> the way you want, against 1.0.8, back in August [1]. > > You didn't read what I said, did you. I did read it. On the contrary, you clearly didn't read what I said because if you did hen the answer to your question: > Do they (or anyone else) > post improved versions of that patch now? would be explicit. Why would Nokia do so when Peter explicitly told them that, as long as Windows wasn't sorted, hotplug would have to wait? > If not, why on earth should Peter do the work to merge it? Again, where on earth did you get the idea that I am asking Peter to merge an old patch which he already rejected? Now, if you want an improved version of the Nokia patch against mainline, I can easily produce one after I'm done with the other stuff. No bother. > Sheesh, relax. I was merely pointing out what _you_ can do to > Complaining he does not do his job does not help; You don't understand, so let me make that clear. I'm not complaining that Peter doesn't do his job. I am complaining that, as far as I am concerned, he is doing a _bad_ job, with evidence to back it up taken from his various actions over the past year. I'm sorry to say that, but I see most of the difficulties we have into moving libusb forward as the result of Peter's actions (or lack thereof) and inflexibility on matters where a good maintainer should be flexible. Therefore I don't see it as a matter of making it easier on him, but it's a matter of making him change his approach to maintenance, so that we have a maintainer that actually does what I consider a proper job for a change (and I'm quite glad you joined to help there, though, after a few months, not much seems to have changed) Now, if you want to pretend that unilaterally rushing the removal of HID, because nobody else on this list thought that there was much point of removing it before the next release, just for the sake of "cleanliness", is the mark of a good maintainer, be my guest. When I see someone that I think does a bad job, I'm not going to beat about the bush. And when I consistently see it occur, over the course of a year, with next to no improvement, I might not be as patient with this as other people would. Regards, /Pete |
From: Peter S. <pe...@st...> - 2011-02-28 05:32:22
|
Pete Batard wrote: > > Do they (or anyone else) post improved versions of that patch now? > > Why would Nokia do so when Peter explicitly told them that, as long > as Windows wasn't sorted, hotplug would have to wait? Maybe if they wanted to try to influence priorities, by adding resources to the project. > rushing the removal of HID No news that I don't like the emulation idea. > whether you want to believe it or not, I actually tried to use git > features like rebase, cherry picking, etc, and found that they > didn't work for me (but feel free to FUD all you want there), Never doubted this. The only thing I doubt is that more eyes on the problems that you encountered would not have allowed them to be solved. > I have never had any problem producing patches for official integration I hope I never came off as suggesting that. You mention that you have to wait because of how the project is or is not moving, and what I am saying is that Git features are helpful for such situations, allowing to work on, regardless of wait states. When waits are over, Git again makes it easy to put pieces together. This is my experience, and I haven't had the same good experience from any other tools, so I'm promoting this one. You're of course free to use any tool. I keep promoting Git because I am still convinced that it could save you time overall. > could we please stop this idea that there is a problem with my > ability to generate patches against mainline? I'm sorry if I made you think that I have (had) this idea. Not at all. > >> months of potential conflicts > > > > Only since you don't want to take advantage of git. > > Only since you don't want to take advantage of RERO. Great. Now what? Hm? I was saying that I think you could avoid merge conflicts by using a particular workflow. > If it's acceptable to feed patches to the mailing list again, It can never be unacceptable to send patches. Quite the contrary. I do prefer git for delivery, but list is much better for review, and works fine for delivery too. > Now, if you have the IRC logs, I could of course use them. Sorry, no logs. I think it was a claim which failed with -pbatard.git but worked with (then) -stuge.git. I'll ask Matt again, next time I see him around. (I asked him a few times to post it on the list.) > I'm not complaining that Peter doesn't do his job. I am complaining > that, as far as I am concerned, he is doing a _bad_ job, .. > Therefore I don't see it as a matter of making it easier on him, > but it's a matter of making him change his approach to maintenance, .. > Everything is not black or white. Yep. //Peter |
From: Pete B. <pb...@gm...> - 2011-02-28 14:50:18
|
On 2011.02.28 05:32, Peter Stuge wrote: >> Why would Nokia do so when Peter explicitly told them that, as long >> as Windows wasn't sorted, hotplug would have to wait? > > Maybe if they wanted to try to influence priorities, by adding > resources to the project. Maybe they considered it, then considered that a good indication of the fate of any further hp proposal would be to observe what happened to the Windows integration, and how quickly things could get resolved there, which lead them to decide it wasn't worth adding resources when they would have to wait months on maintainers. >> rushing the removal of HID > > No news that I don't like the emulation idea. The important word there is "rush". It is a rushed proposal, no matter how you look at it. And it was done without any input from the actual backend developer. I must say, it is quite fascinating, really, to see how your principal argument against RERO always seems to have been that it could open the door for rushed developments and instability, yet the whole thing gets thrown to the wind as soon as you think you're not going to get the "clean" implementation of libusb you wished for, for 1.0.9. >> whether you want to believe it or not, I actually tried to use git >> features like rebase, cherry picking, etc, and found that they >> didn't work for me (but feel free to FUD all you want there), > > Never doubted this. The only thing I doubt is that more eyes on the > problems that you encountered would not have allowed them to be > solved. As I said, FUD. > I keep promoting Git because I am still convinced that > it could save you time overall. And my actual experience has convinced me otherwise. I'll probably try it again at some stage, but after we've done with trying to bridge the gap with -pbatard. >>>> months of potential conflicts >>> >>> Only since you don't want to take advantage of git. >> >> Only since you don't want to take advantage of RERO. Great. Now what? > > Hm? I was saying that I think you could avoid merge conflicts by > using a particular workflow. And I'm saying the exact same thing, by using the RERO workflow for maintenance work. The conflicts wouldn't be there in the first place if we were RERO. Heck, if libusb was RERO, you would have had a closer look at HID one year ago, because we would all have to do so for release, and therefore, if you presented a case for HID removal that was accepted at that time, it wouldn't have had to be something rushed at the last minute. RERO would have helped achieve the goal you wanted, and kept everybody happy (because our users wouldn't have had 12 months to start using HID before finding out it was meant to be removed), as well as avoided requests and debate for HID on OS-X. Yet, despite continued evidence, you still don't seem to want to consider that RERO could be helpful, since, yet again, you've just been ignoring similar advice (given by other people than me) with regards to keeping a strerror in 1.0.9, for the sake of "cleanliness" and not having something flagged as "experimental" (hint: removing strerror has very little to do with "safety"). Ever heard of post mortem? Good way to find out how one's approach fared, and try to take corrective action if needed. >> If it's acceptable to feed patches to the mailing list again, > > It can never be unacceptable to send patches. Quite the contrary. Then please don't go saying, when people send patches in "I think it's a little too early" and point to stuff that needs to be completed before review can start on a patch. >> I'm not complaining that Peter doesn't do his job. I am complaining >> that, as far as I am concerned, he is doing a _bad_ job, > .. >> Therefore I don't see it as a matter of making it easier on him, >> but it's a matter of making him change his approach to maintenance, > .. >> Everything is not black or white. > > Yep. Except, if you want evidence of your doing what I think is a very detrimental job to libusb and/or the Windows integration process, you just have to ask. I have some. I am not saying "Peter is doing a bad job", and stopping there. I am saying "Based on the evidence I have accumulated over the past year, Peter appears to be doing a bad job". Since you are now an official maintainer, where failures are of even greater consequences, and since we haven't seen much of any improvements with regards to moving libusb forward, I think you might want to at least consider changing your approach, especially as people point to you, time and time again, how it doesn't seem to work for the best. Regards, /Pete |
From: Peter S. <pe...@st...> - 2011-02-26 18:25:13
|
Pete Batard wrote: > could we actually properly and logically try to consider looking > ahead for once I have enough under my nose. Time for hotplug will come. //Peter |
From: Pete B. <pb...@gm...> - 2011-02-27 23:01:57
|
On 2011.02.27 01:53, Peter Stuge wrote: > Hans' patches will e.g. need less effort and fewer iterations so are > likely to jump that curve. Yay, one set of patches is likely to take less than a few months to be integrated! Libusb has come so far... > The only reason you have to wait is IMO that you do not want to use > git features that would easily allow working around any wait states. > Git really can be a great help when working in a bursty environment. The only reason libusb integration doesn't work, and people have to wait, IMO, is because you are 100% advert to RERO (well, let's make 99%, since it's apparently OK to rush a patch like HID removal when convenient). Of course, the difference is that, whether you want to believe it or not, I actually tried to use git features like rebase, cherry picking, etc, and found that they didn't work for me (but feel free to FUD all you want there), and, more importantly, whether or not I use git features only affects me (as long as I can produce patch for official, which I can either way), whereas not using RERO affects everyone depending on libusb. Just to make this clear I have never had any problem producing patches for official integration at short notice. I found that it became less of a pain to produce them by not using git, so that's what I ended up doing, but they really haven't been much of a problem to produce. The only reason you haven't seen Windows patches against official lately, is because: 1. there was enough in the backlog that needed clearing, and the consensus was that 1.0.9 would focus on that, without trying to add new stuff from -pbatard that could wait. Therefore there was no reason for me to resubmit a new set of Windows patches to add to the confusion. 2. questions were still left open from the previous patchset, such as CRLF in the repo, .def generation, xusb integration, project files integration, etc, and these questions have only been solved _very recently_. So could we please stop this idea that there is a problem with my ability to generate patches against mainline? It was only a matter of me deciding, based on both current and past libusb events, as to whether the time was ripe to release a new Windows patchset. > You're as able, if not more, to post patches and publish git repos as > anyone else. And because I think it's a good idea to keep commits > close together I set up libusb-pbatard. Trust me, you really do get > to choose what goes in that repo. And you do. And that's good! Indeed. And then I produce patchsets from that repo against mainline, once, twice, three times, four times, five times... and most of it gets _ignored_. And that's bad! >> months of potential conflicts > > Only since you don't want to take advantage of git. Only since you don't want to take advantage of RERO. Great. Now what? >> if you want to pretend that I should have ignored Peter's comments >> with regards to new development having to wait until existing >> Windows was cleared up, and, against his wishes, just proposed a >> new-enum patch against mainline, > > That would have been great. I don't have an overview of the new code > at all now, and a patch would allow review to start. Brilliant. If it's acceptable to feed patches to the mailing list again, expect hotplug, topology, log timestamping, toggleable logging and probably a bunch of other stuff as soon as I'm done with the current exercise. I've been holding on for the right time to submit those, but I do very much like that new song I hear now about shooting away and not worrying about whatever else is happening. > There was a report on IRC of a regression with new enum recently, > hopefully it'll make it to the mailing list also. Before that I was > fairly eager to get new enum in, because of the issues that were > identified with the current code in libusb.git. Oh no, there's a potential bug in new enum! Let me spare you the suspense here, there's probably more than one bug left in new enum, and there's probably still more than one in old enum, as well as in the rest of the code. And I also liked your little FUD exercise on the MinGW issue (yes, MinGW encompasses both of MinGW32 and MinGW64, though I am grateful that you didn't end up blaming me as the culprit of the confusion, for not having written MinGW64 in the summary). Now, if you have the IRC logs, I could of course use them. > No rules. Please read the license. 1. Tell that to Segher :) 2. Please check what other projects do, and how reactive they are on proposed patches. Yes, you're free to do whatever you want as a maintainer, which you have just demonstrated. Does it mean that doing it your way always works out for the best? No. > and good developers [blah blah blah] (i.e. "fit in a specific category") Again, absolutes. You do realize, I hope, that whatever pattern you throw out, there will be "good" developers that do not fit that category, and "bad" developers that fit it. The problem I always have with absolute is that they show a very simplistic (and reduced) view of the world. Please grow up. Everything is not black or white. Regards, /Pete |
From: Nathan H. <hj...@me...> - 2011-02-27 23:38:52
|
I don't yet know what I want to use to store devices references. I will probably wait until we are ready to move on hotplug support to make a decision. I will do some testing in the meantime to see what would fit best. For now I may add a reference to the device location to the matching dictionary passed to IOServiceGetMatchingServices. This may or may not be faster than iterating through the devices on the libusb side. -Nathan On Feb 27, 2011, at 4:02 PM, Pete Batard wrote: > On 2011.02.27 02:49, Segher Boessenkool wrote: >> Yes, and? The problem here is not to use a hash -- unless you *make* >> it a problem by insisting on some generic hash thing. > > I'm not insisting. I asked a simple question, to Nathan, and by > extension to Linux people (if they _wanted_ to look into what may become > our Linux HP implementation), as to whether they thought having hash > table functions in core (which I picked up from glibc by the way - > they're not even homegrown) could prove helpful. > > "I have pipe wrench to spare. Anybody think they could use a pipe > wrench? If not, I'll just keep it in my toolbox for now...". > All I asked. > > Now, if you want to answer "Nobody wants your lousy pipe wrench - and > you're a idiot for proposing it", that's your right, and point taken. > But I'll still rather wait for the go around to be complete, to decide > what I do with it. One person still hasn't spoken. > >> No, that's not the game you can play, unless you propose taking >> over maintainership from Peter? > > Glad you asked, so that I can clarify this item, since I believe there > might be a few people thinking that I want to take over things. > > As much as I'd like to go all Palpatine on libusb, and make it enter a > new reign of terror, I do not, under any circumstances, want to become a > libusb maintainer. > > Now, this being said, do I think I would do a much better job than > Peter? Hell yes! I'd place RERO and timely releases, instead of > ill-placed cleanliness or a vision of "pure" software that ignores users > requests, as a priority, which I am confident would help provide quality > software that both meets our users requirement as well as make libusb > contributors feel welcome here (i.e. no having to wait weeks or months > for their patches to be processed, which is what Hans, Nokia and others > have had or are currently going through). > But do I think I would be a good maintainer? Hell no! We need people > experienced for this job, like yourself or Enrico, and a lot more > impartial or less troublesome than I am. > >>> In case you haven't noticed, what bogs down libusb is that we have too >>> many things up in the air for the libusb maintainers to handle them all >> >> Including various older stuff, and half-baked stuff. If there is >> something you want included that is not yet included, send a tested >> patch against mainline, > > I did. Repeatedly. And they got nowhere. So excuse me for being cautious > about wanting to trust the line above once again. > >> it will be *much* easier to include it in >> mainline that way (just "|git am" in your mail reader, or whatever >> Peter uses for that; *much* less work than looking at some old >> patch, figuring out if anyone liked or disliked it, what tree it >> originally applied against, if it still applies cleanly, and it >> still works correctly. > > Where did you get the idea that I was asking any maintainers to pick up > old patches? That there were unresolved contentions points, linked to > old patches, that needed further discussion does not mean that, once > solved, libusb maintainers should go back and fetch these 3-4 month old > patch. Heck, I've been resubmitting the Windows patchset over and over > again when I felt there was a window to do so. Why would I stop? > > As I said elsewhere, if you want patches against mainline, I don't have > much of an issue producing them at short notice. The only issue is that, > from looking at all the unresolved stuff we still had, it really didn't > make sense to me to try my luck again. > >> The only downside is slightly higher mail volume. >> >>> But of course, the real kicker is, Nokia _did_ post a patch, pretty much >>> the way you want, against 1.0.8, back in August [1]. >> >> You didn't read what I said, did you. > > I did read it. On the contrary, you clearly didn't read what I said > because if you did hen the answer to your question: > >> Do they (or anyone else) >> post improved versions of that patch now? > > would be explicit. Why would Nokia do so when Peter explicitly told them > that, as long as Windows wasn't sorted, hotplug would have to wait? > >> If not, why on earth should Peter do the work to merge it? > > Again, where on earth did you get the idea that I am asking Peter to > merge an old patch which he already rejected? Now, if you want an > improved version of the Nokia patch against mainline, I can easily > produce one after I'm done with the other stuff. No bother. > >> Sheesh, relax. I was merely pointing out what _you_ can do to >> Complaining he does not do his job does not help; > > You don't understand, so let me make that clear. > > I'm not complaining that Peter doesn't do his job. I am complaining > that, as far as I am concerned, he is doing a _bad_ job, with evidence > to back it up taken from his various actions over the past year. I'm > sorry to say that, but I see most of the difficulties we have into > moving libusb forward as the result of Peter's actions (or lack thereof) > and inflexibility on matters where a good maintainer should be flexible. > Therefore I don't see it as a matter of making it easier on him, but > it's a matter of making him change his approach to maintenance, so that > we have a maintainer that actually does what I consider a proper job for > a change (and I'm quite glad you joined to help there, though, after a > few months, not much seems to have changed) > > Now, if you want to pretend that unilaterally rushing the removal of > HID, because nobody else on this list thought that there was much point > of removing it before the next release, just for the sake of > "cleanliness", is the mark of a good maintainer, be my guest. > > When I see someone that I think does a bad job, I'm not going to beat > about the bush. And when I consistently see it occur, over the course of > a year, with next to no improvement, I might not be as patient with this > as other people would. > > Regards, > > /Pete > > ------------------------------------------------------------------------------ > Free Software Download: Index, Search & Analyze Logs and other IT data in > Real-Time with Splunk. Collect, index and harness all the fast moving IT data > generated by your applications, servers and devices whether physical, virtual > or in the cloud. Deliver compliance at lower cost and gain new business > insights. http://p.sf.net/sfu/splunk-dev2dev > _______________________________________________ > Libusb-devel mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libusb-devel |
From: Pete B. <pb...@gm...> - 2011-02-27 23:53:34
|
On 2011.02.27 23:38, Nathan Hjelm wrote: > I will probably wait until we are ready to move on hotplug support to make a decision. That's fine. Thanks for the reply. Regards, /Pete |