Thread: [sqlmap-users] CPU utilization
Brought to you by:
inquisb
Menu
▾
▴
sqlmap-users
|
From: Kasper F. <th...@ma...> - 2010-03-15 11:55:30
|
Hello sqlmap users. I have a question about how sqlmap utilizes the cpu. On my computer the CPU being utilized is nearly always 50%, and since I have two cores I suspect some thread to be in deadlock or maybe doing busy-waiting. If using busy-waiting, I would suggest using an event-based approach instead. /Kasper |
|
From: Miroslav S. <mir...@gm...> - 2010-03-15 12:18:07
|
Dear Kasper. Python has an "technical" issue dealing with multithreading programs. In part that it can't run threads on multiple cores, it has a really nasty "GIL" problem which is discussed here: http://www.snaplogic.com/blog/?p=94. Threading of sqlmap really speeds it up, but the side effect is that CPU-throttling you are talking about, especially in a high number of threads used. Kind regards. On Mon, Mar 15, 2010 at 12:55 PM, Kasper Føns <th...@ma...> wrote: > Hello sqlmap users. > > I have a question about how sqlmap utilizes the cpu. On my computer the > CPU being utilized is nearly always 50%, and since I have two cores I > suspect some thread to be in deadlock or maybe doing busy-waiting. > If using busy-waiting, I would suggest using an event-based approach > instead. > > /Kasper > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > sqlmap-users mailing list > sql...@li... > https://lists.sourceforge.net/lists/listinfo/sqlmap-users > -- Miroslav Stampar E-mail / Jabber: miroslav.stampar (at) gmail.com Mobile: +385921010204 (HR 0921010204) PGP Key ID: 0xB5397B1B |
|
From: Kasper F. <th...@ma...> - 2010-03-15 12:24:53
|
Dead Miroslav That was not the point I was trying to make. It would ofcourse be nice to utilize the 100% of my computer when a program wants to, but I can see that python is crippled in the matter. But what I was asking about was: I think it is strange that sqlmap uses 50% of my cpu - What operations could require that much cpu? It seems that if a connection to a target is slow, sqlmap still uses 50% of the cpu. Is it besy-waiting for the thread that is doing the I/O operation to complete? I'll try again: I would see sqlmap's main bottlenet as the internet connection. I can't find anything to justify to use my CPU as much as it does. Therefore I think some thread is besy-waiting, which really cripples the system (especially if I only had 1 CPU). So my question is, is it busy-waiting? PS: Thanks for the nice link. Was interesting reading. /Kasper On 15-03-2010 13:17, Miroslav Stampar wrote: > Dear Kasper. > > Python has an "technical" issue dealing with multithreading programs. > In part that it can't run threads on multiple cores, it has a really > nasty "GIL" problem which is discussed here: > http://www.snaplogic.com/blog/?p=94. Threading of sqlmap really speeds > it up, but the side effect is that CPU-throttling you are talking > about, especially in a high number of threads used. > > Kind regards. > > On Mon, Mar 15, 2010 at 12:55 PM, Kasper Føns<th...@ma...> wrote: > >> Hello sqlmap users. >> >> I have a question about how sqlmap utilizes the cpu. On my computer the >> CPU being utilized is nearly always 50%, and since I have two cores I >> suspect some thread to be in deadlock or maybe doing busy-waiting. >> If using busy-waiting, I would suggest using an event-based approach >> instead. >> >> /Kasper >> >> ------------------------------------------------------------------------------ >> Download Intel® Parallel Studio Eval >> Try the new software tools for yourself. Speed compiling, find bugs >> proactively, and fine-tune applications for parallel performance. >> See why Intel Parallel Studio got high marks during beta. >> http://p.sf.net/sfu/intel-sw-dev >> _______________________________________________ >> sqlmap-users mailing list >> sql...@li... >> https://lists.sourceforge.net/lists/listinfo/sqlmap-users >> >> > > > |
|
From: Miroslav S. <mir...@gm...> - 2010-05-13 14:15:57
|
Hi all. After the initial Kasper's "complaint", preliminary testing and some researching we've come to this: http://ted.onflash.org/2005/06/python-cpu-tuning.php The problem seems to be really in the "Python internal C FrameLoop" - or in plain speak, python runs the program as fast as it can without any real rest stops. Research results show that if we put a small time delay in page retrieval method, which is a main backbone function of the sqlmap, CPU intensivity dramatically decreases. Sample command used: time python sqlmap.py -u " http://xxx.xxx.xxx.xxx/somedir/get_int.php?id=1" --ignore-proxy --threads=2 --read-file /etc/shells --flush-session 1) With time.sleep(.001) (or in plain speak 1/1000 of a second) at end of getPage method in lib/request/connect module: 5.368u 1.100s 0:14.01 46.1% 0+0k 0+48io 0pf+0w 2) Without: 9.160u 1.932s 0:10.94 101.3% 0+0k 0+48io 0pf+0w You can spot the difference quite easy. So, the real question now is to put it as a default value there or leave the choice to the user. You can see that there is a slow down of 3 seconds after all, but you have to have on your mind that this was a test done on a local VM, while there shouldn't be such a difference in a real life case. Also, benefits are more then obvious (cca. 100% drops to less than 50%) Kind regards. On Mon, Mar 15, 2010 at 2:24 PM, Kasper Føns <th...@ma...> wrote: > Dead Miroslav > > That was not the point I was trying to make. It would ofcourse be nice to > utilize the 100% of my computer when a program wants to, but I can see that > python is crippled in the matter. But what I was asking about was: > I think it is strange that sqlmap uses 50% of my cpu - What operations > could require that much cpu? It seems that if a connection to a target is > slow, sqlmap still uses 50% of the cpu. Is it besy-waiting for the thread > that is doing the I/O operation to complete? > > I'll try again: > I would see sqlmap's main bottlenet as the internet connection. I can't > find anything to justify to use my CPU as much as it does. Therefore I think > some thread is besy-waiting, which really cripples the system (especially if > I only had 1 CPU). So my question is, is it busy-waiting? > > PS: Thanks for the nice link. Was interesting reading. > > /Kasper > > > On 15-03-2010 13:17, Miroslav Stampar wrote: > >> Dear Kasper. >> >> Python has an "technical" issue dealing with multithreading programs. >> In part that it can't run threads on multiple cores, it has a really >> nasty "GIL" problem which is discussed here: >> http://www.snaplogic.com/blog/?p=94. Threading of sqlmap really speeds >> it up, but the side effect is that CPU-throttling you are talking >> about, especially in a high number of threads used. >> >> Kind regards. >> >> On Mon, Mar 15, 2010 at 12:55 PM, Kasper Føns<th...@ma...> wrote: >> >> >>> Hello sqlmap users. >>> >>> I have a question about how sqlmap utilizes the cpu. On my computer the >>> CPU being utilized is nearly always 50%, and since I have two cores I >>> suspect some thread to be in deadlock or maybe doing busy-waiting. >>> If using busy-waiting, I would suggest using an event-based approach >>> instead. >>> >>> /Kasper >>> >>> >>> ------------------------------------------------------------------------------ >>> Download Intel® Parallel Studio Eval >>> Try the new software tools for yourself. Speed compiling, find bugs >>> proactively, and fine-tune applications for parallel performance. >>> See why Intel Parallel Studio got high marks during beta. >>> http://p.sf.net/sfu/intel-sw-dev >>> _______________________________________________ >>> sqlmap-users mailing list >>> sql...@li... >>> https://lists.sourceforge.net/lists/listinfo/sqlmap-users >>> >>> >>> >> >> >> >> > > -- Miroslav Stampar E-mail / Jabber: miroslav.stampar (at) gmail.com Mobile: +385921010204 (HR 0921010204) PGP Key ID: 0xB5397B1B |
|
From: Miroslav S. <mir...@gm...> - 2010-03-15 12:49:47
|
Hi Kasper. Threads are indeed in a blocking state while retrieving page content (basic socket blocking). Basic python urllib2 doesn't support asynchronous page retrievals, but we could put this on a feature request list (http://eventlet.net/doc/, http://twistedmatrix.com/trac/). Kind regards. On Mon, Mar 15, 2010 at 1:24 PM, Kasper Føns <th...@ma...> wrote: > Dead Miroslav > > That was not the point I was trying to make. It would ofcourse be nice to > utilize the 100% of my computer when a program wants to, but I can see that > python is crippled in the matter. But what I was asking about was: > I think it is strange that sqlmap uses 50% of my cpu - What operations could > require that much cpu? It seems that if a connection to a target is slow, > sqlmap still uses 50% of the cpu. Is it besy-waiting for the thread that is > doing the I/O operation to complete? > > I'll try again: > I would see sqlmap's main bottlenet as the internet connection. I can't find > anything to justify to use my CPU as much as it does. Therefore I think some > thread is besy-waiting, which really cripples the system (especially if I > only had 1 CPU). So my question is, is it busy-waiting? > > PS: Thanks for the nice link. Was interesting reading. > > /Kasper > > On 15-03-2010 13:17, Miroslav Stampar wrote: >> >> Dear Kasper. >> >> Python has an "technical" issue dealing with multithreading programs. >> In part that it can't run threads on multiple cores, it has a really >> nasty "GIL" problem which is discussed here: >> http://www.snaplogic.com/blog/?p=94. Threading of sqlmap really speeds >> it up, but the side effect is that CPU-throttling you are talking >> about, especially in a high number of threads used. >> >> Kind regards. >> >> On Mon, Mar 15, 2010 at 12:55 PM, Kasper Føns<th...@ma...> wrote: >> >>> >>> Hello sqlmap users. >>> >>> I have a question about how sqlmap utilizes the cpu. On my computer the >>> CPU being utilized is nearly always 50%, and since I have two cores I >>> suspect some thread to be in deadlock or maybe doing busy-waiting. >>> If using busy-waiting, I would suggest using an event-based approach >>> instead. >>> >>> /Kasper >>> >>> >>> ------------------------------------------------------------------------------ >>> Download Intel® Parallel Studio Eval >>> Try the new software tools for yourself. Speed compiling, find bugs >>> proactively, and fine-tune applications for parallel performance. >>> See why Intel Parallel Studio got high marks during beta. >>> http://p.sf.net/sfu/intel-sw-dev >>> _______________________________________________ >>> sqlmap-users mailing list >>> sql...@li... >>> https://lists.sourceforge.net/lists/listinfo/sqlmap-users >>> >>> >> >> >> > > -- Miroslav Stampar E-mail / Jabber: miroslav.stampar (at) gmail.com Mobile: +385921010204 (HR 0921010204) PGP Key ID: 0xB5397B1B |
|
From: Kasper F. <th...@ma...> - 2010-03-15 12:58:23
|
Hi I still don't get why sqlmap is using that much cpu then? On 15-03-2010 13:49, Miroslav Stampar wrote: > Hi Kasper. > > Threads are indeed in a blocking state while retrieving page content > (basic socket blocking). Basic python urllib2 doesn't support > asynchronous page retrievals, but we could put this on a feature > request list (http://eventlet.net/doc/, > http://twistedmatrix.com/trac/). > > Kind regards. > > On Mon, Mar 15, 2010 at 1:24 PM, Kasper Føns<th...@ma...> wrote: > >> Dead Miroslav >> >> That was not the point I was trying to make. It would ofcourse be nice to >> utilize the 100% of my computer when a program wants to, but I can see that >> python is crippled in the matter. But what I was asking about was: >> I think it is strange that sqlmap uses 50% of my cpu - What operations could >> require that much cpu? It seems that if a connection to a target is slow, >> sqlmap still uses 50% of the cpu. Is it besy-waiting for the thread that is >> doing the I/O operation to complete? >> >> I'll try again: >> I would see sqlmap's main bottlenet as the internet connection. I can't find >> anything to justify to use my CPU as much as it does. Therefore I think some >> thread is besy-waiting, which really cripples the system (especially if I >> only had 1 CPU). So my question is, is it busy-waiting? >> >> PS: Thanks for the nice link. Was interesting reading. >> >> /Kasper >> >> On 15-03-2010 13:17, Miroslav Stampar wrote: >> >>> Dear Kasper. >>> >>> Python has an "technical" issue dealing with multithreading programs. >>> In part that it can't run threads on multiple cores, it has a really >>> nasty "GIL" problem which is discussed here: >>> http://www.snaplogic.com/blog/?p=94. Threading of sqlmap really speeds >>> it up, but the side effect is that CPU-throttling you are talking >>> about, especially in a high number of threads used. >>> >>> Kind regards. >>> >>> On Mon, Mar 15, 2010 at 12:55 PM, Kasper Føns<th...@ma...> wrote: >>> >>> >>>> Hello sqlmap users. >>>> >>>> I have a question about how sqlmap utilizes the cpu. On my computer the >>>> CPU being utilized is nearly always 50%, and since I have two cores I >>>> suspect some thread to be in deadlock or maybe doing busy-waiting. >>>> If using busy-waiting, I would suggest using an event-based approach >>>> instead. >>>> >>>> /Kasper >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Download Intel® Parallel Studio Eval >>>> Try the new software tools for yourself. Speed compiling, find bugs >>>> proactively, and fine-tune applications for parallel performance. >>>> See why Intel Parallel Studio got high marks during beta. >>>> http://p.sf.net/sfu/intel-sw-dev >>>> _______________________________________________ >>>> sqlmap-users mailing list >>>> sql...@li... >>>> https://lists.sourceforge.net/lists/listinfo/sqlmap-users >>>> >>>> >>>> >>> >>> >>> >> >> > > > |
|
From: Alessandro T. <ale...@lo...> - 2010-03-15 13:25:54
|
2010/3/15 Kasper Føns <th...@ma...>: > Hi > > I still don't get why sqlmap is using that much cpu then? One hand on a profiler and dig in the code ;) |
|
From: Miroslav S. <mir...@gm...> - 2010-03-15 13:26:32
|
Hi. That's because, when you do the urllib2 page request, current thread goes into blocking state - no events here, just plain raw CPU power. If you do it with 30 threads for example, all of them go into blocking state. I repeat, python's standard module Urllib2 doesn't support asynchronous page retrievals (neither any other). Could you please be so kind and review sqlmap/lib/techniques/blind/inference.py, methods bisection(...) and getChar(...), and advise with some concrete solution. We'll be more than happy to hear from you. Kind regards. On Mon, Mar 15, 2010 at 1:58 PM, Kasper Føns <th...@ma...> wrote: > Hi > > I still don't get why sqlmap is using that much cpu then? > > On 15-03-2010 13:49, Miroslav Stampar wrote: >> >> Hi Kasper. >> >> Threads are indeed in a blocking state while retrieving page content >> (basic socket blocking). Basic python urllib2 doesn't support >> asynchronous page retrievals, but we could put this on a feature >> request list (http://eventlet.net/doc/, >> http://twistedmatrix.com/trac/). >> >> Kind regards. >> >> On Mon, Mar 15, 2010 at 1:24 PM, Kasper Føns<th...@ma...> wrote: >> >>> >>> Dead Miroslav >>> >>> That was not the point I was trying to make. It would ofcourse be nice to >>> utilize the 100% of my computer when a program wants to, but I can see >>> that >>> python is crippled in the matter. But what I was asking about was: >>> I think it is strange that sqlmap uses 50% of my cpu - What operations >>> could >>> require that much cpu? It seems that if a connection to a target is slow, >>> sqlmap still uses 50% of the cpu. Is it besy-waiting for the thread that >>> is >>> doing the I/O operation to complete? >>> >>> I'll try again: >>> I would see sqlmap's main bottlenet as the internet connection. I can't >>> find >>> anything to justify to use my CPU as much as it does. Therefore I think >>> some >>> thread is besy-waiting, which really cripples the system (especially if I >>> only had 1 CPU). So my question is, is it busy-waiting? >>> >>> PS: Thanks for the nice link. Was interesting reading. >>> >>> /Kasper >>> >>> On 15-03-2010 13:17, Miroslav Stampar wrote: >>> >>>> >>>> Dear Kasper. >>>> >>>> Python has an "technical" issue dealing with multithreading programs. >>>> In part that it can't run threads on multiple cores, it has a really >>>> nasty "GIL" problem which is discussed here: >>>> http://www.snaplogic.com/blog/?p=94. Threading of sqlmap really speeds >>>> it up, but the side effect is that CPU-throttling you are talking >>>> about, especially in a high number of threads used. >>>> >>>> Kind regards. >>>> >>>> On Mon, Mar 15, 2010 at 12:55 PM, Kasper Føns<th...@ma...> wrote: >>>> >>>> >>>>> >>>>> Hello sqlmap users. >>>>> >>>>> I have a question about how sqlmap utilizes the cpu. On my computer the >>>>> CPU being utilized is nearly always 50%, and since I have two cores I >>>>> suspect some thread to be in deadlock or maybe doing busy-waiting. >>>>> If using busy-waiting, I would suggest using an event-based approach >>>>> instead. >>>>> >>>>> /Kasper >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Download Intel® Parallel Studio Eval >>>>> Try the new software tools for yourself. Speed compiling, find bugs >>>>> proactively, and fine-tune applications for parallel performance. >>>>> See why Intel Parallel Studio got high marks during beta. >>>>> http://p.sf.net/sfu/intel-sw-dev >>>>> _______________________________________________ >>>>> sqlmap-users mailing list >>>>> sql...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/sqlmap-users >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >> >> >> > > -- Miroslav Stampar E-mail / Jabber: miroslav.stampar (at) gmail.com Mobile: +385921010204 (HR 0921010204) PGP Key ID: 0xB5397B1B |
|
From: Kasper F. <th...@ma...> - 2010-03-15 13:48:08
|
Hi I am no expert at all in Python. It was just my impression as a programmer that there is some kind of busy-waiting. Without knowing anything, I just threw in "Can this really be true?". Is there really no other way to implement fetching pages then using busy-waiting. Actually I just wanted to know if this could be implemented, so my hardware was not abused. But from what you tell me, it seems that urllib2 is fetching pages using busy-waiting and this is a module that cannot be changed. I checked the file you mentioned but I can't really read the code very well. /Kasper On 15-03-2010 14:26, Miroslav Stampar wrote: > Hi. > > That's because, when you do the urllib2 page request, current thread > goes into blocking state - no events here, just plain raw CPU power. > If you do it with 30 threads for example, all of them go into blocking > state. I repeat, python's standard module Urllib2 doesn't support > asynchronous page retrievals (neither any other). > > Could you please be so kind and review > sqlmap/lib/techniques/blind/inference.py, methods bisection(...) and > getChar(...), and advise with some concrete solution. We'll be more > than happy to hear from you. > > Kind regards. > > On Mon, Mar 15, 2010 at 1:58 PM, Kasper Føns<th...@ma...> wrote: > >> Hi >> >> I still don't get why sqlmap is using that much cpu then? >> >> On 15-03-2010 13:49, Miroslav Stampar wrote: >> >>> Hi Kasper. >>> >>> Threads are indeed in a blocking state while retrieving page content >>> (basic socket blocking). Basic python urllib2 doesn't support >>> asynchronous page retrievals, but we could put this on a feature >>> request list (http://eventlet.net/doc/, >>> http://twistedmatrix.com/trac/). >>> >>> Kind regards. >>> >>> On Mon, Mar 15, 2010 at 1:24 PM, Kasper Føns<th...@ma...> wrote: >>> >>> >>>> Dead Miroslav >>>> >>>> That was not the point I was trying to make. It would ofcourse be nice to >>>> utilize the 100% of my computer when a program wants to, but I can see >>>> that >>>> python is crippled in the matter. But what I was asking about was: >>>> I think it is strange that sqlmap uses 50% of my cpu - What operations >>>> could >>>> require that much cpu? It seems that if a connection to a target is slow, >>>> sqlmap still uses 50% of the cpu. Is it besy-waiting for the thread that >>>> is >>>> doing the I/O operation to complete? >>>> >>>> I'll try again: >>>> I would see sqlmap's main bottlenet as the internet connection. I can't >>>> find >>>> anything to justify to use my CPU as much as it does. Therefore I think >>>> some >>>> thread is besy-waiting, which really cripples the system (especially if I >>>> only had 1 CPU). So my question is, is it busy-waiting? >>>> >>>> PS: Thanks for the nice link. Was interesting reading. >>>> >>>> /Kasper >>>> >>>> On 15-03-2010 13:17, Miroslav Stampar wrote: >>>> >>>> >>>>> Dear Kasper. >>>>> >>>>> Python has an "technical" issue dealing with multithreading programs. >>>>> In part that it can't run threads on multiple cores, it has a really >>>>> nasty "GIL" problem which is discussed here: >>>>> http://www.snaplogic.com/blog/?p=94. Threading of sqlmap really speeds >>>>> it up, but the side effect is that CPU-throttling you are talking >>>>> about, especially in a high number of threads used. >>>>> >>>>> Kind regards. >>>>> >>>>> On Mon, Mar 15, 2010 at 12:55 PM, Kasper Føns<th...@ma...> wrote: >>>>> >>>>> >>>>> >>>>>> Hello sqlmap users. >>>>>> >>>>>> I have a question about how sqlmap utilizes the cpu. On my computer the >>>>>> CPU being utilized is nearly always 50%, and since I have two cores I >>>>>> suspect some thread to be in deadlock or maybe doing busy-waiting. >>>>>> If using busy-waiting, I would suggest using an event-based approach >>>>>> instead. >>>>>> >>>>>> /Kasper >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>>> Download Intel® Parallel Studio Eval >>>>>> Try the new software tools for yourself. Speed compiling, find bugs >>>>>> proactively, and fine-tune applications for parallel performance. >>>>>> See why Intel Parallel Studio got high marks during beta. >>>>>> http://p.sf.net/sfu/intel-sw-dev >>>>>> _______________________________________________ >>>>>> sqlmap-users mailing list >>>>>> sql...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/sqlmap-users >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>> >> >> > > > |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X