As we all know, our network code is notoriously slow. That is bad, since this code is going to be distributed all across a network. It will slow down each host a helluva lot, especially during compilation of other programs (As i have noticed). So, i have taken a few notes on things we can do to check the efficiency.
Here it goes:
1. Efficiency of current design
A. Linked Queue
i. singly linked vs. doubly linked
ii. adding nodes at the front vs. the back
iii. keeping track of the front and last node, so as to avoid having to walk the list to get to the end.
2. How can we test and improve?
A. Keep track of how long the list gets during a network speed test (ie, keep track of maxsize and report it upon pressing CTRL-C, while running ping -f among other things).
B. If the data is convergent, consider using a static sized array.
i. resizable array?
Another thing to check in our design is the efficiency of linux's pthreads. Can we do this project with out it? IS it better off with or without?
Any volunteer to help test this?
--red0x
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
This message got to me for some reason, so I thought I might as well weigh in.
Before saying its too slow, can you define how you arrived at that conclusion ?
You might try removing the threads, and just grab packets off libpcap...
see if you notice a performance improvement without any processing.
It's entirely possible Linux pthreads is too slow for this. From some time ago,
pthreads are implmented using a single process context, thus if a call requiring
I/O (blocking) is used, all threads are blocked. I thought that would've been
fixed in more recent pthread implementations, but heck if I know for sure. You could
look at the pthread library to check and make sure.
Or you can spawn processes in place of threads. Turn those queues into IPC sockets.
Just a few suggestions. Good luck
-davis
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
About how I arrived at this conclusion: I ran agent compiled on my box and tried doing other things. agent eats about 45% of my CPU when nothing is going on. :(
about pthread and a single process: when i run agent, i see 6 agent process, i assume, corresponding to 6 threads we make. Which ones, i dont know. More research is needed.
--red0x
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hey all,
As we all know, our network code is notoriously slow. That is bad, since this code is going to be distributed all across a network. It will slow down each host a helluva lot, especially during compilation of other programs (As i have noticed). So, i have taken a few notes on things we can do to check the efficiency.
Here it goes:
1. Efficiency of current design
A. Linked Queue
i. singly linked vs. doubly linked
ii. adding nodes at the front vs. the back
iii. keeping track of the front and last node, so as to avoid having to walk the list to get to the end.
2. How can we test and improve?
A. Keep track of how long the list gets during a network speed test (ie, keep track of maxsize and report it upon pressing CTRL-C, while running ping -f among other things).
B. If the data is convergent, consider using a static sized array.
i. resizable array?
Another thing to check in our design is the efficiency of linux's pthreads. Can we do this project with out it? IS it better off with or without?
Any volunteer to help test this?
--red0x
This message got to me for some reason, so I thought I might as well weigh in.
Before saying its too slow, can you define how you arrived at that conclusion ?
You might try removing the threads, and just grab packets off libpcap...
see if you notice a performance improvement without any processing.
It's entirely possible Linux pthreads is too slow for this. From some time ago,
pthreads are implmented using a single process context, thus if a call requiring
I/O (blocking) is used, all threads are blocked. I thought that would've been
fixed in more recent pthread implementations, but heck if I know for sure. You could
look at the pthread library to check and make sure.
Or you can spawn processes in place of threads. Turn those queues into IPC sockets.
Just a few suggestions. Good luck
-davis
About how I arrived at this conclusion: I ran agent compiled on my box and tried doing other things. agent eats about 45% of my CPU when nothing is going on. :(
about pthread and a single process: when i run agent, i see 6 agent process, i assume, corresponding to 6 threads we make. Which ones, i dont know. More research is needed.
--red0x