Menu
▾
▴
Re: [Pytables-users] Expr performance with Tables on multicore machines
|
From: Anthony S. <sc...@gm...> - 2012-05-14 20:12:56
|
On Mon, May 14, 2012 at 3:05 PM, Francesc Alted <fa...@py...> wrote: [snip] However, do not expect to use all your cores at full speed in this cases, > as the reductions in numexpr can only make use of one thread (this is > because this has not been implemented yet, not due to a intrinsic > limitation of numexpr). > Hello Francesc, Not to side track the discussion too much, but is there a ticket open for this in numexpr? It seems that at least for certain reductions (sum, mult, etc), splitting this up over many cores would be pretty easy. I may to wrong about this though ;) Be Well Anthony > > Francesc > > > > > I hope this helps. If you need other tips on speeding up the > sum operation, please let us know. > > Be Well > Anthony > > Timer unit: 1e-06 s > > File: pytables_expr_test.py > Function: fn at line 66 > Total time: 1.63254 s > > Line # Hits Time Per Hit % Time Line Contents > ============================================================== > 66 def fn(p, h5table): > 67 ''' > 68 actual function > we are going to minimize. It consists of > 69 the pytables > Table object and a list of parameters. > 70 ''' > 71 1 14 14.0 0.0 uv = > h5table.colinstances > 72 > 73 # store parameters in > a dict object with names > 74 # like p0, p1, p2, > etc. so they can be used in > 75 # the Expr object. > 76 4 21 5.2 0.0 for i in > xrange(len(p)): > 77 3 19 6.3 0.0 k = 'p'+str(i) > 78 3 14 4.7 0.0 uv[k] = p[i] > 79 > 80 # systematic shift on > b is a polynomial in a > 81 1 4 4.0 0.0 db = 'p0 * a*a + p1 > * a + p2' > 82 > 83 # the element-wise > function > 84 1 6 6.0 0.0 fn_str = '(a - (b + > %s))**2' % db > 85 > 86 1 16427 16427.0 1.0 expr = > Expr(fn_str,uservars=uv) > 87 1 21438 21438.0 1.3 expr.eval() > 88 > 89 # returning the "sum > of squares" > 90 1 1594600 1594600.0 97.7 return sum(expr) > > > > > On Mon, May 14, 2012 at 1:59 PM, Johann Goetz <jg...@uc...> wrote: > >> SHORT VERSION: >> >> Please take a look at the fn() function in the attached file (pasted >> below). When I run this with 10M events or more I notice that the total CPU >> usage never goes above the percentage I get using single-threaded eval(). >> Am I at some other limit or can I improve performance by doing something >> else? >> >> LONG VERSION: >> >> I have been trying to use the tables.Expr object to speed up a >> sophisticated calculation over an entire dataset (a pytables Table object). >> The calculation took so long that I had to create a simple example to make >> sure I knew what I was doing. I apologize in advance for the lengthy code >> below, but I wanted the example to mimic exactly what I'm trying to do and >> to be totally self-contained. >> >> I have attached a file (and pasted it below) in which I create a hdf5 >> file with a single large Table of two columns. As you can see, I'm not >> worried about writing speed at all - I'm concerned about read speed. >> >> I would like to draw your attention to the fn() function. This is where I >> evaluate a "chi-squared" value on the dataset. My strategy is to populate >> the "h5table.colinstances" dict object with several parameters which I call >> p0, p1, etc and then create the Expr object using these and the column >> names from the Table. >> >> If I create 10M rows (77 MB file) in the Table (with the command below), >> the evaluation seems to be CPU bound (one of my cores is at 100% - the >> others are idle) and it takes about 7 seconds (about 10 MB/s). Similarly, I >> get about 70 seconds for 100M events. >> >> python pytables_expr_test.py 10000000 >> python pytables_expr_test.py 100000000 >> >> So my question: It seems to me that I am not fully using the CPU power >> available on my computer (see next paragraph). Am I missing something or >> doing something wrong in the fn() function below? >> >> A few side-notes: My hard-disk is capable of over 200 MB/s in sequential >> reading (sustained and tested with large files using the iozone program), I >> have two 4-core CPU's on this machine but the total CPU usage during eval() >> never goes above the percentage I get using single-threaded mode with >> "numexpr.set_num_threads(1)". >> >> I am using pytables 2.3.1 and numexpr 2.0.1 >> >> -- >> Johann T. Goetz, PhD. <http://sites.google.com/site/theodoregoetz/> >> jg...@uc... >> Nefkens Group, UCLA Dept. of Physics & Astronomy >> Hall-B, Jefferson Lab, Newport News, VA >> >> >> ### BEGIN file: pytables_expr_test.py >> >> from tables import openFile, Expr >> >> ### Control of the number of threads used when issuing the >> ### Expr::eval() command >> #import numexpr >> #numexpr.set_num_threads(2) >> >> def create_ntuple_file(filename, npoints, pmodel): >> ''' >> create an hdf5 file with a single table which contains >> npoints number of rows of type row_t (defined below) >> ''' >> from numpy import random, poly1d >> from tables import IsDescription, Float32Col >> >> class row_t(IsDescription): >> ''' >> the rows of the table to be created >> ''' >> a = Float32Col() >> b = Float32Col() >> >> def append_row(h5row, pmodel): >> ''' >> consider this a single "event" being appended >> to the dataset (table) >> ''' >> h5row['a'] = random.uniform(0,10) >> >> h5row['b'] = h5row['a'] # reality (or model) >> h5row['b'] = h5row['b'] - poly1d(pmodel)(h5row['a']) # systematics >> h5row['b'] = h5row['b'] + random.normal(0,0.1) # noise >> >> h5row.append() >> >> h5file = openFile(filename, 'w') >> h5table = h5file.createTable('/', 'table', row_t, "Data") >> h5row = h5table.row >> >> # recording data to file... >> for n in xrange(npoints): >> append_row(h5row, pmodel) >> >> h5file.close() >> >> def create_ntuple_file_if_needed(filename, npoints, pmodel): >> ''' >> looks to see if the file is already there and if so, >> it makes sure its the right size. Otherwise, it >> removes the existing file and creates a new one. >> ''' >> from os import path, remove >> >> print 'model parameters:', pmodel >> >> if path.exists(filename): >> h5file = openFile(filename, 'r') >> h5table = h5file.root.table >> if len(h5table) != npoints: >> h5file.close() >> remove(filename) >> >> if not path.exists(filename): >> create_ntuple_file(filename, npoints, pmodel) >> >> def fn(p, h5table): >> ''' >> actual function we are going to minimize. It consists of >> the pytables Table object and a list of parameters. >> ''' >> uv = h5table.colinstances >> >> # store parameters in a dict object with names >> # like p0, p1, p2, etc. so they can be used in >> # the Expr object. >> for i in xrange(len(p)): >> k = 'p'+str(i) >> uv[k] = p[i] >> >> # systematic shift on b is a polynomial in a >> db = 'p0 * a*a + p1 * a + p2' >> >> # the element-wise function >> fn_str = '(a - (b + %s))**2' % db >> >> expr = Expr(fn_str,uservars=uv) >> expr.eval() >> >> # returning the "sum of squares" >> return sum(expr) >> >> if __name__ == '__main__': >> ''' >> usage: >> python pytables_expr_test.py [npoints] >> >> Hint: try this with 10M points >> ''' >> from sys import argv >> from time import time >> >> npoints = 1000000 >> if len(argv) > 1: >> npoints = int(argv[1]) >> >> filename = 'tmp.'+str(npoints)+'.hdf5' >> >> pmodel = [-0.04,0.002,0.001] >> >> print 'creating file (if it doesn\'t exist)...' >> create_ntuple_file_if_needed(filename, npoints, pmodel) >> >> h5file = openFile(filename, 'r') >> h5table = h5file.root.table >> >> print 'evaluating function' >> starttime = time() >> print fn([0.,0.,0.], h5table) >> print 'evaluated file in',time()-starttime,'seconds.' >> >> #EOF >> >> >> >> ------------------------------------------------------------------------------ >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. Discussions >> will include endpoint security, mobile security and the latest in malware >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> Pytables-users mailing list >> Pyt...@li... >> https://lists.sourceforge.net/lists/listinfo/pytables-users >> >> > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > > > > _______________________________________________ > Pytables-users mailing lis...@li...://lists.sourceforge.net/lists/listinfo/pytables-users > > > > -- > Francesc Alted > > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Pytables-users mailing list > Pyt...@li... > https://lists.sourceforge.net/lists/listinfo/pytables-users > > |
