You can subscribe to this list here.
2010 
_{Jan}
(23) 
_{Feb}
(4) 
_{Mar}
(56) 
_{Apr}
(74) 
_{May}
(107) 
_{Jun}
(79) 
_{Jul}
(212) 
_{Aug}
(122) 
_{Sep}
(289) 
_{Oct}
(176) 
_{Nov}
(531) 
_{Dec}
(268) 

2011 
_{Jan}
(255) 
_{Feb}
(157) 
_{Mar}
(199) 
_{Apr}
(274) 
_{May}
(495) 
_{Jun}
(157) 
_{Jul}
(276) 
_{Aug}
(212) 
_{Sep}
(356) 
_{Oct}
(356) 
_{Nov}
(421) 
_{Dec}
(365) 
2012 
_{Jan}
(530) 
_{Feb}
(236) 
_{Mar}
(495) 
_{Apr}
(286) 
_{May}
(347) 
_{Jun}
(253) 
_{Jul}
(335) 
_{Aug}
(254) 
_{Sep}
(429) 
_{Oct}
(506) 
_{Nov}
(358) 
_{Dec}
(147) 
2013 
_{Jan}
(492) 
_{Feb}
(328) 
_{Mar}
(477) 
_{Apr}
(348) 
_{May}
(248) 
_{Jun}
(237) 
_{Jul}
(526) 
_{Aug}
(407) 
_{Sep}
(253) 
_{Oct}
(263) 
_{Nov}
(202) 
_{Dec}
(184) 
2014 
_{Jan}
(246) 
_{Feb}
(258) 
_{Mar}
(305) 
_{Apr}
(168) 
_{May}
(182) 
_{Jun}
(238) 
_{Jul}
(340) 
_{Aug}
(256) 
_{Sep}
(312) 
_{Oct}
(168) 
_{Nov}
(135) 
_{Dec}
(125) 
2015 
_{Jan}
(75) 
_{Feb}
(326) 
_{Mar}
(440) 
_{Apr}
(155) 
_{May}

_{Jun}

_{Jul}

_{Aug}

_{Sep}

_{Oct}

_{Nov}

_{Dec}

S  M  T  W  T  F  S 




1
(2) 
2
(3) 
3
(3) 
4
(3) 
5
(7) 
6
(17) 
7
(12) 
8
(6) 
9
(4) 
10
(10) 
11
(4) 
12
(3) 
13
(4) 
14
(11) 
15
(4) 
16
(18) 
17
(8) 
18
(3) 
19
(6) 
20
(13) 
21
(6) 
22
(8) 
23
(10) 
24
(3) 
25

26
(1) 
27

28
(23) 
29
(17) 
30
(14) 
31
(23) 

From: Sturla Molden <sturla.molden@gm...>  20140130 22:53:09

Sturla Molden <sturla.molden@...> wrote: > > This is actually a GNU problem. libgomp cannot be used on both sides of a > fork without an exec. Other common OpenMP implementations (Intel, > Microsoft) do not have this problem. It is interesting that Apple's GCD and > Accelerate framework have exactly the same issue as libgomp, which makes me > suspect it is related. > In Python 3.4, multiprocessing has a solution to this problem: multiprocessing.set_start_method('spawn') will cause the processes to be started using fork+exec (with similar code to the current multiprocessing on Windows). However, it cripples the capabilites of multiprocessing: All objects must be pickled for sharing with spawned processes, unike fork where everything is just copied. And the target callable must be toplevel (i.e. not a closure), because a closure cannot be pickled. OTOH, it is not worse than using multiprocessing on Windows. Sturla 
From: Faraz Mirzaei <fmmirzaei@gm...>  20140130 22:52:12

It seems that removing lines 10651067 of cross_validation.py solves the problem for now: 1065: if not isinstance(score, numbers.Number): 1066: raise ValueError("scoring must return a number, got %s (%s)" 1067 " instead." % (str(score), type(score))) Can we patch the next release to reflect this? Thanks, Faraz On Tue, Jan 28, 2014 at 2:11 PM, Joel Nothman <joel.nothman@...>wrote: > Hi Faraz, > > Not supporting multiple scores, in grid search even, is a known problem. > We haven't quite found a good solution. > > Maybe supporting arbitrary scores from `cross_val_score` is a simple > quickfix for some of that... > > > On 29 January 2014 08:59, Faraz Mirzaei <fmmirzaei@...> wrote: > >> Hello, >> >> I had used scikitlearn's cross correlation in regression test >> and performance validation for a software release. Unfortunately, >> scikitlearn was inadvertently updated to 0.14.1, breaking a large number >> of our regression and performance validation tests. >> >> I had used cross_val_score to measure "per class" recalls. I've been >> passing the following recall function to cross_val_score: >> >> def getRecalls(y_true, y_pred): >> >> p, r, f, s = precision_recall_fscore_support(y_true, y_pred) >> >> return r >> >> >> Note that r is a vector of recalls, with one element corresponding to >> each class. In the latest release (0.14.1), cross_val_score is changed to >> ensure that only a scalar is returned out of the scoring function. >> Specifically, in lines 10651067 of cross_validation.py, the following >> exception is raised for nonscalar output of scoring function: >> >> >> if not isinstance(score, numbers.Number): >> >> raise ValueError("scoring must return a number, got %s (%s)" >> >> " instead." % (str(score), type(score))) >> >> >> >> While I understand that for parameter optimization and grid search, one >> needs to use a scalar score function, I don't think cross_val_score is the >> right place to force or check that. Obviously, there are many other use >> cases for cross_val_score other than parameter optimization (namely for >> performance validation and reg test), where nonscalar metric makes sense. >> In our specific application, per class recalls are important and an average >> recall (over all classes) is not sufficient. >> >> >> Thanks, >> >> >> Faraz >> >> >> >>  >> WatchGuard Dimension instantly turns raw network data into actionable >> security intelligence. It gives you realtime visual feedback on key >> security issues and trends. Skip the complicated setup  simply import >> a virtual appliance and go from zero to informed in seconds. >> >> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk >> _______________________________________________ >> Scikitlearngeneral mailing list >> Scikitlearngeneral@... >> https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral >> >> > > >  > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you realtime visual feedback on key > security issues and trends. Skip the complicated setup  simply import > a virtual appliance and go from zero to informed in seconds. > > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk > _______________________________________________ > Scikitlearngeneral mailing list > Scikitlearngeneral@... > https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral > > 
From: Sturla Molden <sturla.molden@gm...>  20140130 22:32:17

Lars Buitinck <larsmans@...> wrote: > But anyway, the modification has not been implemented in scikitlearn > because the combination of OpenMP and Python multiprocessing is rather > problematic. This is actually a GNU problem. libgomp cannot be used on both sides of a fork without an exec. Other common OpenMP implementations (Intel, Microsoft) do not have this problem. It is interesting that Apple's GCD and Accelerate framework have exactly the same issue as libgomp, which makes me suspect it is related. Sturla 
From: Gael Varoquaux <gael.varoquaux@no...>  20140130 22:28:26

On Thu, Jan 30, 2014 at 11:23:28AM 0800, James Jensen wrote: > Bayesian optimization is an efficient method used especially for > functions that are expensive to evaluate. The basic idea is to fit the > function using Gaussian processes, using a surrogate function that > determines where to evaluate next in each iteration. The surrogate > strikes a balance between exploration (sampling intervals you haven't > tried before) and exploitation (if previous samples in a vicinity scored > well, then the likelihood of getting a high score in that area is high). > Some of the math behind it is beyond me, but the general idea is very > intuitive. Brochu, Cora, and de Freitas (2010) "A Tutorial on Bayesian > Optimization of Expensive Cost Functions," is a good introduction. > One useful application of Bayesian optimization is hyperparameter > tuning. Thanks a lot for your enthousiasme and suggestion. Indeed, many of the core developpers would love to see simple Bayesian optimization used for hyperparameter optimization, for instance taking the gist of hyperopt https://github.com/hyperopt/hyperopt and making an extended version of the RandomSearchCV. However there are a number of technical roadblocks to get there. In particular the Gaussian process could be improved (to implement partial_fit for online learning), and the parallel computing engine (joblib) does not support well as producer/consumer pattern. None of these problems are showstoppers, but they reduce the usefulness of a hyperparameter selection object using Bayesian optimization. I would hope that we find time to implement these difficult core aspects and eventually get to implementing a more advanced hyperparameter optimizer. But all the core developers are very busy and spending a lot of time simply maintaining the library (have a look at the number of issues open or pull requests that are waiting to be reviewed to have an idea). If you want to help beyond helping with reviewing/finishing pull requests and closing issues, I suggest that first, to prototype code, you could first submit an example using the Gaussian processes to do optimization of a noisy function. In a second step, after having that example merged, we could think about how to build a BayesianSearchCV object. Cheers, Gaël 
From: Sturla Molden <sturla.molden@gm...>  20140130 22:25:13

As I understand it fro reading about this a LONG time ago (apologies if my memory is rusty), "Bayesian optimization" means maximizing the loglikelihood using the NewtonRaphson method. The word "Bayesian" comes from an obfuscated explanation of what really happens: If we assume a flat or Gaussian prior and approximate the loglikelihood with a second order Taylor series expansion, the posterior is approximated with a Gaussian dustribution. We can then improve this iteratively by refitting the polynomial around the mode. But only statisticans like to explain optimization with NewtonRaphson so difficulty. There is no need to involve Gaussian approximations to the Bayesian posterior here. "Bayesian optimization" is merely a buzzword. This is no more "Bayesian" than ML using Fisher's scoring method, in fact it is identical. Any by the way, NewtonRaphson is not about striking balances between exploitation and exploration. That is also bullshitting. It is about quadratic convergence, and if anything, it is famous for finding local optima and sometimes just failing to converge by overshooting the target (which is why quasiNewton is often preferred). :) Sturla Zach Dwiel <zach@...> wrote: > It seems that with GridSearchCV and RandomizedSearchCV both already > included in scikitlearn, it would make sense to also include other > common, more efficient hyperparameter searchers as well. > > zach > > On Thu, Jan 30, 2014 at 3:11 PM, Hadayat Seddiqi > <hadsed@...> wrote: > > Hi, > > So I was the one who volunteered to do contribute my GP code for a revamp > of scikits module. I'm far from an expert, and I can't say I understand > how this would fit off the top of my head, but if someone is > knowledgeable and willing to work on this then I'd be more than happy to > lend a hand as well. I've been kind of quiet on my own GP code so far.. > just trying to get everything as ready and nice as I can before bugging people again. > > James you mentioned that you might be hesitant to suggest things if you > don't have time to implement. If I read that correctly, you're saying you > might not have the time, but in case you do, feel free to contact (this > goes for anyone, of course). > > Had > > On Thu, Jan 30, 2014 at 3:03 PM, Dan Haiduc > <danuthaiduc@...> wrote: > > Actually, I wanted to create exactly this myself. I was then discouraged > by the fact that Scikitlearn did not pull from a guy who implemented > MultiArmed Bandit > <a href="https://github.com/scikitlearn/scikitlearn/pull/906">https://github.com/scikitlearn/scikitlearn/pull/906</a>>on > the reason that Scikitlearn doesn't do reinforcement learning. I'm new > here (everywhere, not just scikit), and I'm not sure how closely related > MAB is with Bayesian optimization, but I think something along those > lines should definitely be implemented for hyperparameters, since they're > expensive functions almost by definition. > > Great idea! I certainly wish it gets implemented as well. > > On Thu, Jan 30, 2014 at 9:23 PM, James Jensen > <jdjensen@...>wrote: > > I usually hesitate to suggest a new feature in a library like this unless > I am in a position to work on it myself. However, given the number of > people who seem eager to find something to contribute, and given the > recent discussion about improving the Gaussian process module, I thought > I'd venture an idea. > > Bayesian optimization is an efficient method used especially for > functions that are expensive to evaluate. The basic idea is to fit the > function using Gaussian processes, using a surrogate function that > determines where to evaluate next in each iteration. The surrogate > strikes a balance between exploration (sampling intervals you haven't > tried before) and exploitation (if previous samples in a vicinity scored > well, then the likelihood of getting a high score in that area is high). > Some of the math behind it is beyond me, but the general idea is very > intuitive. Brochu, Cora, and de Freitas (2010) "A Tutorial on Bayesian > Optimization of Expensive Cost Functions," is a good introduction. > > One useful application of Bayesian optimization is hyperparameter tuning. > It can be used to optimize the crossvalidation score, as an alternative > to, for example, grid search. Grid search is simple and parallelizable, > there is no overhead in choosing the hyperparameters to try, and the > nature of some estimators allows them to be used with it very > efficiently. Bayesian optimization is serial and has a small amount of > overhead in evaluating the surrogate. But it is generally much more > efficient in finding good solutions, and particularly shines when the > scoring function is costly or when there are more than 1 or 2 > hyperparameters to tune; here grid search is less attractive and > sometimes completely impractical. > > In one of my own applications, involving 4 regularization parameters, > I've been using the BayesOpt library > (http://rmcantin.bitbucket.org/html/index.html), which offers it as a > generalpurpose optimization technique that one can manually integrate > with one's crossvalidation code. In general, it works quite well, but > there are some limitations to its design that can make its integration > inconvenient. Having this functionality directly integrated into > scikitlearn and specifically tailored to hyperparameter tuning would be > useful. I have been impressed with the ease of use of such convenience > classes as GridSearchCV, and dream of having a corresponding BayesOptCV, etc. > > As a generaluse optimization method, Bayesian optimization would belong > elsewhere than in scikitlearn, e.g. in scipy.optimize. But specifically > as a method for hyperparameter tuning, it seems it would fit well in the > scope of scikitlearn, especially since I expect it would not be much > more than a layer or two of functionality on top of what scikitlearn's > GP module offers (or will offer once revised). And it would be of more > general utility than an additional estimator here or there. > > I'm curious to hear what others think about the idea. Would this be a > good fit for scikitlearn? Do we have people with the interest, > expertise, and time to take this on at some point? > >  > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you realtime visual feedback on key > security issues and trends. Skip the complicated setup  simply import a > virtual appliance and go from zero to informed in seconds. > > <a > href="http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk">http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk</a>; > _______________________________________________ Scikitlearngeneral > mailing list Scikitlearngeneral@... <a > href="https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral">https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral</a>; > >  > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you realtime visual feedback on key > security issues and trends. Skip the complicated setup  simply import a > virtual appliance and go from zero to informed in seconds. > > <a > href="http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk">http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk</a>; > _______________________________________________ Scikitlearngeneral > mailing list Scikitlearngeneral@... <a > href="https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral">https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral</a>; > >  > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you realtime visual feedback on key > security issues and trends. Skip the complicated setup  simply import a > virtual appliance and go from zero to informed in seconds. > > <a > href="http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk">http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk</a>; > _______________________________________________ Scikitlearngeneral > mailing list Scikitlearngeneral@... <a > href="https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral">https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral</a>; > >  > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you realtime visual feedback on key > security issues and trends. Skip the complicated setup  simply import a > virtual appliance and go from zero to informed in seconds. <a > href="http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk">http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk</a>; 
From: Lars Buitinck <larsmans@gm...>  20140130 22:15:49

20140130 Thomas Johnson <thomas.j.johnson@...>: > The scikitlearn docs say that the SVM/SVC classes are based on libsvm. The > libsvm faq (http://www.csie.ntu.edu.tw/~cjlin/libsvm/faq.html#f432) says > that libsvm can automatically parallelize kernel evaluations using openmp if > compiled correctly. Is there any way to parallelize the SVM or SVC > implementations in scikitlearn? Actually, that FAQ explains how to *modify the source code* to get a parallel LibSVM. I don't call that automatic :p But anyway, the modification has not been implemented in scikitlearn because the combination of OpenMP and Python multiprocessing is rather problematic. 
From: James Jensen <jdjensen@en...>  20140130 21:25:27

Hi, Had, It's true that I'd have limited time (working on a PhD). I imagine most possible contributors are also quite busy. Mainly, I lack the expertise necessary to do this properly; I understand Bayesian optimization at a high level but don't have much of a foundation in the underlying math, and am an amateur programmer not yet accustomed to writing code that would meet scikitlearn standards. That being said, if there are ways I can help make this happen, I'd be glad to do so. James On 01/30/2014 12:11 PM, Hadayat Seddiqi wrote: > Hi, > > So I was the one who volunteered to do contribute my GP code for a > revamp of scikits module. I'm far from an expert, and I can't say I > understand how this would fit off the top of my head, but if someone > is knowledgeable and willing to work on this then I'd be more than > happy to lend a hand as well. I've been kind of quiet on my own GP > code so far.. just trying to get everything as ready and nice as I can > before bugging people again. > > James you mentioned that you might be hesitant to suggest things if > you don't have time to implement. If I read that correctly, you're > saying you might not have the time, but in case you do, feel free to > contact (this goes for anyone, of course). > > Had > > > > On Thu, Jan 30, 2014 at 3:03 PM, Dan Haiduc <danuthaiduc@... > <mailto:danuthaiduc@...>> wrote: > > Actually, I wanted to create exactly this myself. > I was then discouraged by the fact that Scikitlearn did not pull > from a guy who implemented MultiArmed Bandit > <https://github.com/scikitlearn/scikitlearn/pull/906>; on the > reason that Scikitlearn doesn't do reinforcement learning. > I'm new here (everywhere, not just scikit), and I'm not sure how > closely related MAB is with Bayesian optimization, but I think > something along those lines should definitely be implemented for > hyperparameters, since they're expensive functions almost by > definition. > > Great idea! I certainly wish it gets implemented as well. > > > On Thu, Jan 30, 2014 at 9:23 PM, James Jensen > <jdjensen@... <mailto:jdjensen@...>> wrote: > > I usually hesitate to suggest a new feature in a library like this > unless I am in a position to work on it myself. However, given the > number of people who seem eager to find something to > contribute, and > given the recent discussion about improving the Gaussian > process module, > I thought I'd venture an idea. > > Bayesian optimization is an efficient method used especially for > functions that are expensive to evaluate. The basic idea is to > fit the > function using Gaussian processes, using a surrogate function that > determines where to evaluate next in each iteration. The surrogate > strikes a balance between exploration (sampling intervals you > haven't > tried before) and exploitation (if previous samples in a > vicinity scored > well, then the likelihood of getting a high score in that area > is high). > Some of the math behind it is beyond me, but the general idea > is very > intuitive. Brochu, Cora, and de Freitas (2010) "A Tutorial on > Bayesian > Optimization of Expensive Cost Functions," is a good introduction. > > One useful application of Bayesian optimization is hyperparameter > tuning. It can be used to optimize the crossvalidation score, > as an > alternative to, for example, grid search. Grid search is > simple and > parallelizable, there is no overhead in choosing the > hyperparameters to > try, and the nature of some estimators allows them to be used > with it > very efficiently. Bayesian optimization is serial and has a > small amount > of overhead in evaluating the surrogate. But it is generally > much more > efficient in finding good solutions, and particularly shines > when the > scoring function is costly or when there are more than 1 or 2 > hyperparameters to tune; here grid search is less attractive and > sometimes completely impractical. > > In one of my own applications, involving 4 regularization > parameters, > I've been using the BayesOpt library > (http://rmcantin.bitbucket.org/html/index.html), which offers > it as a > generalpurpose optimization technique that one can manually > integrate > with one's crossvalidation code. In general, it works quite > well, but > there are some limitations to its design that can make its > integration > inconvenient. Having this functionality directly integrated into > scikitlearn and specifically tailored to hyperparameter > tuning would be > useful. I have been impressed with the ease of use of such > convenience > classes as GridSearchCV, and dream of having a corresponding > BayesOptCV, > etc. > > As a generaluse optimization method, Bayesian optimization > would belong > elsewhere than in scikitlearn, e.g. in scipy.optimize. But > specifically > as a method for hyperparameter tuning, it seems it would fit > well in the > scope of scikitlearn, especially since I expect it would not > be much > more than a layer or two of functionality on top of what > scikitlearn's > GP module offers (or will offer once revised). And it would be > of more > general utility than an additional estimator here or there. > > I'm curious to hear what others think about the idea. Would > this be a > good fit for scikitlearn? Do we have people with the interest, > expertise, and time to take this on at some point? > > > > > >  > WatchGuard Dimension instantly turns raw network data into > actionable > security intelligence. It gives you realtime visual feedback > on key > security issues and trends. Skip the complicated setup  > simply import > a virtual appliance and go from zero to informed in seconds. > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk > _______________________________________________ > Scikitlearngeneral mailing list > Scikitlearngeneral@... > <mailto:Scikitlearngeneral@...> > https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral > > > >  > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you realtime visual feedback on key > security issues and trends. Skip the complicated setup  simply > import > a virtual appliance and go from zero to informed in seconds. > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk > _______________________________________________ > Scikitlearngeneral mailing list > Scikitlearngeneral@... > <mailto:Scikitlearngeneral@...> > https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral > > 
From: Zach Dwiel <zach@dw...>  20140130 20:16:26

It seems that with GridSearchCV and RandomizedSearchCV both already included in scikitlearn, it would make sense to also include other common, more efficient hyperparameter searchers as well. zach On Thu, Jan 30, 2014 at 3:11 PM, Hadayat Seddiqi <hadsed@...> wrote: > Hi, > > So I was the one who volunteered to do contribute my GP code for a revamp > of scikits module. I'm far from an expert, and I can't say I understand how > this would fit off the top of my head, but if someone is knowledgeable and > willing to work on this then I'd be more than happy to lend a hand as well. > I've been kind of quiet on my own GP code so far.. just trying to get > everything as ready and nice as I can before bugging people again. > > James you mentioned that you might be hesitant to suggest things if you > don't have time to implement. If I read that correctly, you're saying you > might not have the time, but in case you do, feel free to contact (this > goes for anyone, of course). > > Had > > > > On Thu, Jan 30, 2014 at 3:03 PM, Dan Haiduc <danuthaiduc@...> wrote: > >> Actually, I wanted to create exactly this myself. >> I was then discouraged by the fact that Scikitlearn did not pull from a >> guy who implemented MultiArmed Bandit<https://github.com/scikitlearn/scikitlearn/pull/906>on the reason that Scikitlearn doesn't do reinforcement learning. >> I'm new here (everywhere, not just scikit), and I'm not sure how closely >> related MAB is with Bayesian optimization, but I think something along >> those lines should definitely be implemented for hyperparameters, since >> they're expensive functions almost by definition. >> >> Great idea! I certainly wish it gets implemented as well. >> >> >> On Thu, Jan 30, 2014 at 9:23 PM, James Jensen <jdjensen@...>wrote: >> >>> I usually hesitate to suggest a new feature in a library like this >>> unless I am in a position to work on it myself. However, given the >>> number of people who seem eager to find something to contribute, and >>> given the recent discussion about improving the Gaussian process module, >>> I thought I'd venture an idea. >>> >>> Bayesian optimization is an efficient method used especially for >>> functions that are expensive to evaluate. The basic idea is to fit the >>> function using Gaussian processes, using a surrogate function that >>> determines where to evaluate next in each iteration. The surrogate >>> strikes a balance between exploration (sampling intervals you haven't >>> tried before) and exploitation (if previous samples in a vicinity scored >>> well, then the likelihood of getting a high score in that area is high). >>> Some of the math behind it is beyond me, but the general idea is very >>> intuitive. Brochu, Cora, and de Freitas (2010) "A Tutorial on Bayesian >>> Optimization of Expensive Cost Functions," is a good introduction. >>> >>> One useful application of Bayesian optimization is hyperparameter >>> tuning. It can be used to optimize the crossvalidation score, as an >>> alternative to, for example, grid search. Grid search is simple and >>> parallelizable, there is no overhead in choosing the hyperparameters to >>> try, and the nature of some estimators allows them to be used with it >>> very efficiently. Bayesian optimization is serial and has a small amount >>> of overhead in evaluating the surrogate. But it is generally much more >>> efficient in finding good solutions, and particularly shines when the >>> scoring function is costly or when there are more than 1 or 2 >>> hyperparameters to tune; here grid search is less attractive and >>> sometimes completely impractical. >>> >>> In one of my own applications, involving 4 regularization parameters, >>> I've been using the BayesOpt library >>> (http://rmcantin.bitbucket.org/html/index.html), which offers it as a >>> generalpurpose optimization technique that one can manually integrate >>> with one's crossvalidation code. In general, it works quite well, but >>> there are some limitations to its design that can make its integration >>> inconvenient. Having this functionality directly integrated into >>> scikitlearn and specifically tailored to hyperparameter tuning would be >>> useful. I have been impressed with the ease of use of such convenience >>> classes as GridSearchCV, and dream of having a corresponding BayesOptCV, >>> etc. >>> >>> As a generaluse optimization method, Bayesian optimization would belong >>> elsewhere than in scikitlearn, e.g. in scipy.optimize. But specifically >>> as a method for hyperparameter tuning, it seems it would fit well in the >>> scope of scikitlearn, especially since I expect it would not be much >>> more than a layer or two of functionality on top of what scikitlearn's >>> GP module offers (or will offer once revised). And it would be of more >>> general utility than an additional estimator here or there. >>> >>> I'm curious to hear what others think about the idea. Would this be a >>> good fit for scikitlearn? Do we have people with the interest, >>> expertise, and time to take this on at some point? >>> >>> >>> >>> >>> >>> >>>  >>> WatchGuard Dimension instantly turns raw network data into actionable >>> security intelligence. It gives you realtime visual feedback on key >>> security issues and trends. Skip the complicated setup  simply import >>> a virtual appliance and go from zero to informed in seconds. >>> >>> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk >>> _______________________________________________ >>> Scikitlearngeneral mailing list >>> Scikitlearngeneral@... >>> https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral >>> >> >> >> >>  >> WatchGuard Dimension instantly turns raw network data into actionable >> security intelligence. It gives you realtime visual feedback on key >> security issues and trends. Skip the complicated setup  simply import >> a virtual appliance and go from zero to informed in seconds. >> >> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk >> _______________________________________________ >> Scikitlearngeneral mailing list >> Scikitlearngeneral@... >> https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral >> >> > > >  > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you realtime visual feedback on key > security issues and trends. Skip the complicated setup  simply import > a virtual appliance and go from zero to informed in seconds. > > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk > _______________________________________________ > Scikitlearngeneral mailing list > Scikitlearngeneral@... > https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral > > 
From: Hadayat Seddiqi <hadsed@gm...>  20140130 20:11:49

Hi, So I was the one who volunteered to do contribute my GP code for a revamp of scikits module. I'm far from an expert, and I can't say I understand how this would fit off the top of my head, but if someone is knowledgeable and willing to work on this then I'd be more than happy to lend a hand as well. I've been kind of quiet on my own GP code so far.. just trying to get everything as ready and nice as I can before bugging people again. James you mentioned that you might be hesitant to suggest things if you don't have time to implement. If I read that correctly, you're saying you might not have the time, but in case you do, feel free to contact (this goes for anyone, of course). Had On Thu, Jan 30, 2014 at 3:03 PM, Dan Haiduc <danuthaiduc@...> wrote: > Actually, I wanted to create exactly this myself. > I was then discouraged by the fact that Scikitlearn did not pull from a > guy who implemented MultiArmed Bandit<https://github.com/scikitlearn/scikitlearn/pull/906>on the reason that Scikitlearn doesn't do reinforcement learning. > I'm new here (everywhere, not just scikit), and I'm not sure how closely > related MAB is with Bayesian optimization, but I think something along > those lines should definitely be implemented for hyperparameters, since > they're expensive functions almost by definition. > > Great idea! I certainly wish it gets implemented as well. > > > On Thu, Jan 30, 2014 at 9:23 PM, James Jensen <jdjensen@...>wrote: > >> I usually hesitate to suggest a new feature in a library like this >> unless I am in a position to work on it myself. However, given the >> number of people who seem eager to find something to contribute, and >> given the recent discussion about improving the Gaussian process module, >> I thought I'd venture an idea. >> >> Bayesian optimization is an efficient method used especially for >> functions that are expensive to evaluate. The basic idea is to fit the >> function using Gaussian processes, using a surrogate function that >> determines where to evaluate next in each iteration. The surrogate >> strikes a balance between exploration (sampling intervals you haven't >> tried before) and exploitation (if previous samples in a vicinity scored >> well, then the likelihood of getting a high score in that area is high). >> Some of the math behind it is beyond me, but the general idea is very >> intuitive. Brochu, Cora, and de Freitas (2010) "A Tutorial on Bayesian >> Optimization of Expensive Cost Functions," is a good introduction. >> >> One useful application of Bayesian optimization is hyperparameter >> tuning. It can be used to optimize the crossvalidation score, as an >> alternative to, for example, grid search. Grid search is simple and >> parallelizable, there is no overhead in choosing the hyperparameters to >> try, and the nature of some estimators allows them to be used with it >> very efficiently. Bayesian optimization is serial and has a small amount >> of overhead in evaluating the surrogate. But it is generally much more >> efficient in finding good solutions, and particularly shines when the >> scoring function is costly or when there are more than 1 or 2 >> hyperparameters to tune; here grid search is less attractive and >> sometimes completely impractical. >> >> In one of my own applications, involving 4 regularization parameters, >> I've been using the BayesOpt library >> (http://rmcantin.bitbucket.org/html/index.html), which offers it as a >> generalpurpose optimization technique that one can manually integrate >> with one's crossvalidation code. In general, it works quite well, but >> there are some limitations to its design that can make its integration >> inconvenient. Having this functionality directly integrated into >> scikitlearn and specifically tailored to hyperparameter tuning would be >> useful. I have been impressed with the ease of use of such convenience >> classes as GridSearchCV, and dream of having a corresponding BayesOptCV, >> etc. >> >> As a generaluse optimization method, Bayesian optimization would belong >> elsewhere than in scikitlearn, e.g. in scipy.optimize. But specifically >> as a method for hyperparameter tuning, it seems it would fit well in the >> scope of scikitlearn, especially since I expect it would not be much >> more than a layer or two of functionality on top of what scikitlearn's >> GP module offers (or will offer once revised). And it would be of more >> general utility than an additional estimator here or there. >> >> I'm curious to hear what others think about the idea. Would this be a >> good fit for scikitlearn? Do we have people with the interest, >> expertise, and time to take this on at some point? >> >> >> >> >> >> >>  >> WatchGuard Dimension instantly turns raw network data into actionable >> security intelligence. It gives you realtime visual feedback on key >> security issues and trends. Skip the complicated setup  simply import >> a virtual appliance and go from zero to informed in seconds. >> >> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk >> _______________________________________________ >> Scikitlearngeneral mailing list >> Scikitlearngeneral@... >> https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral >> > > > >  > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you realtime visual feedback on key > security issues and trends. Skip the complicated setup  simply import > a virtual appliance and go from zero to informed in seconds. > > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk > _______________________________________________ > Scikitlearngeneral mailing list > Scikitlearngeneral@... > https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral > > 
From: Thomas Johnson <thomas.johnson@gm...>  20140130 20:10:21

The scikitlearn docs say that the SVM/SVC classes are based on libsvm. The libsvm faq (http://www.csie.ntu.edu.tw/~cjlin/libsvm/faq.html#f432) says that libsvm can automatically parallelize kernel evaluations using openmp if compiled correctly. Is there any way to parallelize the SVM or SVC implementations in scikitlearn? 
From: Dan Haiduc <danuthaiduc@gm...>  20140130 20:03:27

Actually, I wanted to create exactly this myself. I was then discouraged by the fact that Scikitlearn did not pull from a guy who implemented MultiArmed Bandit<https://github.com/scikitlearn/scikitlearn/pull/906>on the reason that Scikitlearn doesn't do reinforcement learning. I'm new here (everywhere, not just scikit), and I'm not sure how closely related MAB is with Bayesian optimization, but I think something along those lines should definitely be implemented for hyperparameters, since they're expensive functions almost by definition. Great idea! I certainly wish it gets implemented as well. On Thu, Jan 30, 2014 at 9:23 PM, James Jensen <jdjensen@...> wrote: > I usually hesitate to suggest a new feature in a library like this > unless I am in a position to work on it myself. However, given the > number of people who seem eager to find something to contribute, and > given the recent discussion about improving the Gaussian process module, > I thought I'd venture an idea. > > Bayesian optimization is an efficient method used especially for > functions that are expensive to evaluate. The basic idea is to fit the > function using Gaussian processes, using a surrogate function that > determines where to evaluate next in each iteration. The surrogate > strikes a balance between exploration (sampling intervals you haven't > tried before) and exploitation (if previous samples in a vicinity scored > well, then the likelihood of getting a high score in that area is high). > Some of the math behind it is beyond me, but the general idea is very > intuitive. Brochu, Cora, and de Freitas (2010) "A Tutorial on Bayesian > Optimization of Expensive Cost Functions," is a good introduction. > > One useful application of Bayesian optimization is hyperparameter > tuning. It can be used to optimize the crossvalidation score, as an > alternative to, for example, grid search. Grid search is simple and > parallelizable, there is no overhead in choosing the hyperparameters to > try, and the nature of some estimators allows them to be used with it > very efficiently. Bayesian optimization is serial and has a small amount > of overhead in evaluating the surrogate. But it is generally much more > efficient in finding good solutions, and particularly shines when the > scoring function is costly or when there are more than 1 or 2 > hyperparameters to tune; here grid search is less attractive and > sometimes completely impractical. > > In one of my own applications, involving 4 regularization parameters, > I've been using the BayesOpt library > (http://rmcantin.bitbucket.org/html/index.html), which offers it as a > generalpurpose optimization technique that one can manually integrate > with one's crossvalidation code. In general, it works quite well, but > there are some limitations to its design that can make its integration > inconvenient. Having this functionality directly integrated into > scikitlearn and specifically tailored to hyperparameter tuning would be > useful. I have been impressed with the ease of use of such convenience > classes as GridSearchCV, and dream of having a corresponding BayesOptCV, > etc. > > As a generaluse optimization method, Bayesian optimization would belong > elsewhere than in scikitlearn, e.g. in scipy.optimize. But specifically > as a method for hyperparameter tuning, it seems it would fit well in the > scope of scikitlearn, especially since I expect it would not be much > more than a layer or two of functionality on top of what scikitlearn's > GP module offers (or will offer once revised). And it would be of more > general utility than an additional estimator here or there. > > I'm curious to hear what others think about the idea. Would this be a > good fit for scikitlearn? Do we have people with the interest, > expertise, and time to take this on at some point? > > > > > > >  > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you realtime visual feedback on key > security issues and trends. Skip the complicated setup  simply import > a virtual appliance and go from zero to informed in seconds. > > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk > _______________________________________________ > Scikitlearngeneral mailing list > Scikitlearngeneral@... > https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral > 
From: James Jensen <jdjensen@en...>  20140130 19:47:03

I usually hesitate to suggest a new feature in a library like this unless I am in a position to work on it myself. However, given the number of people who seem eager to find something to contribute, and given the recent discussion about improving the Gaussian process module, I thought I'd venture an idea. Bayesian optimization is an efficient method used especially for functions that are expensive to evaluate. The basic idea is to fit the function using Gaussian processes, using a surrogate function that determines where to evaluate next in each iteration. The surrogate strikes a balance between exploration (sampling intervals you haven't tried before) and exploitation (if previous samples in a vicinity scored well, then the likelihood of getting a high score in that area is high). Some of the math behind it is beyond me, but the general idea is very intuitive. Brochu, Cora, and de Freitas (2010) "A Tutorial on Bayesian Optimization of Expensive Cost Functions," is a good introduction. One useful application of Bayesian optimization is hyperparameter tuning. It can be used to optimize the crossvalidation score, as an alternative to, for example, grid search. Grid search is simple and parallelizable, there is no overhead in choosing the hyperparameters to try, and the nature of some estimators allows them to be used with it very efficiently. Bayesian optimization is serial and has a small amount of overhead in evaluating the surrogate. But it is generally much more efficient in finding good solutions, and particularly shines when the scoring function is costly or when there are more than 1 or 2 hyperparameters to tune; here grid search is less attractive and sometimes completely impractical. In one of my own applications, involving 4 regularization parameters, I've been using the BayesOpt library (http://rmcantin.bitbucket.org/html/index.html), which offers it as a generalpurpose optimization technique that one can manually integrate with one's crossvalidation code. In general, it works quite well, but there are some limitations to its design that can make its integration inconvenient. Having this functionality directly integrated into scikitlearn and specifically tailored to hyperparameter tuning would be useful. I have been impressed with the ease of use of such convenience classes as GridSearchCV, and dream of having a corresponding BayesOptCV, etc. As a generaluse optimization method, Bayesian optimization would belong elsewhere than in scikitlearn, e.g. in scipy.optimize. But specifically as a method for hyperparameter tuning, it seems it would fit well in the scope of scikitlearn, especially since I expect it would not be much more than a layer or two of functionality on top of what scikitlearn's GP module offers (or will offer once revised). And it would be of more general utility than an additional estimator here or there. I'm curious to hear what others think about the idea. Would this be a good fit for scikitlearn? Do we have people with the interest, expertise, and time to take this on at some point? 
From: federico vaggi <vaggi.federico@gm...>  20140130 12:25:38

Most computing clusters usually run centOS, I think. Even if they were running Debian, it's very rare that all the nodes of the cluster are connected to the internet, usually you have to build the binaries yourself, which for some packages is a huge pain. On Wed, Jan 29, 2014 at 12:36 PM, Olivier Grisel <olivier.grisel@...>wrote: > Maybe some organization has a Debianbased compute cluster with popcon > installed and the sysadmin installed sklearn on all the nodes? > >  > Olivier > > >  > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you realtime visual feedback on key > security issues and trends. Skip the complicated setup  simply import > a virtual appliance and go from zero to informed in seconds. > > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk > _______________________________________________ > Scikitlearngeneral mailing list > Scikitlearngeneral@... > https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral > 
From: Manoj Kumar <manojkumarsivaraj334@gm...>  20140130 06:48:30

Hi Andy. Thanks for the response :) I'm looking into the project ideas but I'm am unable to zero in on a single idea for GSoC . My knowledge is limited to linear and clustering models, however I am willing to learn and read the literature well before GSoC and I am a pretty quick learner. It would be really nice if you or some of the other sklearn devs, suggest a couple of more ideas (maybe 2 or 3 estimators together or improving on existing estimators), that would help me write a successful GSoC proposal. Thanks again, On Thu, Jan 30, 2014 at 2:23 AM, Andy <t3kcit@...> wrote: > Hey Manoj. > I agree that the description is vague. > I think what Vlad was trying to say that refurbishing only makes sense if > it comes with longtime support by an active user. > > Basically, "refurbishing" means >  have a simple and sklearnconsistent interface >  be numerically stable, reliable and repeatable >  serve all feasible major usecases >  be easy to apply to the problems that people have in practice > > While you could certainly do the first, and probably the second given some > familiarity, > doing the last two is hard if you are not using the method actively in > your daytoday data mangling. > And even if the implementation was refurbished, but you are not around > afterwards, it is not clear who will be able > to maintain it. > > I don't think implementing coresets is a good idea, because it is mostly > helpful for cluster computing afaik. > Also, it adds more abstractions on top of a suboptimal interface and > implementation. > Additionally, I would really like to limit the number of additional > estimators before 1.0. > > If you feel up to the task of really making this a great implementation, > and also taking care of it in the long run, > please go ahead with the proposal. But I think that might be a bit much to > ask for a GSoC. > > Cheers, > Andy > > ps: only my opinion ;) > > > > On 01/18/2014 08:30 PM, Manoj Kumar wrote: > > Hello, > > I found this idea "Improving Gaussian Mixture Models" , repeating in > 2012 and 2013, so I assumed this to be of real interest to the scikitlearn > community. I have a fundamental knowledge of Gaussian Models, and the EM > algorithm. I would like to take this project forward as part of GSoC. I > took a quick look at the issues tracker, and I found a number of issues. > > I mailed Vlad (since his name was mentioned there as a mentor) and this > is what he had to say > > " > Hey Manoj, > > I just noticed I'm listed as a possible mentor for that. I think when I > put my name there I was thinking of HMM instead of GMM, oops! > > I'm guessing that the module is not really maintained and it would be good > if somebody who is involved with GMMs actively would take it under their > wing. > > I guess the point of the GSoC idea that was on there was that somebody > proposed > to do a GSoC project to implement coresets for GMM fitting (there are two > links > there). I have absolutely no experience with this method. > Of course in order to add a major new feature to a suboptimally maintained > model, > some refactoring needed to be listed as well. > > Again, my feeling is that this idea came from a potential student and it > isn't a > burning need. What do you think about it? > > Best, > Vlad > " > Can someone clearly explain, what the community expects out of such a > project, the project description ("Refurbish the current GMM code to put it > to the scikit's standards") in the wiki page, seems a bit vague to me. > > Thanks. >  > Regards, > Manoj Kumar, > Mech Undergrad > http://manojbits.wordpress.com > > >  > CenturyLink Cloud: The Leader in Enterprise Cloud Services. > Learn Why More Businesses Are Choosing CenturyLink Cloud For > Critical Workloads, Development Environments & Everything In Between. > Get a Quote or Start a Free Trial Today. http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Scikitlearngeneral mailing listScikitlearngeneral@...nethttps://lists.sourceforge.net/lists/listinfo/scikitlearngeneral > > > > >  > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you realtime visual feedback on key > security issues and trends. Skip the complicated setup  simply import > a virtual appliance and go from zero to informed in seconds. > > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk > _______________________________________________ > Scikitlearngeneral mailing list > Scikitlearngeneral@... > https://lists.sourceforge.net/lists/listinfo/scikitlearngeneral > >  Regards, Manoj Kumar, Mech Undergrad http://manojbits.wordpress.com 