Menu
▾
▴
tmva-users — Open mailing list for TMVA users
You can subscribe to this list here.
| 2006 |
Jan
|
Feb
|
Mar
(1) |
Apr
(4) |
May
(1) |
Jun
(1) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(1) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2007 |
Jan
|
Feb
|
Mar
(2) |
Apr
(10) |
May
(1) |
Jun
(13) |
Jul
(69) |
Aug
(40) |
Sep
(45) |
Oct
(21) |
Nov
(15) |
Dec
(2) |
| 2008 |
Jan
(44) |
Feb
(21) |
Mar
(28) |
Apr
(33) |
May
(35) |
Jun
(16) |
Jul
(12) |
Aug
(29) |
Sep
(12) |
Oct
(24) |
Nov
(36) |
Dec
(22) |
| 2009 |
Jan
(25) |
Feb
(19) |
Mar
(47) |
Apr
(23) |
May
(39) |
Jun
(14) |
Jul
(33) |
Aug
(12) |
Sep
(31) |
Oct
(31) |
Nov
(19) |
Dec
(13) |
| 2010 |
Jan
(7) |
Feb
(27) |
Mar
(26) |
Apr
(17) |
May
(10) |
Jun
(11) |
Jul
(17) |
Aug
(20) |
Sep
(31) |
Oct
(13) |
Nov
(19) |
Dec
(6) |
| 2011 |
Jan
(13) |
Feb
(17) |
Mar
(36) |
Apr
(19) |
May
(4) |
Jun
(14) |
Jul
(24) |
Aug
(22) |
Sep
(47) |
Oct
(35) |
Nov
(24) |
Dec
(18) |
| 2012 |
Jan
(28) |
Feb
(19) |
Mar
(23) |
Apr
(36) |
May
(27) |
Jun
(39) |
Jul
(29) |
Aug
(23) |
Sep
(17) |
Oct
(36) |
Nov
(60) |
Dec
(28) |
| 2013 |
Jan
(34) |
Feb
(23) |
Mar
(44) |
Apr
(39) |
May
(89) |
Jun
(55) |
Jul
(31) |
Aug
(47) |
Sep
(6) |
Oct
(21) |
Nov
(21) |
Dec
(10) |
| 2014 |
Jan
(19) |
Feb
(32) |
Mar
(11) |
Apr
(33) |
May
(22) |
Jun
(7) |
Jul
(16) |
Aug
(4) |
Sep
(20) |
Oct
(17) |
Nov
(12) |
Dec
(6) |
| 2015 |
Jan
(9) |
Feb
(7) |
Mar
(16) |
Apr
(5) |
May
(13) |
Jun
(27) |
Jul
(25) |
Aug
(11) |
Sep
(10) |
Oct
(7) |
Nov
(47) |
Dec
(2) |
| 2016 |
Jan
(9) |
Feb
(2) |
Mar
(4) |
Apr
(18) |
May
(2) |
Jun
(8) |
Jul
|
Aug
(27) |
Sep
(47) |
Oct
(28) |
Nov
(3) |
Dec
(9) |
| 2017 |
Jan
(11) |
Feb
(23) |
Mar
(7) |
Apr
(7) |
May
(20) |
Jun
|
Jul
(6) |
Aug
(1) |
Sep
|
Oct
(3) |
Nov
(11) |
Dec
(8) |
| 2018 |
Jan
(9) |
Feb
(8) |
Mar
(2) |
Apr
(2) |
May
(2) |
Jun
|
Jul
(2) |
Aug
(1) |
Sep
(2) |
Oct
|
Nov
|
Dec
|
| 2020 |
Jan
|
Feb
|
Mar
(2) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
|
Dec
|
| 2021 |
Jan
|
Feb
(1) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
(2) |
Nov
|
Dec
|
| 2022 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(1) |
Jul
|
Aug
(1) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2023 |
Jan
|
Feb
|
Mar
(1) |
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2024 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(1) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
|
|
From: Chris J. <jo...@he...> - 2017-05-31 11:36:30
|
Hi, Is it really necessary to have 150M background events, when you only have 18k or so signal. This is a huge mis-balance, and for sure the cause of your memory usage. I suspect you could reduce the amount of background by a number of orders of magnitude without affecting the training performance... I presume you reweight the signal and background to have an effective equal amount of each ? in which case each background event is going to have a minuscule weight with the above numbers. Chris On 17/05/17 09:57, Gabriel Emmanuel Gallardo wrote: > Hello, > >> On 17 May 2017, at 2:58 PM, Andre Georg Holzner >> <and...@ce... <mailto:and...@ce...>> wrote: >> >> Good morning, >> >> On 17/05/17 04:21, Gabriel Emmanuel Gallardo wrote: >>> Dear experts, >>> >>> I am trying to train a BDT. I see that it is using 10-20GB or more of >>> RAM on my machine. Is there any way to reduce the memory usage? >>> >> It is difficult to tell without knowing more details, such as the >> number of input variables you use and the number of samples >> and training parameters such as what the maximum depth of the trees >> you allow for and how many trees you want to build. > In one channel, where I have about 18k (710) signal events (passing > preselection) and 149M (539339) background events, > I attempted to train 600 trees, with a minimum node size of 5%, and a > depth of 5. > > In another channel, I had 18k (5136) signal and 149M (3M) background, > with the same training parameters. >> >> Assuming you run this on Linux and you are looking at this in top, >> does the 10-20 GByte refer to the VIRT or the RES/RSS >> column ? > In top, the maximum memory usage for the first channel was; > VIRT 8124m > RES 4.6g > SHR 31m > > And for the second: > VIRT 10.1g > RES 6.8g > SHR 45m > > I give a 20GB upper limit because when I try to submit a similar job to > CERN’s Condor system and request 20GB of memory, my process may > sometimes be killed to to excessive memory usage. > >> >> >> best regards, >> >> Andre >>> Thank you in advance. >>> >>> Best, >>> Gabriel >>> > > Gabriel > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > > _______________________________________________ > TMVA-users mailing list > TMV...@li... > https://lists.sourceforge.net/lists/listinfo/tmva-users > |
|
From: Wolf B. <wol...@gm...> - 2017-05-17 09:08:48
|
Hello Gabriel, in my experience BDTs don't use a lot of memory. One exception I know is using a large input tree with many unused events and/or branches. In that case it saves a lot of time + RAM if you make a skim of your tree first and then do the training, i.e. keep the cut in AddTree or PrepareTrainingAndTestTree minimal and only have the relevant branches in the tree. If that applies, you could try: a) create TEntryList with only those events you need for TMVA b) disable all unused branches c) tree->SetEntryList(EntryListFromStepA); auto newTree=tree->CopyTree(); d) use newTree for TMVA Hope this helps. Cheers, Wolf Am Mittwoch, 17. Mai 2017, 02:21:19 CEST schrieb Gabriel Emmanuel Gallardo: > Dear experts, > > I am trying to train a BDT. I see that it is using 10-20GB or more of RAM on > my machine. Is there any way to reduce the memory usage? > > Thank you in advance. > > Best, > Gabriel |
|
From: Gabriel E. G. <gab...@ce...> - 2017-05-17 08:58:03
|
Hello, On 17 May 2017, at 2:58 PM, Andre Georg Holzner <and...@ce...<mailto:and...@ce...>> wrote: Good morning, On 17/05/17 04:21, Gabriel Emmanuel Gallardo wrote: Dear experts, I am trying to train a BDT. I see that it is using 10-20GB or more of RAM on my machine. Is there any way to reduce the memory usage? It is difficult to tell without knowing more details, such as the number of input variables you use and the number of samples and training parameters such as what the maximum depth of the trees you allow for and how many trees you want to build. In one channel, where I have about 18k (710) signal events (passing preselection) and 149M (539339) background events, I attempted to train 600 trees, with a minimum node size of 5%, and a depth of 5. In another channel, I had 18k (5136) signal and 149M (3M) background, with the same training parameters. Assuming you run this on Linux and you are looking at this in top, does the 10-20 GByte refer to the VIRT or the RES/RSS column ? In top, the maximum memory usage for the first channel was; VIRT 8124m RES 4.6g SHR 31m And for the second: VIRT 10.1g RES 6.8g SHR 45m I give a 20GB upper limit because when I try to submit a similar job to CERN’s Condor system and request 20GB of memory, my process may sometimes be killed to to excessive memory usage. best regards, Andre Thank you in advance. Best, Gabriel Gabriel |
|
From: Andre H. <And...@ce...> - 2017-05-17 07:32:42
|
Good morning, On 17/05/17 04:21, Gabriel Emmanuel Gallardo wrote: > Dear experts, > > I am trying to train a BDT. I see that it is using 10-20GB or more of > RAM on my machine. Is there any way to reduce the memory usage? > It is difficult to tell without knowing more details, such as the number of input variables you use and the number of samples and training parameters such as what the maximum depth of the trees you allow for and how many trees you want to build. Assuming you run this on Linux and you are looking at this in top, does the 10-20 GByte refer to the VIRT or the RES/RSS column ? best regards, Andre > Thank you in advance. > > Best, > Gabriel > > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > _______________________________________________ > TMVA-users mailing list > TMV...@li... > https://lists.sourceforge.net/lists/listinfo/tmva-users |
|
From: Gabriel E. G. <gab...@ce...> - 2017-05-17 02:35:51
|
Dear experts, I am trying to train a BDT. I see that it is using 10-20GB or more of RAM on my machine. Is there any way to reduce the memory usage? Thank you in advance. Best, Gabriel |
|
From: Federico B. <fed...@ce...> - 2017-05-12 08:59:16
|
Dear Helge, dear all,
sorry to resurrect this topic after almost one year, but this problem persists.
I see that now the two lines are commented and the error message does not appear anymore, but the problem of unchanged Nsig and Nbkg is still there.
The plots and the S/sqrt(S+B) don’t change.
I tried to dig into the code, but I am not able to spot the root of the problem.
Should I open a Jira ticket?
Thank you
cheers,
Federico
+++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------------------------------------------------------
Federico Betti
PhD Student (LHCb group)
Università di Bologna, Dipartimento di Fisica
Istituto Nazionale di Fisica Nucleare (INFN), Sezione di Bologna
Address: via Irnerio 46, Bologna, Italy
Phone: +39 051 20 91071
email: fed...@ce...<mailto:fed...@ce...>
---------------------------------------------------------------------------------------
+++++++++++++++++++++++++++++++++++++++++++++++++++
On 13 Jun 2016, at 14:27, Federico Betti <fed...@ce...<mailto:fed...@ce...>> wrote:
Hi Helge,
I’m glad to have been helpful to spot this issue!
cheers,
Federico
On 13 Jun 2016, at 14:06, Helge Voss <Hel...@ce...<mailto:Hel...@ce...>> wrote:
Hi Ferderico,
thanks for spotting this! It's yet another necessary update when
moving to ROOT6 that I didn't catch myself. I fixed it now
in the "HEAD" of the root repository .... I'll ask some of the "ROOT
guys" to perhaps also make the fix in the oder branches
(still not being a 'git-expert" :( I don't seem to manage)
if you have a 'local source' for ROOT/TMVA, you can fix it easily by
'commenting' (or deleting) the two lines in
$ROOTSYS/tmva/tmvagui/src/mvaeffs.cxx
// fDrawButton->Connect("Clicked()","TGNumberEntry",fSigInput,
"ValueSet(Long_t)");
// fDrawButton->Connect("Clicked()","TGNumberEntry",fBkgInput,
"ValueSet(Long_t)");
Cheers,
Helge
On 13 June 2016 at 12:41, Federico Betti <fed...@ce...<mailto:fed...@ce...>> wrote:
Dear TMVA experts,
I recently switched to ROOT 6 version, and I did my first ROOT6-BDT training. Everything seems fine, except when I run the TMVAGui (via TMVA::TMVAGui(“myBDT.root”)) and I plot the cut efficiencies. The plot is ok, but when I try to change the number of signal and background events (in the little panel that comes with this plot) and click draw, the plot doesn’t change and I get the following error:
Error in <TClingCallFunc::exec>: Not enough arguments provided for ValueSet (0 instead of the minimum 1)
Error in <TClingCallFunc::exec>: Not enough arguments provided for ValueSet (0 instead of the minimum 1)
I tried to look for this error in the web, but I am not able to find something similar. What could be the reason?
Thanks in advance for your answer
cheers,
Federico
------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
TMVA-users mailing list
TMV...@li...<mailto:TMV...@li...>
https://lists.sourceforge.net/lists/listinfo/tmva-users
------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
TMVA-users mailing list
TMV...@li...<mailto:TMV...@li...>
https://lists.sourceforge.net/lists/listinfo/tmva-users
|
|
From: Federico B. <fed...@ce...> - 2017-05-10 16:54:14
|
Hi again, Ups, I see now that this bug has been already reported here https://sft.its.cern.ch/jira/browse/ROOT-8517?jql=text%20~%20%22evaluateallmethods%22<https://sft.its.cern.ch/jira/browse/ROOT-8517?jql=text%20~%20"evaluateallmethods">. I hope it will be solved soon :) Thank you and sorry for the noise cheers, Federico +++++++++++++++++++++++++++++++++++++++++++++++++++ --------------------------------------------------------------------------------------- Federico Betti PhD Student (LHCb group) Università di Bologna, Dipartimento di Fisica Istituto Nazionale di Fisica Nucleare (INFN), Sezione di Bologna Address: via Irnerio 46, Bologna, Italy Phone: +39 051 20 91071 email: fed...@ce...<mailto:fed...@ce...> --------------------------------------------------------------------------------------- +++++++++++++++++++++++++++++++++++++++++++++++++++ On 10 May 2017, at 18:18, Federico Betti <fed...@ce...<mailto:fed...@ce...>> wrote: Hi, I must correct myself, the error appears also without VarTransform, for example if I book two methods: factory->BookMethod( dataloader, TMVA::Types::kBDT, "BDT", "!H:!V:NTrees=800:MaxDepth=2:BoostType=AdaBoost:SeparationType=GiniIndex:nCuts=20:PruneMethod=NoPruning:DoPreselection=False”); factory->BookMethod( dataloader, TMVA::Types::kBDT, "BDTG", "!H:!V:NTrees=800:MinNodeSize=2.5%:BoostType=Grad:Shrinkage=0.10:UseBaggedBoost:BaggedSampleFraction=0.5:nCuts=20:MaxDepth=2" ); cheers, Federico +++++++++++++++++++++++++++++++++++++++++++++++++++ --------------------------------------------------------------------------------------- Federico Betti PhD Student (LHCb group) Università di Bologna, Dipartimento di Fisica Istituto Nazionale di Fisica Nucleare (INFN), Sezione di Bologna Address: via Irnerio 46, Bologna, Italy Phone: +39 051 20 91071 email: fed...@ce...<mailto:fed...@ce...> --------------------------------------------------------------------------------------- +++++++++++++++++++++++++++++++++++++++++++++++++++ On 10 May 2017, at 17:43, Federico Betti <fed...@ce...<mailto:fed...@ce...>> wrote: Dear TMVA experts, I am trying to train a BDT using event-by-event weights and Gaussian+Decorrelation transformation of input variables. The problem is that I get a seg fault during the execution of EvaluateAllMethods(), in particular during the ROCIntegral calculation. I attach both the macro and the log file. The error appears here: : Evaluation results ranked by best signal efficiency and purity (area) : ------------------------------------------------------------------------------------------------------------------- : DataSet MVA : Name: Method: ROC-integ *** Break *** segmentation violation =========================================================== There was a crash. This is the entire stack trace of all threads: =========================================================== #0 0x00000030bc2ac7be in waitpid () from /lib64/libc.so.6 #1 0x00000030bc23e5c9 in do_system () from /lib64/libc.so.6 #2 0x00007f1e7a26f5ba in TUnixSystem::StackTrace() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libCore.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libCore.so> #3 0x00007f1e7a271abc in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libCore.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libCore.so> #4 <signal handler called> #5 0x00007f1e77086330 in TMVA::ROCCurve::GetROCIntegral() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so> #6 0x00007f1e76f5ce25 in TMVA::Factory::EvaluateAllMethods() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so> #7 0x00000000004047ff in test_Class () at src/test_Class.cpp:130 #8 0x00000000004038c1 in main () at src/test_Class.cpp:26 =========================================================== The lines below might hint at the cause of the crash. You may get help by asking at the ROOT forum http://root.cern.ch/forum. Only if you are really convinced it is a bug in ROOT then please submit a report at http://root.cern.ch/bugs. Please post the ENTIRE stack trace from above as an attachment in addition to anything else that might help us fixing this issue. =========================================================== #5 0x00007f1e77086330 in TMVA::ROCCurve::GetROCIntegral() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so> #6 0x00007f1e76f5ce25 in TMVA::Factory::EvaluateAllMethods() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so> #7 0x00000000004047ff in test_Class () at src/test_Class.cpp:130 #8 0x00000000004038c1 in main () at src/test_Class.cpp:26 =========================================================== I guess the problem is due to the presence of negative weights. I can not skip events with negative weights, since their presence make the input variables distributions quite discriminating between signal and background. The problem disappears either when I comment: dataloader->SetSignalWeightExpression("N_sig_sw"); line or when I remove var tranformation in: factory->BookMethod( dataloader, TMVA::Types::kBDT, "BDT_transfVarAll", "!H:!V:NTrees=100:MaxDepth=2:BoostType=AdaBoost:SeparationType=GiniIndex:nCuts=20:PruneMethod=NoPruning:DoPreselection=False:VarTransform=G,D"); But I don’t understand whether this is a bug or I am just not allowed to use EvaluateAllMethods when I have both negative weights and VarTransform. I could also skip EvaluateAllMethods, but then I will not be able to compare different methods at the same time. Thanks for your help cheers, Federico +++++++++++++++++++++++++++++++++++++++++++++++++++ --------------------------------------------------------------------------------------- Federico Betti PhD Student (LHCb group) Università di Bologna, Dipartimento di Fisica Istituto Nazionale di Fisica Nucleare (INFN), Sezione di Bologna Address: via Irnerio 46, Bologna, Italy Phone: +39 051 20 91071 email: fed...@ce...<mailto:fed...@ce...> --------------------------------------------------------------------------------------- +++++++++++++++++++++++++++++++++++++++++++++++++++ <log_test_Class><test_Class.cpp>------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org<http://slashdot.org/>! http://sdm.link/slashdot_______________________________________________ TMVA-users mailing list TMV...@li...<mailto:TMV...@li...> https://lists.sourceforge.net/lists/listinfo/tmva-users |
|
From: Federico B. <fed...@ce...> - 2017-05-10 16:51:58
|
Hi Peter,
thanks a lot for your answer.
Actually, my point was that this problem arises because the cut used for signal seems to be applied to background.
But I found out that, during the PrepareTrainingAndTestTree, for some reasons both mycuts and mycutb are applied both to signal and background (don’t know why), but then the right cuts are correctly applied to the samples.
So the problem seems not existing.
cheers,
Federico
+++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------------------------------------------------------
Federico Betti
PhD Student (LHCb group)
Università di Bologna, Dipartimento di Fisica
Istituto Nazionale di Fisica Nucleare (INFN), Sezione di Bologna
Address: via Irnerio 46, Bologna, Italy
Phone: +39 051 20 91071
email: fed...@ce...<mailto:fed...@ce...>
---------------------------------------------------------------------------------------
+++++++++++++++++++++++++++++++++++++++++++++++++++
On 10 May 2017, at 18:23, Peter Speckmayer <pet...@gm...<mailto:pet...@gm...>> wrote:
Hi,
I don't have the code ready to test, but you could try to use 'Alt$(N_sig_sw, true)' instead of just N_sig_sw. This would check if there is the variable existing and if not use the alternative version which then would be true.
Cheers,
Peter
Federico Betti <fed...@ce...<mailto:fed...@ce...>> schrieb am Di., 9. Mai 2017, 10:44:
Dear experts,
I have a critical problem with the method PrepareTrainingAndTestTree of the DataLoader.
I am using ROOT 6.08.06.
I attach here the minimal code and a small copy of the ntuples to reproduce the problem.
Basically I define two different cuts for signal and background, and the cut of the signal involves one variable (N_sig_sw) which is present in signal ntuple (pipi_Down_2016_sig_sPlot_small.root) but not in background ntuple (pipi_Down_2016_sig_small.root).
TCut global_cut = "D0_LoKi_DTF_CHI2NDOF>0 &&D0_LoKi_DTF_CHI2NDOF<6.5 &&B_LoKi_DTF_mu_PT>0”;
TCut mycuts = global_cut &&"D0_LoKi_DTF_M>1840 &&D0_LoKi_DTF_M<1890 &&N_sig_sw>0.";
TCut mycutb = global_cut &&"D0_LoKi_DTF_M>1900";
Then, later in the code, I use:
dataloader->PrepareTrainingAndTestTree( mycuts, mycutb,
"NormMode=NumEvents:VerboseLevel=Debug:nTrain_Signal=0:nTrain_Background=0" );
Unfortunately I get this error:
<DEBUG> : Splitmode is: "RANDOM" the mixmode is: "SAMEASSPLITMODE"
<DEBUG> : Weight expression for class 'Signal': ""
<DEBUG> : Building event vectors 2
<DEBUG> : Dataset[dataset] : transform input variables
<DEBUG> : Dataset[dataset] : transform regression targets
<DEBUG> : Dataset[dataset] : transform spectator variables
<DEBUG> : Dataset[dataset] : transform cuts
<DEBUG> : Dataset[dataset] : transform weights
<DEBUG> : Dataset[dataset] : enable branches
<DEBUG> : Dataset[dataset] : enable branches: input variables
<DEBUG> : Dataset[dataset] : enable branches: targets
<DEBUG> : Dataset[dataset] : enable branches: spectators
<DEBUG> : Dataset[dataset] : enable branches: cuts
<DEBUG> : Dataset[dataset] : enable branches: weights
<DEBUG> : Dataset[dataset] : tree initialized
<DEBUG> : Weight expression for class 'Background': ""
<DEBUG> : Building event vectors 2
<DEBUG> : Dataset[dataset] : transform input variables
<DEBUG> : Dataset[dataset] : transform regression targets
<DEBUG> : Dataset[dataset] : transform spectator variables
<DEBUG> : Dataset[dataset] : transform cuts
Error in <TTreeFormula::Compile>: Bad numerical expression : "N_sig_sw"
<FATAL> : Expression (D0_LoKi_DTF_CHI2NDOF>0 &&D0_LoKi_DTF_CHI2NDOF<6.5 &&B_LoKi_DTF_mu_PT>0)&&(D0_LoKi_DTF_M>1840 &&D0_LoKi_DTF_M<1890 &&N_sig_sw>0.) could not be resolved to a valid formula.
So it seems that the cut applied to background ntuple is mycuts and not mycutb.
I tried to swap mycuts and mycutb in PrepareTrainingAndTestTree (just in case it’s a documentation issue of the constructor), but the problem persists.
May I ask for your help?
It sounds like a bug, but it is quite likely I am doing something stupid.
Thanks a lot
cheers,
Federico
+++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------------------------------------------------------
Federico Betti
PhD Student (LHCb group)
Università di Bologna, Dipartimento di Fisica
Istituto Nazionale di Fisica Nucleare (INFN), Sezione di Bologna
Address: via Irnerio 46, Bologna, Italy
Phone: +39 051 20 91071
email: fed...@ce...<mailto:fed...@ce...>
---------------------------------------------------------------------------------------
+++++++++++++++++++++++++++++++++++++++++++++++++++
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org<http://Slashdot.org>! http://sdm.link/slashdot_______________________________________________
TMVA-users mailing list
TMV...@li...<mailto:TMV...@li...>
https://lists.sourceforge.net/lists/listinfo/tmva-users
|
|
From: Federico B. <fed...@ce...> - 2017-05-10 16:51:08
|
Hi,
I must correct myself, the error appears also without VarTransform, for example if I book two methods:
factory->BookMethod( dataloader, TMVA::Types::kBDT, "BDT",
"!H:!V:NTrees=800:MaxDepth=2:BoostType=AdaBoost:SeparationType=GiniIndex:nCuts=20:PruneMethod=NoPruning:DoPreselection=False”);
factory->BookMethod( dataloader, TMVA::Types::kBDT, "BDTG",
"!H:!V:NTrees=800:MinNodeSize=2.5%:BoostType=Grad:Shrinkage=0.10:UseBaggedBoost:BaggedSampleFraction=0.5:nCuts=20:MaxDepth=2" );
cheers,
Federico
+++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------------------------------------------------------
Federico Betti
PhD Student (LHCb group)
Università di Bologna, Dipartimento di Fisica
Istituto Nazionale di Fisica Nucleare (INFN), Sezione di Bologna
Address: via Irnerio 46, Bologna, Italy
Phone: +39 051 20 91071
email: fed...@ce...<mailto:fed...@ce...>
---------------------------------------------------------------------------------------
+++++++++++++++++++++++++++++++++++++++++++++++++++
On 10 May 2017, at 17:43, Federico Betti <fed...@ce...<mailto:fed...@ce...>> wrote:
Dear TMVA experts,
I am trying to train a BDT using event-by-event weights and Gaussian+Decorrelation transformation of input variables.
The problem is that I get a seg fault during the execution of EvaluateAllMethods(), in particular during the ROCIntegral calculation.
I attach both the macro and the log file. The error appears here:
: Evaluation results ranked by best signal efficiency and purity (area)
: -------------------------------------------------------------------------------------------------------------------
: DataSet MVA
: Name: Method: ROC-integ
*** Break *** segmentation violation
===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0 0x00000030bc2ac7be in waitpid () from /lib64/libc.so.6
#1 0x00000030bc23e5c9 in do_system () from /lib64/libc.so.6
#2 0x00007f1e7a26f5ba in TUnixSystem::StackTrace() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libCore.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libCore.so>
#3 0x00007f1e7a271abc in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libCore.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libCore.so>
#4 <signal handler called>
#5 0x00007f1e77086330 in TMVA::ROCCurve::GetROCIntegral() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so>
#6 0x00007f1e76f5ce25 in TMVA::Factory::EvaluateAllMethods() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so>
#7 0x00000000004047ff in test_Class () at src/test_Class.cpp:130
#8 0x00000000004038c1 in main () at src/test_Class.cpp:26
===========================================================
The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum.
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5 0x00007f1e77086330 in TMVA::ROCCurve::GetROCIntegral() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so>
#6 0x00007f1e76f5ce25 in TMVA::Factory::EvaluateAllMethods() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so>
#7 0x00000000004047ff in test_Class () at src/test_Class.cpp:130
#8 0x00000000004038c1 in main () at src/test_Class.cpp:26
===========================================================
I guess the problem is due to the presence of negative weights.
I can not skip events with negative weights, since their presence make the input variables distributions quite discriminating between signal and background.
The problem disappears either when I comment:
dataloader->SetSignalWeightExpression("N_sig_sw");
line or when I remove var tranformation in:
factory->BookMethod( dataloader, TMVA::Types::kBDT, "BDT_transfVarAll",
"!H:!V:NTrees=100:MaxDepth=2:BoostType=AdaBoost:SeparationType=GiniIndex:nCuts=20:PruneMethod=NoPruning:DoPreselection=False:VarTransform=G,D");
But I don’t understand whether this is a bug or I am just not allowed to use EvaluateAllMethods when I have both negative weights and VarTransform.
I could also skip EvaluateAllMethods, but then I will not be able to compare different methods at the same time.
Thanks for your help
cheers,
Federico
+++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------------------------------------------------------
Federico Betti
PhD Student (LHCb group)
Università di Bologna, Dipartimento di Fisica
Istituto Nazionale di Fisica Nucleare (INFN), Sezione di Bologna
Address: via Irnerio 46, Bologna, Italy
Phone: +39 051 20 91071
email: fed...@ce...<mailto:fed...@ce...>
---------------------------------------------------------------------------------------
+++++++++++++++++++++++++++++++++++++++++++++++++++
<log_test_Class><test_Class.cpp>------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org<http://Slashdot.org>! http://sdm.link/slashdot_______________________________________________
TMVA-users mailing list
TMV...@li...<mailto:TMV...@li...>
https://lists.sourceforge.net/lists/listinfo/tmva-users
|
|
From: Peter S. <pet...@gm...> - 2017-05-10 16:23:38
|
Hi, I don't have the code ready to test, but you could try to use 'Alt$(N_sig_sw, true)' instead of just N_sig_sw. This would check if there is the variable existing and if not use the alternative version which then would be true. Cheers, Peter Federico Betti <fed...@ce...> schrieb am Di., 9. Mai 2017, 10:44: > Dear experts, > > I have a critical problem with the method PrepareTrainingAndTestTree of > the DataLoader. > I am using ROOT 6.08.06. > I attach here the minimal code and a small copy of the ntuples to > reproduce the problem. > > Basically I define two different cuts for signal and background, and the > cut of the signal involves one variable (N_sig_sw) which is present in > signal ntuple (pipi_Down_2016_sig_sPlot_small.root) but not in background > ntuple (pipi_Down_2016_sig_small.root). > > TCut global_cut = "D0_LoKi_DTF_CHI2NDOF>0 &&D0_LoKi_DTF_CHI2NDOF<6.5 > &&B_LoKi_DTF_mu_PT>0”; > TCut mycuts = global_cut &&"D0_LoKi_DTF_M>1840 &&D0_LoKi_DTF_M<1890 > &&N_sig_sw>0."; > TCut mycutb = global_cut &&"D0_LoKi_DTF_M>1900"; > > Then, later in the code, I use: > dataloader->PrepareTrainingAndTestTree( mycuts, mycutb, > > "NormMode=NumEvents:VerboseLevel=Debug:nTrain_Signal=0:nTrain_Background=0" > ); > > Unfortunately I get this error: > <DEBUG> : Splitmode is: "RANDOM" the mixmode is: > "SAMEASSPLITMODE" > <DEBUG> : Weight expression for class 'Signal': "" > <DEBUG> : Building event vectors 2 > <DEBUG> : Dataset[dataset] : transform input > variables > <DEBUG> : Dataset[dataset] : transform regression > targets > <DEBUG> : Dataset[dataset] : transform spectator > variables > <DEBUG> : Dataset[dataset] : transform cuts > <DEBUG> : Dataset[dataset] : transform weights > <DEBUG> : Dataset[dataset] : enable branches > <DEBUG> : Dataset[dataset] : enable branches: > input variables > <DEBUG> : Dataset[dataset] : enable branches: > targets > <DEBUG> : Dataset[dataset] : enable branches: > spectators > <DEBUG> : Dataset[dataset] : enable branches: cuts > <DEBUG> : Dataset[dataset] : enable branches: > weights > <DEBUG> : Dataset[dataset] : tree initialized > <DEBUG> : Weight expression for class > 'Background': "" > <DEBUG> : Building event vectors 2 > <DEBUG> : Dataset[dataset] : transform input > variables > <DEBUG> : Dataset[dataset] : transform regression > targets > <DEBUG> : Dataset[dataset] : transform spectator > variables > <DEBUG> : Dataset[dataset] : transform cuts > Error in <TTreeFormula::Compile>: Bad numerical expression : "N_sig_sw" > <FATAL> : Expression (D0_LoKi_DTF_CHI2NDOF>0 > &&D0_LoKi_DTF_CHI2NDOF<6.5 &&B_LoKi_DTF_mu_PT>0)&&(D0_LoKi_DTF_M>1840 > &&D0_LoKi_DTF_M<1890 &&N_sig_sw>0.) could not be resolved to a valid > formula. > > So it seems that the cut applied to background ntuple is mycuts and not > mycutb. > I tried to swap mycuts and mycutb in PrepareTrainingAndTestTree (just in > case it’s a documentation issue of the constructor), but the problem > persists. > > May I ask for your help? > It sounds like a bug, but it is quite likely I am doing something stupid. > > Thanks a lot > > cheers, > Federico > > > > +++++++++++++++++++++++++++++++++++++++++++++++++++ > > --------------------------------------------------------------------------------------- > > * Federico Betti * PhD Student (LHCb group) > Università di Bologna, Dipartimento di Fisica > Istituto Nazionale di Fisica Nucleare (INFN), Sezione di Bologna > Address: via Irnerio 46, Bologna, Italy > Phone: +39 051 20 91071 > email: fed...@ce... > > --------------------------------------------------------------------------------------- > +++++++++++++++++++++++++++++++++++++++++++++++++++ > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > TMVA-users mailing list > TMV...@li... > https://lists.sourceforge.net/lists/listinfo/tmva-users > |
|
From: Federico B. <fed...@ce...> - 2017-05-10 15:58:47
|
Dear TMVA experts,
I am trying to train a BDT using event-by-event weights and Gaussian+Decorrelation transformation of input variables.
The problem is that I get a seg fault during the execution of EvaluateAllMethods(), in particular during the ROCIntegral calculation.
I attach both the macro and the log file. The error appears here:
: Evaluation results ranked by best signal efficiency and purity (area)
: -------------------------------------------------------------------------------------------------------------------
: DataSet MVA
: Name: Method: ROC-integ
*** Break *** segmentation violation
===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0 0x00000030bc2ac7be in waitpid () from /lib64/libc.so.6
#1 0x00000030bc23e5c9 in do_system () from /lib64/libc.so.6
#2 0x00007f1e7a26f5ba in TUnixSystem::StackTrace() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libCore.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libCore.so>
#3 0x00007f1e7a271abc in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libCore.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libCore.so>
#4 <signal handler called>
#5 0x00007f1e77086330 in TMVA::ROCCurve::GetROCIntegral() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so>
#6 0x00007f1e76f5ce25 in TMVA::Factory::EvaluateAllMethods() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so>
#7 0x00000000004047ff in test_Class () at src/test_Class.cpp:130
#8 0x00000000004038c1 in main () at src/test_Class.cpp:26
===========================================================
The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum.
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5 0x00007f1e77086330 in TMVA::ROCCurve::GetROCIntegral() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so>
#6 0x00007f1e76f5ce25 in TMVA::Factory::EvaluateAllMethods() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so<http://lhcb.cern.ch/lib/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc49-opt/lib/libTMVA.so>
#7 0x00000000004047ff in test_Class () at src/test_Class.cpp:130
#8 0x00000000004038c1 in main () at src/test_Class.cpp:26
===========================================================
I guess the problem is due to the presence of negative weights.
I can not skip events with negative weights, since their presence make the input variables distributions quite discriminating between signal and background.
The problem disappears either when I comment:
dataloader->SetSignalWeightExpression("N_sig_sw");
line or when I remove var tranformation in:
factory->BookMethod( dataloader, TMVA::Types::kBDT, "BDT_transfVarAll",
"!H:!V:NTrees=100:MaxDepth=2:BoostType=AdaBoost:SeparationType=GiniIndex:nCuts=20:PruneMethod=NoPruning:DoPreselection=False:VarTransform=G,D");
But I don’t understand whether this is a bug or I am just not allowed to use EvaluateAllMethods when I have both negative weights and VarTransform.
I could also skip EvaluateAllMethods, but then I will not be able to compare different methods at the same time.
Thanks for your help
cheers,
Federico
+++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------------------------------------------------------
Federico Betti
PhD Student (LHCb group)
Università di Bologna, Dipartimento di Fisica
Istituto Nazionale di Fisica Nucleare (INFN), Sezione di Bologna
Address: via Irnerio 46, Bologna, Italy
Phone: +39 051 20 91071
email: fed...@ce...<mailto:fed...@ce...>
---------------------------------------------------------------------------------------
+++++++++++++++++++++++++++++++++++++++++++++++++++
|
|
From: Federico B. <fed...@ce...> - 2017-05-02 23:14:07
|
Dear experts,
I have a critical problem with the method PrepareTrainingAndTestTree of the DataLoader.
I am using ROOT 6.08.06.
I attach here the minimal code and a small copy of the ntuples to reproduce the problem.
Basically I define two different cuts for signal and background, and the cut of the signal involves one variable (N_sig_sw) which is present in signal ntuple (pipi_Down_2016_sig_sPlot_small.root) but not in background ntuple (pipi_Down_2016_sig_small.root).
TCut global_cut = "D0_LoKi_DTF_CHI2NDOF>0 &&D0_LoKi_DTF_CHI2NDOF<6.5 &&B_LoKi_DTF_mu_PT>0”;
TCut mycuts = global_cut &&"D0_LoKi_DTF_M>1840 &&D0_LoKi_DTF_M<1890 &&N_sig_sw>0.";
TCut mycutb = global_cut &&"D0_LoKi_DTF_M>1900";
Then, later in the code, I use:
dataloader->PrepareTrainingAndTestTree( mycuts, mycutb,
"NormMode=NumEvents:VerboseLevel=Debug:nTrain_Signal=0:nTrain_Background=0" );
Unfortunately I get this error:
<DEBUG> : Splitmode is: "RANDOM" the mixmode is: "SAMEASSPLITMODE"
<DEBUG> : Weight expression for class 'Signal': ""
<DEBUG> : Building event vectors 2
<DEBUG> : Dataset[dataset] : transform input variables
<DEBUG> : Dataset[dataset] : transform regression targets
<DEBUG> : Dataset[dataset] : transform spectator variables
<DEBUG> : Dataset[dataset] : transform cuts
<DEBUG> : Dataset[dataset] : transform weights
<DEBUG> : Dataset[dataset] : enable branches
<DEBUG> : Dataset[dataset] : enable branches: input variables
<DEBUG> : Dataset[dataset] : enable branches: targets
<DEBUG> : Dataset[dataset] : enable branches: spectators
<DEBUG> : Dataset[dataset] : enable branches: cuts
<DEBUG> : Dataset[dataset] : enable branches: weights
<DEBUG> : Dataset[dataset] : tree initialized
<DEBUG> : Weight expression for class 'Background': ""
<DEBUG> : Building event vectors 2
<DEBUG> : Dataset[dataset] : transform input variables
<DEBUG> : Dataset[dataset] : transform regression targets
<DEBUG> : Dataset[dataset] : transform spectator variables
<DEBUG> : Dataset[dataset] : transform cuts
Error in <TTreeFormula::Compile>: Bad numerical expression : "N_sig_sw"
<FATAL> : Expression (D0_LoKi_DTF_CHI2NDOF>0 &&D0_LoKi_DTF_CHI2NDOF<6.5 &&B_LoKi_DTF_mu_PT>0)&&(D0_LoKi_DTF_M>1840 &&D0_LoKi_DTF_M<1890 &&N_sig_sw>0.) could not be resolved to a valid formula.
So it seems that the cut applied to background ntuple is mycuts and not mycutb.
I tried to swap mycuts and mycutb in PrepareTrainingAndTestTree (just in case it’s a documentation issue of the constructor), but the problem persists.
May I ask for your help?
It sounds like a bug, but it is quite likely I am doing something stupid.
Thanks a lot
cheers,
Federico
+++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------------------------------------------------------
Federico Betti
PhD Student (LHCb group)
Università di Bologna, Dipartimento di Fisica
Istituto Nazionale di Fisica Nucleare (INFN), Sezione di Bologna
Address: via Irnerio 46, Bologna, Italy
Phone: +39 051 20 91071
email: fed...@ce...<mailto:fed...@ce...>
---------------------------------------------------------------------------------------
+++++++++++++++++++++++++++++++++++++++++++++++++++
|
|
From: Vojtech P. <voj...@ce...> - 2017-05-02 22:38:04
|
Dear all,
I have a weird problem. I train my BDT with five variables twice -
on two samples that are defined by the event number (even or odd). Both
training and testing is OK if I use the even events. However, if I use
the odd events, the training finishes still but the testing emits the
error below. I checked that there is no event with std::isnan(el2_pt),
el2_pt != el2_pt or el2_pt < 0. I also printed out a couple of tens of
events and el2_pt values seem quite OK.
What can be wrong?
Thank you for your help in advance. Just in case, I attach the text
output of the trainings&testings job for the odd events.
Best regards
Vojtech
P.S. The setup I use is the following:
ROOT version, architecture, OS, compiler: 6.04.16-x86_64-slc6-gcc49
BDT set up in the following way:
factory = TMVA.Factory(nameRoot, outf,
"!V:!Silent:Color:DrawProgressBar:Transformations=I:AnalysisType=Classification")
factory.PrepareTrainingAndTestTree(TCut(""),
"nTrain_Signal=0:nTrain_Background=0:nTest_Signal=0:nTest_Background=0:SplitMode=Random:V\
")
factory.BookMethod(TMVA.Types.kBDT, "BDT",
"!H:!V:NTrees=850:MinNodeSize=5%:MaxDepth=2:BoostType=AdaBoost:AdaBoostBeta=0.5:UseBaggedBoost:BaggedSampleFraction\
=0.5:SeparationType=GiniIndex:nCuts=20:NegWeightTreatment=IgnoreNegWeightsInTraining")
--- Factory : Ranking input variables (method specific)...
--- BDT : Ranking result (top variable is best ranked)
--- BDT : ------------------------------------------
--- BDT : Rank : Variable : Variable Importance
--- BDT : ------------------------------------------
--- BDT : 1 : m_LQ_min : 2.384e-01
--- BDT : 2 : m_LQ_max : 2.209e-01
--- BDT : 3 : el2_pt : 1.925e-01
--- BDT : 4 : jet2_pt : 1.835e-01
--- BDT : 5 : m_ll : 1.648e-01
--- BDT : ------------------------------------------
--- Factory :
--- Factory : === Destroy and recreate all methods via
weight files for testing ===
--- Factory :
--- MethodBase : Reading weight file:
weights/BDT_training_20170502/TMVA_isEE_SR_LQ900_1_20170502_BDT.weights.xml
--- BDT : Read method "BDT" of type "BDT"
--- BDT : MVA method was trained with TMVA Version:
4.2.1
--- BDT : MVA method was trained with ROOT Version:
6.04/16
--- Factory : Test all methods...
--- Factory : Test method: BDT for Classification
performance
--- BDT : Evaluation of BDT on testing sample
(320923 events)
--- BDT : Elapsed time for evaluation of 320923
events: 53.2 sec
--- Factory : Evaluate all methods...
--- Factory : Evaluate classifier: BDT
--- BDT : Loop over test events and fill histograms
with classifier response...
--- Factory : Write evaluation histograms to file
--- TFHandler_BDT : Plot event variables for BDT
--- TFHandler_BDT :
-----------------------------------------------------------
--- TFHandler_BDT : Variable Mean RMS [
Min Max ]
--- TFHandler_BDT :
-----------------------------------------------------------
--- TFHandler_BDT : m_LQ_min: 238.76 150.45 [
32.296 2749.5 ]
--- TFHandler_BDT : m_ll: 334.55 111.59 [
130.00 4714.5 ]
--- TFHandler_BDT : jet2_pt: 124.72 83.853 [
60.000 2455.4 ]
--- TFHandler_BDT : el2_pt: 111.48 -nan [
80.000 1893.1 ]
--- TFHandler_BDT : m_LQ_max: 392.64 266.74 [
42.108 4034.3 ]
--- TFHandler_BDT :
-----------------------------------------------------------
--- Tools : -nan -nan -nan -nan -nan -nan
--- <FATAL> Tools : <GetSeparation> signal and
background histograms have different or invalid dimensions:
***> abort program execution
|
|
From: Vojtech P. <voj...@ce...> - 2017-05-02 21:45:13
|
P.S. Just adding some info on the setup I use:
ROOT version, architecture, OS, compiler: 6.04.16-x86_64-slc6-gcc49
BDT setup in the following way:
factory = TMVA.Factory(nameRoot, outf,
"!V:!Silent:Color:DrawProgressBar:Transformations=I:AnalysisType=Classification")
factory.PrepareTrainingAndTestTree(TCut(""),
"nTrain_Signal=0:nTrain_Background=0:nTest_Signal=0:nTest_Background=0:SplitMode=Random:V\
")
factory.BookMethod(TMVA.Types.kBDT, "BDT",
"!H:!V:NTrees=850:MinNodeSize=5%:MaxDepth=2:BoostType=AdaBoost:AdaBoostBeta=0.5:UseBaggedBoost:BaggedSampleFraction\
=0.5:SeparationType=GiniIndex:nCuts=20:NegWeightTreatment=IgnoreNegWeightsInTraining")
On 05/02/2017 10:57 PM, Vojtech Pleskot wrote:
> Dear all,
>
> I have a weird problem. I train my BDT with five variables twice -
> on two samples that are defined by the event number (even or odd).
> Both training and testing is OK if I use the even events. However, if
> I use the odd events, the training finishes still but the testing
> emits the error below. I checked that there is no event with
> std::isnan(el2_pt), el2_pt != el2_pt or el2_pt < 0. I also printed out
> a couple of tens of events and el2_pt values seem quite OK.
> What can be wrong?
> Thank you for your help in advance. Just in case, I attach the
> text output of both trainings&testings - for the even (0 in the file
> name) and odd (1 in the file name) events.
> Best regards
>
> Vojtech
>
>
> --- Factory : Ranking input variables (method
> specific)...
> --- BDT : Ranking result (top variable is best
> ranked)
> --- BDT : ------------------------------------------
> --- BDT : Rank : Variable : Variable Importance
> --- BDT : ------------------------------------------
> --- BDT : 1 : m_LQ_min : 2.384e-01
> --- BDT : 2 : m_LQ_max : 2.209e-01
> --- BDT : 3 : el2_pt : 1.925e-01
> --- BDT : 4 : jet2_pt : 1.835e-01
> --- BDT : 5 : m_ll : 1.648e-01
> --- BDT : ------------------------------------------
> --- Factory :
> --- Factory : === Destroy and recreate all methods
> via weight files for testing ===
> --- Factory :
> --- MethodBase : Reading weight file:
> weights/BDT_training_20170502/TMVA_isEE_SR_LQ900_1_20170502_BDT.weights.xml
> --- BDT : Read method "BDT" of type "BDT"
> --- BDT : MVA method was trained with TMVA
> Version: 4.2.1
> --- BDT : MVA method was trained with ROOT
> Version: 6.04/16
> --- Factory : Test all methods...
> --- Factory : Test method: BDT for Classification
> performance
> --- BDT : Evaluation of BDT on testing sample
> (320923 events)
> --- BDT : Elapsed time for evaluation of 320923
> events: 53.2 sec
> --- Factory : Evaluate all methods...
> --- Factory : Evaluate classifier: BDT
> --- BDT : Loop over test events and fill
> histograms with classifier response...
> --- Factory : Write evaluation histograms to file
> --- TFHandler_BDT : Plot event variables for BDT
> --- TFHandler_BDT :
> -----------------------------------------------------------
> --- TFHandler_BDT : Variable Mean RMS
> [ Min Max ]
> --- TFHandler_BDT :
> -----------------------------------------------------------
> --- TFHandler_BDT : m_LQ_min: 238.76 150.45 [
> 32.296 2749.5 ]
> --- TFHandler_BDT : m_ll: 334.55 111.59 [
> 130.00 4714.5 ]
> --- TFHandler_BDT : jet2_pt: 124.72 83.853 [
> 60.000 2455.4 ]
> --- TFHandler_BDT : el2_pt: 111.48 -nan [
> 80.000 1893.1 ]
> --- TFHandler_BDT : m_LQ_max: 392.64 266.74 [
> 42.108 4034.3 ]
> --- TFHandler_BDT :
> -----------------------------------------------------------
> --- Tools : -nan -nan -nan -nan -nan -nan
> --- <FATAL> Tools : <GetSeparation> signal and
> background histograms have different or invalid dimensions:
> ***> abort program execution
>
|
|
From: Aishik G. <ais...@gm...> - 2017-05-02 17:26:15
|
Hello Everyone, I've been adding background training and test trees separately using the argument TMVA.Types.kTraining and TMVA.Types.kTesting respectively in the DataLoader.AddBackgroundTree() function. However now I need to add a few more background trees whose events must be split 50-50 between the test and training set. If I load these trees in the begining as kMaxTreeType, then regardless of the tree types I specify for the following background trees, they all get split 50-50 between the training and test samples. And if add this kMaxTreeType background tree at the end, then my program crashes because it says I did not specify a tree type. Is there any way to split events of a few background trees but not for all? Any advice will be appreciated. Best Regards, Aishik |
|
From: Peter S. <pet...@gm...> - 2017-04-28 05:45:42
|
Hi, Indeed, I overlooked 'string'. One could add a number for each potential string and add this as a spectator and keep track of the mapping outside of the trees. cheers, Peter Wolf Behrenhoff <wo...@be...> schrieb am Do., 27. Apr. 2017, 17:40: > Dear Peter, > > > > your answer does not fit to the question. > > > > To my knowledge, AddSpectator can only be used to add numbers/numeric > expressions (not strings), and worse, the result will be a Float_t. So if > you try to add a Long64_t as spectator, you will only find a Float_t in the > resulting tree (with possible data loss). > > > > I would love to have something like "AddSpectatorBranch" to add a full > Branch as spectator - which should be copied unchanged by TMVA. > > > > Cheers, > > Wolf > > > > > > Am Donnerstag, 27. April 2017, 14:24:12 CEST schrieb Peter Speckmayer: > > > Hi, > > > > > > By using 'AddSpectator' instead of 'AddVariable' you can add variables > > > which are not taken into account for the computation of the classifiers. > > > > > > Hope this helps > > > > > > Cheers, > > > Peter > > > > > > Garay Garcia, Jasone <jas...@de...> schrieb am Do., 27. Apr. > 2017, > > > > > > 15:55: > > > > Dear Experts, > > > > > > > > is it possible to add a string as an spectator variable? If not, is > there > > > > any trick to make this possible? > > > > > > > > Thanks a lot in advance, > > > > Cheers, > > > > Jasone > > > > > > > > > > > > > -------------------------------------------------------------------------- > > > > ---- Check out the vibrant tech community on one of the world's most > > > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > > _______________________________________________ > > > > TMVA-users mailing list > > > > TMV...@li... > > > > https://lists.sourceforge.net/lists/listinfo/tmva-users > > > > > > > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > TMVA-users mailing list > TMV...@li... > https://lists.sourceforge.net/lists/listinfo/tmva-users > |
|
From: Wolf B. <wo...@be...> - 2017-04-27 15:39:55
|
Dear Peter, your answer does not fit to the question. To my knowledge, AddSpectator can only be used to add numbers/numeric expressions (not strings), and worse, the result will be a Float_t. So if you try to add a Long64_t as spectator, you will only find a Float_t in the resulting tree (with possible data loss). I would love to have something like "AddSpectatorBranch" to add a full Branch as spectator - which should be copied unchanged by TMVA. Cheers, Wolf Am Donnerstag, 27. April 2017, 14:24:12 CEST schrieb Peter Speckmayer: > Hi, > > By using 'AddSpectator' instead of 'AddVariable' you can add variables > which are not taken into account for the computation of the classifiers. > > Hope this helps > > Cheers, > Peter > > Garay Garcia, Jasone <jas...@de...> schrieb am Do., 27. Apr. 2017, > > 15:55: > > Dear Experts, > > > > is it possible to add a string as an spectator variable? If not, is there > > any trick to make this possible? > > > > Thanks a lot in advance, > > Cheers, > > Jasone > > > > > > -------------------------------------------------------------------------- > > ---- Check out the vibrant tech community on one of the world's most > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > _______________________________________________ > > TMVA-users mailing list > > TMV...@li... > > https://lists.sourceforge.net/lists/listinfo/tmva-users |
|
From: Peter S. <pet...@gm...> - 2017-04-27 14:24:30
|
Hi, By using 'AddSpectator' instead of 'AddVariable' you can add variables which are not taken into account for the computation of the classifiers. Hope this helps Cheers, Peter Garay Garcia, Jasone <jas...@de...> schrieb am Do., 27. Apr. 2017, 15:55: > Dear Experts, > > is it possible to add a string as an spectator variable? If not, is there > any trick to make this possible? > > Thanks a lot in advance, > Cheers, > Jasone > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > TMVA-users mailing list > TMV...@li... > https://lists.sourceforge.net/lists/listinfo/tmva-users > |
|
From: Garay G. J. <jas...@de...> - 2017-04-27 13:55:23
|
Dear Experts, is it possible to add a string as an spectator variable? If not, is there any trick to make this possible? Thanks a lot in advance, Cheers, Jasone |
|
From: Helge V. <Hel...@ce...> - 2017-04-05 13:10:37
|
Hi, well, if you would want to make all the 'cuts' (etc.) yourself, you'd have a hell of a lot to do in order to 'reproduce' a boosted decision tree. :) What you should do for your analysis however is: Use TMVAReader (see TMVAClassificationApplication.C for an example), calcualte for each even the 'MVA-output variable (the 'result or the 'answer' of the BDT to your event), treat THIS output as your 'simgle variable' on which you place a cut. And the best position of this 'cut' you can judge by looking at the MVA-output distributions (see gui) cheers, Helge On 5 April 2017 at 14:55, Ben Smith <ben...@gm...> wrote: > Hi, > > I managed to produce ROC curves with TMVA for a few classifiers and I see > that BDT is the one to pick. I have 4 variables in my problem, but I don't > see how could I use this to make actual cuts on my variables (and this is > the whole goal here). Where is this information?? I don't see it in the > TMVA.root file. Is there a macro to produce it? > > Also, I ran a few macros, for example "compareanapp.C" which gives the > "output classifier" showing signal and background. In the manual, it says > that signal accumulates by convention in the higher values, and background > in the lower values. This is what I see, but which quantity is being > plotted??? It says simply the name of the classifier, but what does that > mean? There has to be a mathematical definition somewhere. > > Thanks a lot for any help! > > Ben > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > TMVA-users mailing list > TMV...@li... > https://lists.sourceforge.net/lists/listinfo/tmva-users > |
|
From: Peter S. <pet...@gm...> - 2017-04-05 13:06:45
|
Hi Ben, To apply a classifier to your data you have to use the TMVA::Reader. There is the main example of how to do that in the tmva/tutorial folder: TMVAClassificationApplication.C. You basically have to load the classifier in the reader, define all variables just as for the TMVA::Factory and connect them to local variables which you then --- for each sample --- you have to fill with your data. And then you can call 'Evaluate'. --- Voila! --- You get the classifier value for each of your samples one by one. cheers, Peter On Wed, Apr 5, 2017 at 2:55 PM Ben Smith <ben...@gm...> wrote: > Hi, > > I managed to produce ROC curves with TMVA for a few classifiers and I see > that BDT is the one to pick. I have 4 variables in my problem, but I don't > see how could I use this to make actual cuts on my variables (and this is > the whole goal here). Where is this information?? I don't see it in the > TMVA.root file. Is there a macro to produce it? > > Also, I ran a few macros, for example "compareanapp.C" which gives the > "output classifier" showing signal and background. In the manual, it says > that signal accumulates by convention in the higher values, and background > in the lower values. This is what I see, but which quantity is being > plotted??? It says simply the name of the classifier, but what does that > mean? There has to be a mathematical definition somewhere. > > Thanks a lot for any help! > > Ben > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > TMVA-users mailing list > TMV...@li... > https://lists.sourceforge.net/lists/listinfo/tmva-users > |
|
From: Ben S. <ben...@gm...> - 2017-04-05 12:55:33
|
Hi, I managed to produce ROC curves with TMVA for a few classifiers and I see that BDT is the one to pick. I have 4 variables in my problem, but I don't see how could I use this to make actual cuts on my variables (and this is the whole goal here). Where is this information?? I don't see it in the TMVA.root file. Is there a macro to produce it? Also, I ran a few macros, for example "compareanapp.C" which gives the "output classifier" showing signal and background. In the manual, it says that signal accumulates by convention in the higher values, and background in the lower values. This is what I see, but which quantity is being plotted??? It says simply the name of the classifier, but what does that mean? There has to be a mathematical definition somewhere. Thanks a lot for any help! Ben |
|
From: Kim A. <kia...@ce...> - 2017-03-30 19:08:24
|
Hi all, This message was intended for the tmva-dev list. Please ignore and sorry for the spam. Cheers, Kim Kim Albertsson wrote: > Hi all, > > I was wondering about the roc curve calculation used by the > TMVA::Factory. It seemingly does not take event weights into account, > and thus will calculate the curve incorrectly. > > Am I missing something here, is this not the way to calculate roc curves? > > There also seems to be two ways of calculating a roc curve: > ROCCurve.cxx and ROCCalc.cxx, and three ways of calculating the roc > integral: in the the above two files and additionally > TMVA::MethodBase::GetROCIntegral. > > What is the intended purpose for each of these and why is the > functionality split up in several places? > > Insight into this would be much appreciated! > > Cheers, > Kim |
|
From: Kim A. <kia...@ce...> - 2017-03-30 13:13:44
|
Hi all, I was wondering about the roc curve calculation used by the TMVA::Factory. It seemingly does not take event weights into account, and thus will calculate the curve incorrectly. Am I missing something here, is this not the way to calculate roc curves? There also seems to be two ways of calculating a roc curve: ROCCurve.cxx and ROCCalc.cxx, and three ways of calculating the roc integral: in the the above two files and additionally TMVA::MethodBase::GetROCIntegral. What is the intended purpose for each of these and why is the functionality split up in several places? Insight into this would be much appreciated! Cheers, Kim |
|
From: Olivia J. <oli...@b2...> - 2017-03-28 16:57:20
|
Hi, I am just doing a follow up with my previous email to check whether you are interested in our Contacts List. If you can let me know your targeted criteria, I can assist you with the count/costs, and sample details for your consideration. Target Industry: _____________; (Any) Target Geography: ______________; (USA/Canada/UK/Europe/Across the world) Target Job Title: _________________; (CEO, CFO, CMO, VP, Others etc.) Looking forward to hear from you soon! Thank You From: Olivia Jones [mailto:oli...@b2...] Sent: Monday, February 27, 2017 12:59 PM To: 'tmv...@li...' Subject: Business information and media industry contacts Hi, Hope you doing well! We've released a potential client's list for Business information and media industry professionals for the year 2017. Would you be interested in acquiring client's list to sell or market your products in your location and worldwide? And, please let me know can I consider the below criteria as your target market? Accordingly, will get back to you with Data counts, Accuracy, Update and much more information Job titles/categories:- * Publishers, CEOs, Chief Revenue Officers * Business development and licensing executives * Marketers * Investors and deal makers * Human capital practitioners * Data strategists, systems engineers, online managers * Many more If you would like to reach any other Industries/Titles. You can just fill in below and revert back on the same email. Target Criteria:- Target Industries: ______? (We maintain data across all Business to Business (b2b) Industries across North America, UK and Europe) Job Titles: ______? Geography: ______? Would be pleased to hear back from you Regards, Olivia Jones Marketing Executive _____ Instructions to remove from this mailing: Reply with subject line as "Leave out" and indicate your email address to be removed from our database. |
31 messages has been excluded from this view by a project administrator.
