Screenshot instructions:
Windows
Mac
Red Hat Linux
Ubuntu
Click URL instructions:
Rightclick on ad, choose "Copy Link", then paste here →
(This may not be possible with some types of ads)
From: Michael Maul <mike.maul@gm...>  20121213 00:25:59
Attachments:
Message as HTML

I am having problems getting cgdescent to converge. I am re implementing code done in octave which converges with fminuc. I'v read through the CGDescent users guide but am unsure where to start tuning. During some testing it cgdescent makes very very small increases of theta at each call of the gradient function. I've tried altering the initial theta's to various values but it doesn't help. Is there perhaps a setting in cgparams that might help? Below is my cost and gradient functions if that helps shed any light on my problem: (defun sigmoid (theta) (/ 1 (+ 1 (exp ( theta)))) ) (defun cost (X y theta) (let* ((m (idxshape X 0)) (h (sigmoid (* X theta))) (j (* (/ 1 m) ( ( (mat.* (mattranspose (log h)) y)) (mat.* (mattranspose (log ( 1 h))) ( 1 y)))))) (j 0 0) ) ) (defun gradient (X y theta) (let* ((m (idxshape X 0)) (h (sigmoid (* X theta)))) ($*0 (* (/ 1 m) (mat.* (mattranspose ( h y)) X)) 0)) ) (cgdescent theta (lambda (t) (cost X y t)) (lambda (gx t) (arraycopy (gradient X y t) gx)) 1e6) 
From: juenglin <juenglin@cs...>  20121213 02:23:53

It sounds like you already figured out how to get some debug information out (set PrintLevel in the cg parameters). You could try restarting On Thu, 13 Dec 2012 00:25:32 +0000, Michael Maul wrote: > I am having problems getting cgdescent to converge. I am re > implementing code done in octave which converges with fminuc. > > Iv read through the CGDescent users guide but am unsure where to > start tuning. During some testing it cgdescent makes very very > small increases of theta at each call of the gradient function. > > Ive tried altering the initial thetas to various values but it doesnt > help. Is there perhaps a setting in cgparams that might help? > > Below is my cost and gradient functions if that helps shed any light > on my problem: > > (defun sigmoid (theta) > (/ 1 (+ 1 (exp ( theta)))) > ) > > (defun cost (X y theta) > (let* ((m (idxshape X 0)) > (h (sigmoid (* X theta))) > (j (* (/ 1 m) ( ( (mat.* (mattranspose (log h)) y)) > (mat.* (mattranspose (log ( 1 h))) > ( 1 y)))))) > (j 0 0) > ) > ) > > (defun gradient (X y theta) > (let* ((m (idxshape X 0)) > (h (sigmoid (* X theta)))) > ($*0 (* (/ 1 m) (mat.* (mattranspose ( h y)) X)) 0)) > ) > > (cgdescent theta (lambda (t) (cost X y t)) > (lambda (gx t) (arraycopy (gradient X y t) gx)) 1e6) 
From: juenglin <juenglin@cs...>  20121213 02:27:58

<sent off too early> You could try to force earlier restarts per the restart_fac parameter. There is a tiny bit of explanation on the parameters in the file cg_user.h. On Wed, 12 Dec 2012 18:23:45 0800, juenglin wrote: > It sounds like you already figured out how to get some debug > information out (set PrintLevel in the cg parameters). > > You could try restarting > > > On Thu, 13 Dec 2012 00:25:32 +0000, Michael Maul wrote: >> I am having problems getting cgdescent to converge. I am re >> implementing code done in octave which converges with fminuc. >> >> Iv read through the CGDescent users guide but am unsure where to >> start tuning. During some testing it cgdescent makes very very >> small increases of theta at each call of the gradient function. >> >> Ive tried altering the initial thetas to various values but it >> doesnt >> help. Is there perhaps a setting in cgparams that might help? >> >> Below is my cost and gradient functions if that helps shed any light >> on my problem: >> >> (defun sigmoid (theta) >> (/ 1 (+ 1 (exp ( theta)))) >> ) >> >> (defun cost (X y theta) >> (let* ((m (idxshape X 0)) >> (h (sigmoid (* X theta))) >> (j (* (/ 1 m) ( ( (mat.* (mattranspose (log h)) y)) >> (mat.* (mattranspose (log ( 1 h))) >> ( 1 y)))))) >> (j 0 0) >> ) >> ) >> >> (defun gradient (X y theta) >> (let* ((m (idxshape X 0)) >> (h (sigmoid (* X theta)))) >> ($*0 (* (/ 1 m) (mat.* (mattranspose ( h y)) X)) 0)) >> ) >> >> (cgdescent theta (lambda (t) (cost X y t)) >> (lambda (gx t) (arraycopy (gradient X y t) gx)) 1e6) > > > >  > LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial > Remotely access PCs and mobile devices and provide instant support > Improve your efficiency, and focus on delivering more valueadd > services > Discover what IT Professionals Know. Rescue delivers > http://p.sf.net/sfu/logmein_12329d2d > _______________________________________________ > Lushusers mailing list > Lushusers@... > https://lists.sourceforge.net/lists/listinfo/lushusers 
From: Michael Maul <mike.maul@gm...>  20121213 19:07:51
Attachments:
Message as HTML

It turns out I did not need to adjust restart_fac. My cost and gradient were munged, they gave same results as octave fn's with initial theta but they diverged. But I got them dialed in not. It even converged in cgdescent. *(cgdescent theta (lambda (t) (cost X y t)) * * (lambda (gx t) (arraycopy (gradient X y t) gx) 0) 1e4)* * = [d 25.4828 0.2088 0.2041]* But a successive run crashes or fails to converge with the same parameters depending on it's mood.* * * ? (cgdescent theta (lambda (t) (cost X y t)) * * (lambda (gx t) (arraycopy (gradient X y t) gx) 0) 1e4)* **** lush runtime error: FPU exception 'overflow' at 0xb73bd1b2 * *** in: C__cg_descent* * *** exp : Runtime error in compiled code* I've seen these FPU exception overflows before in lush. Perhaps it's my platform, I'm running in an VirtualBox VM? I'll try on a metal machine. It could however be something not cleaning up properly and affecting the initial state of the next run.., or it could be something else entirely. Anyways if anyone has some advice, would love to hear it. BTW the working cost and gradient fn's are below *(defun sigmoidnc (theta)* * (/ 1 (+ 1 (exp ( theta))))* *)* *(defun cost (X y theta)* * (let* ((m (idxshape X 0))* * (h (sigmoidnc (idxm2dotm1 X theta)))* * (j (* (/ 1 m)* * ( (mat.* (idxreshape (log (sigmoidnc (idxm2dotm1 X theta))) (list 1 m)) ( y ))* * (mat.* (idxreshape (log ( 1 (sigmoidnc (idxm2dotm1 X theta)))) (list 1 m)) ( 1 y ))))* * ))* * (j 0 0) * * )* * )* * * *(defun gradient (X y theta)* * (let* ((m (idxshape X 0))* * (h (sigmoidnc (idxm2dotm1 X theta))))* * (idxflatten (* (/ 1 m) (mat.* ( h (mattranspose y)) X))))* *)* Also for anyone following along this is how to set and use cgparams. *(defparameter cgp (cgparams)) * *(assign (:cgp:access 'restart_fac) 5)* *(assign (:cgp:access 'PrintLevel) 1)* *(cgdescent theta (lambda (t) (cost X y t)) * * (lambda (gx t) (arraycopy (gradient X y t) gx) 0) 1e4 cgp)* 
From: Michael Maul <mike.maul@gm...>  20121217 03:56:47
Attachments:
Message as HTML

It just occurred to me I should check for evidence of the behavior problem in testcgdescent. The wood test runs okay consistently. However after 4 consecutive runs of the rosenbrock test it fail to converge. The bad behavior is not as evident in testcgdescent, but if you run rosenbock enough you should also see it. (libload "optim/cgdescent") (libload "optim/testfuncs/rosenbrock") (libload "optim/testfuncs/wood") (cgdescent [1.2 1] rosenbrock rosenbrockgrad 1e6) On Mon, Dec 17, 2012 at 1:45 AM, Michael Maul <mike.maul@...> wrote: > >>Also, I wonder how your example can work with the gradient function (the > second lambda) taking two arguments. > From lgbfs.lsh and cgdescent.lsh > #? (cgdescent <x0> <f> <g> <gtol> [<cgp> <arg0> <arg1> ...]) > #? (lbfgs <x0> <f> <g> <gtol> [<params> <arg0> <arg1> ...]) > ... > ;; Function <g> takes the arguments <gx>, <x>, <arg0>, <arg1>, ..., > ;; writes the gradient at <x> into <gx> and returns nothing. > > The second arg is where updated theta's are stored. > > You can get code and data here: > https://github.com/mmaul/scratchpad > > I had do do some work on lapack and blas to get libfgs working. If you > don't have a working libfgs then you can build lush from my lush2 repo on > git hub where I have the modified code ( > https://github.com/mmaul/lush2.git ) > > > On Sun, Dec 16, 2012 at 5:27 AM, juenglin <juenglin@...> wrote: > >> >> You didn't give enough information for others to reproduce what you are >> seeing. >> What are the values for X Y and theta, for example? >> >> Also, I wonder how your example can work with the gradient function (the >> second lambda) taking two arguments. >> >> >> >> On Thu, 13 Dec 2012 19:07:18 +0000, Michael Maul wrote: >> >>> It turns out I did not need to adjust restart_fac. My cost and >>> gradient were munged, they gave same results as octave fns with >>> >>> initial theta but they diverged. But I got them dialed in not. It even >>> converged in cgdescent. >>> >>> (CGDESCENT THETA (LAMBDA (T) (COST X Y T)) >>> (LAMBDA (GX T) (ARRAYCOPY (GRADIENT X Y T) GX) 0) 1E4) >>> = [D 25.4828 0.2088 0.2041] >>> >>> But a successive run crashes or fails to converge with the same >>> parameters depending on its mood. >>> >>> ? (CGDESCENT THETA (LAMBDA (T) (COST X Y T)) >>> (LAMBDA (GX T) (ARRAYCOPY (GRADIENT X Y T) GX) 0) 1E4) >>> *** LUSH RUNTIME ERROR: FPU EXCEPTION OVERFLOW AT 0XB73BD1B2 >>> ** IN: C__CG_DESCENT >>> *** EXP : RUNTIME ERROR IN COMPILED CODE >>> >>> Ive seen these FPU exception overflows before in lush. Perhaps its my >>> platform, Im running in an VirtualBox VM? Ill try on a metal machine. >>> >>> It could however be something not cleaning up properly and >>> affecting the initial state of the next run.., or it could be >>> something else entirely. >>> >>> Anyways if anyone has some advice, would love to hear it. BTW the >>> working cost and gradient fns are below >>> >>> (DEFUN SIGMOIDNC (THETA) >>> (/ 1 (+ 1 (EXP ( THETA)))) >>> ) >>> (DEFUN COST (X Y THETA) >>> (LET* ((M (IDXSHAPE X 0)) >>> (H (SIGMOIDNC (IDXM2DOTM1 X THETA))) >>> (J (* (/ 1 M) >>> ( (MAT.* (IDXRESHAPE (LOG (SIGMOIDNC (IDXM2DOTM1 X >>> THETA))) (LIST 1 M)) ( Y )) >>> (MAT.* (IDXRESHAPE (LOG ( 1 (SIGMOIDNC (IDXM2DOTM1 X >>> THETA)))) (LIST 1 M)) ( 1 Y )))) >>> )) >>> (J 0 0) >>> ) >>> ) >>> >>> (DEFUN GRADIENT (X Y THETA) >>> (LET* ((M (IDXSHAPE X 0)) >>> (H (SIGMOIDNC (IDXM2DOTM1 X THETA)))) >>> (IDXFLATTEN (* (/ 1 M) (MAT.* ( H (MATTRANSPOSE Y)) >>> >>> X)))) >>> ) >>> >>> Also for anyone following along this is how to set and use cgparams. >>> (defparameter cgp (cgparams)) >>> >>> (ASSIGN (:CGP:ACCESS RESTART_FAC) 5) >>> (ASSIGN (:CGP:ACCESS PRINTLEVEL) 1) >>> >>> (CGDESCENT THETA (LAMBDA (T) (COST X Y T)) >>> (LAMBDA (GX T) (ARRAYCOPY (GRADIENT X Y T) GX) 0) 1E4 CGP) >>> >> >> > 
From: Ralf Juengling <juenglin@cs...>  20130418 00:24:19

Hi Michael, There was a bug in the CG_DESCENT interface, which I believe was causing the seemingly random FPU exceptions. I just submitted a fix to svn. Also there is easier syntax for making changes to the parameters, ? (defparameter cgp (cgparams)) ? (cgp restart_fac 5) = 5 Ralf On Mon, 17 Dec 2012, Michael Maul wrote: > >>Also, I wonder how your example can work with the gradient function (the second lambda) taking > two arguments. > From lgbfs.lsh and cgdescent.lsh > #? (cgdescent <x0> <f> <g> <gtol> [<cgp> <arg0> <arg1> ...]) > #? (lbfgs <x0> <f> <g> <gtol> [<params> <arg0> <arg1> ...]) > ... > ;; Function <g> takes the arguments <gx>, <x>, <arg0>, <arg1>, ..., > ;; writes the gradient at <x> into <gx> and returns nothing. > > The second arg is where updated theta's are stored. > > You can get code and data here: > https://github.com/mmaul/scratchpad > > I had do do some work on lapack and blas to get libfgs working. If you don't have a working > libfgs then you can build lush from my lush2 repo on git hub where I have the modified code > ( https://github.com/mmaul/lush2.git ) > > > On Sun, Dec 16, 2012 at 5:27 AM, juenglin <juenglin@...> wrote: > > You didn't give enough information for others to reproduce what you are seeing. > What are the values for X Y and theta, for example? > > Also, I wonder how your example can work with the gradient function (the second > lambda) taking two arguments. > > > On Thu, 13 Dec 2012 19:07:18 +0000, Michael Maul wrote: > It turns out I did not need to adjust restart_fac. My cost and > gradient were munged, they gave same results as octave fns with > initial theta but they diverged. But I got them dialed in not. It even > converged in cgdescent. > > (CGDESCENT THETA (LAMBDA (T) (COST X Y T)) > (LAMBDA (GX T) (ARRAYCOPY (GRADIENT X Y T) GX) 0) 1E4) > = [D 25.4828 0.2088 0.2041] > But a successive run crashes or fails to converge with the same > parameters depending on its mood. > > ? (CGDESCENT THETA (LAMBDA (T) (COST X Y T)) > (LAMBDA (GX T) (ARRAYCOPY (GRADIENT X Y T) GX) 0) 1E4) > *** LUSH RUNTIME ERROR: FPU EXCEPTION OVERFLOW AT 0XB73BD1B2 > ** IN: C__CG_DESCENT > *** EXP : RUNTIME ERROR IN COMPILED CODE > > Ive seen these FPU exception overflows before in lush. Perhaps its my > platform, Im running in an VirtualBox VM? Ill try on a metal machine. > It could however be something not cleaning up properly and > affecting the initial state of the next run.., or it could be > something else entirely. > > Anyways if anyone has some advice, would love to hear it. BTW the > working cost and gradient fns are below > > (DEFUN SIGMOIDNC (THETA) > (/ 1 (+ 1 (EXP ( THETA)))) > ) > (DEFUN COST (X Y THETA) > (LET* ((M (IDXSHAPE X 0)) > (H (SIGMOIDNC (IDXM2DOTM1 X THETA))) > (J (* (/ 1 M) > ( (MAT.* (IDXRESHAPE (LOG (SIGMOIDNC (IDXM2DOTM1 X > THETA))) (LIST 1 M)) ( Y )) > (MAT.* (IDXRESHAPE (LOG ( 1 (SIGMOIDNC (IDXM2DOTM1 X > THETA)))) (LIST 1 M)) ( 1 Y )))) > )) > (J 0 0) > ) > ) > > (DEFUN GRADIENT (X Y THETA) > (LET* ((M (IDXSHAPE X 0)) > (H (SIGMOIDNC (IDXM2DOTM1 X THETA)))) > (IDXFLATTEN (* (/ 1 M) (MAT.* ( H (MATTRANSPOSE Y)) > X)))) > ) > > Also for anyone following along this is how to set and use cgparams. > (defparameter cgp (cgparams)) > > (ASSIGN (:CGP:ACCESS RESTART_FAC) 5) > (ASSIGN (:CGP:ACCESS PRINTLEVEL) 1) > > (CGDESCENT THETA (LAMBDA (T) (COST X Y T)) > (LAMBDA (GX T) (ARRAYCOPY (GRADIENT X Y T) GX) 0) 1E4 CGP) > > > > > 