Contents
clear all
close all
fig = 1;
LASSO
nInstances = 250;
nVars = 50;
X = randn(nInstances,nVars);
y = X*((rand(nVars,1) > .5).*randn(nVars,1)) + randn(nInstances,1);
wLS = X\y;
lambda = 100*ones(nVars,1);
R = chol(X'*X + diag(lambda));
wRR = R\(R'\(X'*y));
lambda = 100*ones(nVars,1);
funObj = @(w)GaussianLoss(w,X,y);
w_init = wRR;
fprintf('\nComputing LASSO Coefficients...\n');
wLASSO = L1GeneralProjection(funObj,w_init,lambda);
zeroThreshold = 1e-4;
fprintf('Number of non-zero variables in Least Squares solution: %d\n',sum(abs(wLS)>zeroThreshold));
fprintf('Number of non-zero variables in Ridge Regression solution: %d\n',sum(abs(wRR)>zeroThreshold));
fprintf('Number of non-zero variables in LASSO solution: %d\n',sum(abs(wLASSO)>zeroThreshold));
figure(fig);fig=fig+1;
clf;hold on;
subplot(2,2,1);
stem(wLS,'r');
xlim([1 nVars]);
yl = ylim;
title('Least Squares');
subplot(2,2,2);
stem(wRR,'b');
xlim([1 nVars]);
ylim(yl);
title('Ridge Regression');
subplot(2,2,3);
stem(wLASSO,'g');
xlim([1 nVars]);
title('LASSO');
ylim(yl);
pause;
Computing LASSO Coefficients...
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 2.39373e+003 7.89553e+002 24
2 3 1.00000e+000 2.34215e+003 4.42028e+001 23
3 4 1.00000e+000 2.34203e+003 2.11742e-012 23
Solution Found
Number of non-zero variables in Least Squares solution: 50
Number of non-zero variables in Ridge Regression solution: 50
Number of non-zero variables in LASSO solution: 23
Elastic Net
lambdaL2 = 100*ones(nVars,1);
lambdaL1 = 100*ones(nVars,1);
penalizedFunObj = @(w)penalizedL2(w,funObj,lambdaL2);
fprintf('\nComputing Elastic Net Coefficients...\n');
wElastic = L1GeneralProjection(penalizedFunObj,w_init,lambdaL1);
fprintf('Number of non-zero variables in Elastic Net solution: %d\n',sum(abs(wElastic)>zeroThreshold));
subplot(2,2,4);
stem(wElastic,'c');
xlim([1 nVars]);
ylim(yl);
title('Elastic Net');
pause;
Computing Elastic Net Coefficients...
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 4.17805e+003 2.92622e+002 25
2 3 1.00000e+000 4.17320e+003 5.43099e-001 26
3 4 1.00000e+000 4.17320e+003 2.52953e-012 27
Solution Found
Number of non-zero variables in Elastic Net solution: 27
Logistic Regression
X = [ones(nInstances,1) X];
y = sign(y);
funObj = @(w)LogisticLoss(w,X,y);
w_init = zeros(nVars+1,1);
fprintf('\nComputing Maximum Likelihood Logistic Regression Coefficients\n');
mfOptions.Method = 'newton';
wLogML = minFunc(funObj,w_init,mfOptions);
fprintf('\nComputing L2-Regularized Logistic Regression Coefficients...\n');
lambda = 15*ones(nVars+1,1);
lambda(1) = 0;
funObjL2 = @(w)penalizedL2(w,funObj,lambda);
wLogL2 = minFunc(funObjL2,w_init,mfOptions);
fprintf('\nComputing L1-Regularized Logistic Regression Coefficients...\n');
wLogL1 = L1GeneralProjection(funObj,w_init,lambda);
fprintf('\nComputing Elastic-Net Logistic Regression Coefficients...\n');
wLogL1L2 = L1GeneralProjection(funObjL2,w_init,lambda);
figure(fig);fig=fig+1;
clf;hold on;
subplot(2,2,1);
stem(wLogML,'r');
xlim([1 nVars+1]);
title('Maximum Likelihood Logistic Regression');
subplot(2,2,2);
stem(wLogL2,'b');
xlim([1 nVars+1]);
title('L2-Regularized Logistic Regression');
subplot(2,2,3);
stem(wLogL1,'g');
xlim([1 nVars+1]);
title('L1-Regularized Logistic Regression');
subplot(2,2,4);
stem(wLogL1L2,'c');
xlim([1 nVars+1]);
title('Elastic-Net Logistic Regression');
fprintf('Number of Features Selected by Maximum Likelihood Logistic Regression classifier: %d (out of %d)\n',nnz(wLogML(2:end)),nVars);
fprintf('Number of Features Selected by L2-regualrized Logistic Regression classifier: %d (out of %d)\n',nnz(wLogL2(2:end)),nVars);
fprintf('Number of Features Selected by L1-regualrized Logistic Regression classifier: %d (out of %d)\n',nnz(wLogL1(2:end)),nVars);
fprintf('Number of Features Selected by Elastic-Net Logistic Regression classifier: %d (out of %d)\n',nnz(wLogL1L2(2:end)),nVars);
fprintf('Classification error rate on training data for L1-regularied Logistic Regression: %.2f\n',sum(y ~= sign(X*wLogL1))/length(y));
pause;
Computing Maximum Likelihood Logistic Regression Coefficients
Iteration FunEvals Step Length Function Val Opt Cond
1 2 1.00000e+000 6.86571e+001 1.75209e+002
2 3 1.00000e+000 3.93386e+001 7.27380e+001
3 4 1.00000e+000 2.36271e+001 3.16103e+001
4 5 1.00000e+000 1.40112e+001 1.51457e+001
5 6 1.00000e+000 7.27709e+000 7.47367e+000
6 7 1.00000e+000 3.11288e+000 3.44318e+000
7 8 1.00000e+000 1.17398e+000 1.34078e+000
8 9 1.00000e+000 4.31534e-001 5.00461e-001
9 10 1.00000e+000 1.58648e-001 1.85245e-001
10 11 1.00000e+000 5.84188e-002 6.83519e-002
11 12 1.00000e+000 2.15182e-002 2.51923e-002
12 13 1.00000e+000 7.92459e-003 9.28030e-003
13 14 1.00000e+000 2.91768e-003 3.41748e-003
14 15 1.00000e+000 1.07399e-003 1.25811e-003
15 16 1.00000e+000 3.95267e-004 4.63053e-004
16 17 1.00000e+000 1.45455e-004 1.70401e-004
17 18 1.00000e+000 5.35217e-005 6.26999e-005
18 19 1.00000e+000 1.96928e-005 2.30693e-005
19 20 1.00000e+000 7.24546e-006 8.48754e-006
Optimality Condition below TolFun
Computing L2-Regularized Logistic Regression Coefficients...
Iteration FunEvals Step Length Function Val Opt Cond
1 2 1.00000e+000 1.07461e+002 8.19553e+001
2 3 1.00000e+000 1.05742e+002 5.21509e+000
3 4 1.00000e+000 1.05735e+002 2.46316e-002
4 5 1.00000e+000 1.05735e+002 5.37614e-007
Optimality Condition below TolFun
Computing L1-Regularized Logistic Regression Coefficients...
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 1.48387e+002 1.92726e+001 10
2 3 1.00000e+000 1.47836e+002 1.21742e+000 13
3 4 1.00000e+000 1.47833e+002 6.76962e-003 13
Directional Derivative too small
Computing Elastic-Net Logistic Regression Coefficients...
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 1.56629e+002 7.87331e+000 12
2 3 1.00000e+000 1.56585e+002 6.01884e-002 13
3 4 1.00000e+000 1.56585e+002 3.68082e-006 13
Directional Derivative too small
Number of Features Selected by Maximum Likelihood Logistic Regression classifier: 50 (out of 50)
Number of Features Selected by L2-regualrized Logistic Regression classifier: 50 (out of 50)
Number of Features Selected by L1-regualrized Logistic Regression classifier: 12 (out of 50)
Number of Features Selected by Elastic-Net Logistic Regression classifier: 12 (out of 50)
Classification error rate on training data for L1-regularied Logistic Regression: 0.14
L-BFGS Variant
nVars = 5000;
X = [ones(nInstances,1) randn(nInstances,nVars-1)];
y = sign(X*((rand(nVars,1) > .5).*randn(nVars,1)) + randn(nInstances,1));
lambda = 10*ones(nVars,1);
lambda(1) = 0;
funObj = @(w)LogisticLoss(w,X,y);
options.order = -1;
fprintf('\nComputing Logistic Regression Coefficients for model with %d variables with L-BFGS\n',nVars);
wLogistic = L1GeneralProjection(funObj,zeros(nVars,1),lambda,options);
figure(fig);fig=fig+1;
clf;
stem(wLogistic,'g');
title(sprintf('L1-Regularized Logistic Regression (%d vars)',nVars));
fprintf('Number of Features Selected by Logistic Regression classifier: %d (out of %d)\n',nnz(wLogistic(2:end)),nVars);
fprintf('Classification error rate on training data: %.2f\n',sum(y ~= sign(X*wLogistic))/length(y));
pause;
Computing Logistic Regression Coefficients for model with 5000 variables with L-BFGS
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 9.21773e-006 1.73034e+002 4.21082e+003 44
2 3 1.00000e+000 1.53066e+002 1.58428e+003 921
3 4 1.00000e+000 1.51406e+002 1.39955e+003 814
4 5 1.00000e+000 1.49160e+002 1.42850e+003 565
5 6 1.00000e+000 1.47468e+002 9.66323e+002 543
6 7 1.00000e+000 1.45119e+002 3.92766e+002 405
7 8 1.00000e+000 1.42404e+002 2.04276e+002 259
8 9 1.00000e+000 1.41837e+002 2.77657e+002 176
9 11 3.70098e-001 1.41319e+002 1.93528e+002 137
10 12 1.00000e+000 1.40610e+002 1.12085e+002 177
11 13 1.00000e+000 1.40521e+002 1.12108e+002 177
12 14 1.00000e+000 1.40290e+002 1.08923e+002 173
13 15 1.00000e+000 1.39979e+002 9.18733e+001 156
14 16 1.00000e+000 1.39655e+002 6.15887e+001 137
15 18 4.51754e-001 1.39617e+002 5.48989e+001 131
16 19 1.00000e+000 1.39573e+002 5.05404e+001 140
17 21 2.82124e-001 1.39545e+002 4.48069e+001 143
18 22 1.00000e+000 1.39510e+002 4.31349e+001 142
19 23 1.00000e+000 1.39491e+002 4.00321e+001 144
20 24 1.00000e+000 1.39442e+002 3.39714e+001 142
21 26 4.92678e-003 1.39441e+002 3.35485e+001 141
22 27 1.00000e+000 1.39425e+002 3.02178e+001 140
23 28 1.00000e+000 1.39393e+002 2.51520e+001 140
24 29 1.00000e+000 1.39374e+002 2.29708e+001 135
25 30 1.00000e+000 1.39361e+002 2.17834e+001 134
26 31 1.00000e+000 1.39351e+002 2.05736e+001 135
27 32 1.00000e+000 1.39342e+002 1.90641e+001 135
28 33 1.00000e+000 1.39329e+002 1.71747e+001 133
29 34 1.00000e+000 1.39317e+002 1.58740e+001 132
30 35 1.00000e+000 1.39309e+002 1.58843e+001 132
31 36 1.00000e+000 1.39306e+002 1.45999e+001 132
32 37 1.00000e+000 1.39299e+002 1.31419e+001 132
33 38 1.00000e+000 1.39293e+002 1.22677e+001 132
34 39 1.00000e+000 1.39286e+002 1.12153e+001 131
35 40 1.00000e+000 1.39284e+002 1.04426e+001 130
36 41 1.00000e+000 1.39282e+002 9.81087e+000 130
37 42 1.00000e+000 1.39281e+002 9.22432e+000 129
38 43 1.00000e+000 1.39278e+002 8.11503e+000 128
39 44 1.00000e+000 1.39275e+002 7.12848e+000 129
40 45 1.00000e+000 1.39272e+002 6.52562e+000 128
41 46 1.00000e+000 1.39271e+002 6.15033e+000 128
42 47 1.00000e+000 1.39271e+002 6.05703e+000 127
43 48 1.00000e+000 1.39270e+002 5.86874e+000 127
44 49 1.00000e+000 1.39269e+002 5.51452e+000 126
45 50 1.00000e+000 1.39267e+002 4.78758e+000 124
46 51 1.00000e+000 1.39266e+002 4.40646e+000 122
47 52 1.00000e+000 1.39266e+002 4.16427e+000 124
48 53 1.00000e+000 1.39265e+002 3.99603e+000 125
49 54 1.00000e+000 1.39265e+002 3.73976e+000 125
50 55 1.00000e+000 1.39265e+002 3.47428e+000 125
51 56 1.00000e+000 1.39264e+002 3.17192e+000 124
52 57 1.00000e+000 1.39264e+002 2.85125e+000 124
53 58 1.00000e+000 1.39263e+002 2.59896e+000 124
54 59 1.00000e+000 1.39263e+002 2.50296e+000 124
55 60 1.00000e+000 1.39263e+002 2.36530e+000 124
56 61 1.00000e+000 1.39263e+002 2.09276e+000 123
57 62 1.00000e+000 1.39263e+002 1.99303e+000 122
58 63 1.00000e+000 1.39262e+002 1.88515e+000 122
59 64 1.00000e+000 1.39262e+002 1.77991e+000 124
60 65 1.00000e+000 1.39262e+002 1.69069e+000 124
61 66 1.00000e+000 1.39262e+002 1.55651e+000 124
62 67 1.00000e+000 1.39262e+002 1.40401e+000 124
63 68 1.00000e+000 1.39262e+002 1.34019e+000 124
64 69 1.00000e+000 1.39262e+002 1.29385e+000 124
65 70 1.00000e+000 1.39262e+002 1.21956e+000 123
66 71 1.00000e+000 1.39262e+002 1.15352e+000 123
67 72 1.00000e+000 1.39262e+002 1.07838e+000 122
68 73 1.00000e+000 1.39262e+002 9.96773e-001 122
69 74 1.00000e+000 1.39262e+002 9.25587e-001 122
70 75 1.00000e+000 1.39262e+002 8.55251e-001 123
71 76 1.00000e+000 1.39262e+002 7.36927e-001 123
72 77 1.00000e+000 1.39262e+002 6.94010e-001 123
73 78 1.00000e+000 1.39262e+002 6.71398e-001 123
74 79 1.00000e+000 1.39262e+002 6.26718e-001 123
75 80 1.00000e+000 1.39262e+002 6.06825e-001 123
76 81 1.00000e+000 1.39262e+002 5.62267e-001 123
77 82 1.00000e+000 1.39262e+002 5.00978e-001 123
78 83 1.00000e+000 1.39262e+002 4.66108e-001 123
79 84 1.00000e+000 1.39262e+002 4.39002e-001 123
80 85 1.00000e+000 1.39262e+002 4.29464e-001 123
81 86 1.00000e+000 1.39262e+002 4.08420e-001 123
82 87 1.00000e+000 1.39262e+002 3.74898e-001 123
83 88 1.00000e+000 1.39262e+002 3.36313e-001 123
84 89 1.00000e+000 1.39262e+002 3.17204e-001 123
85 90 1.00000e+000 1.39262e+002 3.05943e-001 123
86 91 1.00000e+000 1.39262e+002 2.92069e-001 123
87 92 1.00000e+000 1.39262e+002 2.78704e-001 123
88 93 1.00000e+000 1.39262e+002 2.48789e-001 123
89 94 1.00000e+000 1.39262e+002 2.27548e-001 123
Directional Derivative too small
Number of Features Selected by Logistic Regression classifier: 168 (out of 5000)
Classification error rate on training data: 0.00
Lasso Regularization Path
nInstances = 100;
nVars = 10;
X = randn(nInstances,nVars);
y = X*((rand(nVars,1) > .5).*randn(nVars,1)) + randn(nInstances,1);
[f,g] = GaussianLoss(zeros(nVars,1),X,y);
lambdaMax = max(abs(g));
lambdaInc = .01;
fprintf('Computing Least Squares L1-Regularization path\n');
funObj = @(w)GaussianLoss(w,X,y);
w = zeros(nVars,1);
options = struct('verbose',0);
for mult = 1-lambdaInc:-lambdaInc:0
lambda = mult*lambdaMax*ones(nVars,1);
w(:,end+1) = L1GeneralProjection(funObj,w(:,end),lambda,options);
end
figure(fig);fig=fig+1;
prettyPlot(1:-lambdaInc:0,w',[],'Regression Coefficients vs. L1-Regularization Strength (L1-Regularized Least Squares)','percent of lambdaMax','coefficient values',0,-1);
pause;
Computing Least Squares L1-Regularization path
Logistic Regression Regularization Path
X = [ones(nInstances,1) X];
y = sign(y);
funObjBias = @(w)LogisticLoss(w,ones(nInstances,1),y);
bias = minFunc(funObjBias,0,struct('Display','none'));
w = [bias;zeros(nVars,1)];
[f,g] = LogisticLoss(w,X,y);
lambdaMax = max(abs(g));
lambdaInc = .01;
fprintf('Computing Logistic Regression L1-Regularization path\n');
funObj = @(w)LogisticLoss(w,X,y);
for mult = 1-lambdaInc:-lambdaInc:lambdaInc
lambda = [0;mult*lambdaMax*ones(nVars,1)];
w(:,end+1) = L1GeneralProjection(funObj,w(:,end),lambda,options);
end
figure(fig);fig=fig+1;
prettyPlot(1:-lambdaInc:lambdaInc,w',{'Bias'},'Regression Coefficients vs. L1-Regularization Strength (L1-Regularized Logistic Regression)','percent of lambdaMax','coefficient values',0,-1);
pause;
Computing Logistic Regression L1-Regularization path
Probit Regression and Smooth SVM
nInstances = 500;
nVars = 100;
X = randn(nInstances,nVars);
y = sign(X*((rand(nVars,1) > .5).*randn(nVars,1)) + randn(nInstances,1)/5);
X = [ones(nInstances,1) X];
lambda = [0;10*ones(nVars,1)];
fprintf('\nComputing Logistic Regression Coefficients...\n');
funObj = @(w)LogisticLoss(w,X,y);
wLogit = L1GeneralProjection(funObj,zeros(nVars+1,1),lambda);
fprintf('\nComputing Probit Regression Coefficients...\n');
funObj = @(w)ProbitLoss(w,X,y);
wProbit = L1GeneralProjection(funObj,zeros(nVars+1,1),lambda);
fprintf('\nComputing Smooth Support Vector Machine Coefficients...\n');
funObj = @(w)SSVMLoss(w,X,y);
wSSVM = L1GeneralProjection(funObj,zeros(nVars+1,1),lambda);
figure(fig);fig=fig+1;
clf;hold on;
subplot(2,2,1);
stem(wLogit,'r');
xlim([1 nVars]);
title('L1-Regularized Logistic Regression');
subplot(2,2,2);
stem(wProbit,'b');
xlim([1 nVars]);
title('L1-Regularized Probit Regression');
subplot(2,2,3);
stem(wSSVM,'g');
xlim([1 nVars]);
title('L1-Regularized Smooth SVM');
subplot(2,2,4);
stem(max(0,1-y.*(X*wSSVM)),'c');
xlim([1 nInstances]);
title('Hinge Loss for L1-Regularized SSVM');
fprintf('Number of non-zero variables for Logistic Regression: %d out of %d\n',nnz(wLogit(2:end)),nVars);
fprintf('Number of non-zero variables for Probit Regression: %d out of %d\n',nnz(wProbit(2:end)),nVars);
fprintf('Number of non-zero variables for Smooth Support Vector Machine: %d out of %d\n',nnz(wSSVM(2:end)),nVars);
fprintf('(%d support vectors)\n',sum(1-y.*(X*wSSVM)>=0));
pause;
Computing Logistic Regression Coefficients...
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 2.38261e+002 1.87373e+002 41
2 3 1.00000e+000 2.27381e+002 3.91114e+001 42
3 4 1.00000e+000 2.26603e+002 3.17923e+000 41
4 5 1.00000e+000 2.26597e+002 1.77292e-001 40
5 6 1.00000e+000 2.26597e+002 1.88117e-005 40
Directional Derivative too small
Computing Probit Regression Coefficients...
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 2.05704e+002 4.43303e+002 49
2 3 1.00000e+000 1.83290e+002 1.14945e+002 49
3 4 1.00000e+000 1.80111e+002 1.93984e+001 51
4 5 1.00000e+000 1.79981e+002 9.61545e-001 51
5 6 1.00000e+000 1.79981e+002 2.75225e-003 51
Directional Derivative too small
Computing Smooth Support Vector Machine Coefficients...
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 1.84180e+002 9.92656e+002 64
2 3 1.00000e+000 1.53622e+002 3.66691e+002 68
3 4 1.00000e+000 1.46121e+002 1.35405e+002 67
4 5 1.00000e+000 1.45175e+002 1.82467e+001 69
5 6 1.00000e+000 1.45165e+002 1.08914e+000 69
6 7 1.00000e+000 1.45165e+002 7.98028e-013 70
Solution Found
Number of non-zero variables for Logistic Regression: 39 out of 100
Number of non-zero variables for Probit Regression: 50 out of 100
Number of non-zero variables for Smooth Support Vector Machine: 69 out of 100
(244 support vectors)
Non-Parametric Logistic Regression with Sparse Prototypes
nInstances = 100;
nVars = 2;
nExamplePoints = 4;
examplePoints = randn(nExamplePoints,nVars);
X = randn(nInstances,nVars);
y = zeros(nInstances,1);
for i = 1:nInstances
dists = sum((repmat(X(i,:),nExamplePoints,1) - examplePoints).^2,2);
[minVal minInd] = min(dists);
y(i,1) = sign(mod(minInd,2)-.5);
end
XX = kernelRBF(X,X,1);
fprintf('Computing Non-Parametric Logistic Regression Coefficients...\n');
funObj = @(u)LogisticLoss(u,XX,y);
lambda = .5*ones(nInstances,1);
u = L1GeneralProjection(funObj,zeros(nInstances,1),lambda);
fprintf('Generating Plot...\n');
increment = 100;
figure(fig);fig=fig+1;
clf; hold on;
plot(X(y==1,1),X(y==1,2),'.','color','g');
plot(X(y==-1,1),X(y==-1,2),'.','color','b');
domainx = xlim;
domain1 = domainx(1):(domainx(2)-domainx(1))/increment:domainx(2);
domainy = ylim;
domain2 = domainy(1):(domainy(2)-domainy(1))/increment:domainy(2);
d1 = repmat(domain1',[1 length(domain1)]);
d2 = repmat(domain2,[length(domain2) 1]);
vals = sign(kernelRBF([d1(:) d2(:)],X,1)*u);
zData = reshape(vals,size(d1));
contourf(d1,d2,zData+rand(size(zData))/1000,[-1 0],'k');
colormap([0 0 .5;0 .5 0]);
plot(X(y==1,1),X(y==1,2),'.','color','g');
plot(X(y==-1,1),X(y==-1,2),'.','color','b');
xlim(domainx);
ylim(domainy);
prototypes = X(u~=0,:);
fprintf('(%d prototypes)\n',size(prototypes,1));
h=plot(prototypes(:,1),prototypes(:,2),'ro');
set(h,'MarkerSize',10);
legend(h,'Non-Zero Data Points (Prototypes)');
pause;
Computing Non-Parametric Logistic Regression Coefficients...
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 6.65244e+001 2.97794e+002 55
2 4 1.33068e-001 5.83662e+001 7.68004e+001 84
3 6 1.46773e-001 5.69147e+001 2.57668e+001 79
4 7 1.00000e+000 5.31851e+001 1.90626e+001 69
5 8 1.00000e+000 5.13678e+001 1.08702e+001 66
6 9 1.00000e+000 5.02830e+001 7.14048e+000 61
7 10 1.00000e+000 4.96056e+001 5.69569e+000 58
8 11 1.00000e+000 4.91226e+001 4.91491e+000 56
9 12 1.00000e+000 4.87476e+001 4.02543e+000 51
10 13 1.00000e+000 4.84567e+001 3.69057e+000 49
11 14 1.00000e+000 4.82197e+001 3.30454e+000 47
12 15 1.00000e+000 4.80126e+001 2.81299e+000 44
13 16 1.00000e+000 4.78469e+001 2.34462e+000 40
14 17 1.00000e+000 4.77053e+001 2.12236e+000 39
15 18 1.00000e+000 4.75869e+001 1.84627e+000 37
16 19 1.00000e+000 4.75035e+001 2.04420e+000 35
17 20 1.00000e+000 4.74351e+001 1.71762e+000 35
18 21 1.00000e+000 4.73679e+001 1.32018e+000 34
19 22 1.00000e+000 4.73055e+001 1.27707e+000 33
20 23 1.00000e+000 4.72530e+001 1.24876e+000 32
21 24 1.00000e+000 4.72067e+001 1.08390e+000 31
22 25 1.00000e+000 4.71605e+001 1.05979e+000 31
23 26 1.00000e+000 4.71170e+001 1.02855e+000 30
24 27 1.00000e+000 4.70721e+001 9.94736e-001 30
25 28 1.00000e+000 4.70282e+001 9.90541e-001 30
26 29 1.00000e+000 4.69855e+001 9.64119e-001 29
27 30 1.00000e+000 4.69390e+001 9.31735e-001 29
28 31 1.00000e+000 4.68931e+001 9.25353e-001 29
29 32 1.00000e+000 4.68477e+001 9.18538e-001 29
30 33 1.00000e+000 4.68040e+001 8.17241e-001 28
31 34 1.00000e+000 4.67789e+001 7.84656e-001 26
32 35 1.00000e+000 4.67614e+001 5.79838e-001 25
33 36 1.00000e+000 4.67439e+001 5.36360e-001 25
34 37 1.00000e+000 4.67279e+001 5.14753e-001 25
35 38 1.00000e+000 4.67127e+001 4.96863e-001 25
36 39 1.00000e+000 4.66982e+001 4.81290e-001 25
37 40 1.00000e+000 4.66846e+001 4.40809e-001 24
38 41 1.00000e+000 4.66714e+001 4.25819e-001 24
39 42 1.00000e+000 4.66586e+001 4.15230e-001 24
40 43 1.00000e+000 4.66460e+001 4.05906e-001 24
41 44 1.00000e+000 4.66336e+001 3.98828e-001 24
42 45 1.00000e+000 4.66216e+001 3.55073e-001 23
43 46 1.00000e+000 4.66109e+001 3.48598e-001 23
44 47 1.00000e+000 4.66003e+001 3.46464e-001 23
45 48 1.00000e+000 4.65924e+001 3.76186e-001 22
46 49 1.00000e+000 4.65859e+001 3.03385e-001 22
47 50 1.00000e+000 4.65797e+001 2.96325e-001 22
48 51 1.00000e+000 4.65737e+001 2.95264e-001 22
49 52 1.00000e+000 4.65679e+001 2.94351e-001 22
50 53 1.00000e+000 4.65621e+001 2.93426e-001 22
51 54 1.00000e+000 4.65564e+001 2.92505e-001 22
52 55 1.00000e+000 4.65508e+001 2.91607e-001 22
53 56 1.00000e+000 4.65453e+001 2.67256e-001 21
54 57 1.00000e+000 4.65402e+001 2.67951e-001 21
55 58 1.00000e+000 4.65351e+001 2.67618e-001 21
56 59 1.00000e+000 4.65300e+001 2.66996e-001 21
57 60 1.00000e+000 4.65250e+001 2.66316e-001 21
58 61 1.00000e+000 4.65200e+001 2.65653e-001 21
59 62 1.00000e+000 4.65150e+001 2.65028e-001 21
60 63 1.00000e+000 4.65100e+001 2.64445e-001 21
61 64 1.00000e+000 4.65050e+001 2.63900e-001 21
62 65 1.00000e+000 4.65000e+001 2.63389e-001 21
63 66 1.00000e+000 4.64951e+001 2.62907e-001 21
64 67 1.00000e+000 4.64902e+001 2.62451e-001 21
65 68 1.00000e+000 4.64852e+001 2.62069e-001 21
66 69 1.00000e+000 4.64803e+001 2.61903e-001 21
67 70 1.00000e+000 4.64754e+001 2.61700e-001 21
68 71 1.00000e+000 4.64705e+001 2.61484e-001 21
69 72 1.00000e+000 4.64656e+001 2.61260e-001 21
70 73 1.00000e+000 4.64607e+001 2.61031e-001 21
71 74 1.00000e+000 4.64558e+001 2.60796e-001 21
72 75 1.00000e+000 4.64510e+001 2.60558e-001 21
73 76 1.00000e+000 4.64461e+001 2.60317e-001 21
74 77 1.00000e+000 4.64413e+001 2.60074e-001 21
75 78 1.00000e+000 4.64364e+001 2.59829e-001 21
76 79 1.00000e+000 4.64316e+001 2.59585e-001 21
77 80 1.00000e+000 4.64267e+001 2.59299e-001 21
78 81 1.00000e+000 4.64222e+001 2.25148e-001 20
79 82 1.00000e+000 4.64178e+001 2.32579e-001 20
80 83 1.00000e+000 4.64135e+001 2.33496e-001 20
81 84 1.00000e+000 4.64092e+001 2.33225e-001 20
82 85 1.00000e+000 4.64049e+001 2.32743e-001 20
83 86 1.00000e+000 4.64006e+001 2.32218e-001 20
84 87 1.00000e+000 4.63963e+001 2.31685e-001 20
85 88 1.00000e+000 4.63920e+001 2.31154e-001 20
86 89 1.00000e+000 4.63878e+001 2.30627e-001 20
87 90 1.00000e+000 4.63835e+001 2.30108e-001 20
88 91 1.00000e+000 4.63793e+001 2.29596e-001 20
89 92 1.00000e+000 4.63750e+001 2.29093e-001 20
90 93 1.00000e+000 4.63708e+001 2.28599e-001 20
91 94 1.00000e+000 4.63666e+001 2.28113e-001 20
92 95 1.00000e+000 4.63623e+001 2.27637e-001 20
93 96 1.00000e+000 4.63581e+001 2.27170e-001 20
94 97 1.00000e+000 4.63539e+001 2.26712e-001 20
95 98 1.00000e+000 4.63497e+001 2.26263e-001 20
96 99 1.00000e+000 4.63456e+001 2.17626e-001 19
97 100 1.00000e+000 4.63415e+001 2.18504e-001 19
98 101 1.00000e+000 4.63373e+001 2.18556e-001 19
99 102 1.00000e+000 4.63333e+001 1.95286e-001 18
100 103 1.00000e+000 4.63290e+001 2.01684e-001 18
101 104 1.00000e+000 4.63247e+001 2.04453e-001 18
102 105 1.00000e+000 4.63210e+001 2.06399e-001 17
103 106 1.00000e+000 4.63175e+001 1.87822e-001 17
104 107 1.00000e+000 4.63140e+001 1.83219e-001 17
105 108 1.00000e+000 4.63106e+001 1.81308e-001 17
106 109 1.00000e+000 4.63072e+001 1.80044e-001 17
107 110 1.00000e+000 4.63038e+001 1.79017e-001 17
108 111 1.00000e+000 4.63004e+001 1.78150e-001 17
109 112 1.00000e+000 4.62971e+001 1.77651e-001 17
110 113 1.00000e+000 4.62938e+001 1.77201e-001 17
111 114 1.00000e+000 4.62905e+001 1.76781e-001 17
112 115 1.00000e+000 4.62872e+001 1.76380e-001 17
113 116 1.00000e+000 4.62840e+001 1.75989e-001 17
114 117 1.00000e+000 4.62808e+001 1.56616e-001 16
115 118 1.00000e+000 4.62775e+001 1.53530e-001 16
116 119 1.00000e+000 4.62743e+001 1.52523e-001 16
117 120 1.00000e+000 4.62711e+001 1.52258e-001 16
118 121 1.00000e+000 4.62680e+001 1.52024e-001 16
119 122 1.00000e+000 4.62648e+001 1.51799e-001 16
120 123 1.00000e+000 4.62616e+001 1.51573e-001 16
121 124 1.00000e+000 4.62585e+001 1.51345e-001 16
122 125 1.00000e+000 4.62553e+001 1.51112e-001 16
123 126 1.00000e+000 4.62522e+001 1.50877e-001 16
124 127 1.00000e+000 4.62491e+001 1.50641e-001 16
125 128 1.00000e+000 4.62459e+001 1.50404e-001 16
126 129 1.00000e+000 4.62428e+001 1.50167e-001 16
127 130 1.00000e+000 4.62397e+001 1.49932e-001 16
128 131 1.00000e+000 4.62366e+001 1.49700e-001 16
129 132 1.00000e+000 4.62335e+001 1.49470e-001 16
130 133 1.00000e+000 4.62304e+001 1.49243e-001 16
131 134 1.00000e+000 4.62274e+001 1.49020e-001 16
132 135 1.00000e+000 4.62243e+001 1.48801e-001 16
133 136 1.00000e+000 4.62212e+001 1.48586e-001 16
134 137 1.00000e+000 4.62183e+001 1.30687e-001 15
135 140 1.00000e-006 4.62182e+001 1.30687e-001 15
136 143 1.00000e-006 4.62181e+001 1.30687e-001 15
137 146 1.00000e-006 4.62180e+001 1.30687e-001 15
138 149 1.00000e-006 4.62179e+001 1.30686e-001 15
139 152 1.00000e-006 4.62178e+001 1.30686e-001 15
140 155 1.00000e-006 4.62177e+001 1.30686e-001 15
141 158 1.00000e-006 4.62176e+001 1.30686e-001 15
142 161 1.00000e-006 4.62175e+001 1.30686e-001 15
143 164 1.00000e-006 4.62174e+001 1.30686e-001 15
144 167 1.00000e-006 4.62173e+001 1.30686e-001 15
145 170 1.00000e-006 4.62172e+001 1.30686e-001 15
146 173 1.00000e-006 4.62171e+001 1.30685e-001 15
147 176 1.00000e-006 4.62170e+001 1.30685e-001 15
148 179 1.00000e-006 4.62169e+001 1.30685e-001 15
149 182 1.00000e-006 4.62169e+001 1.16686e-001 14
150 184 1.00000e-003 4.62070e+001 1.16569e-001 14
151 186 1.00000e-003 4.61976e+001 7.82667e-002 13
152 189 9.78674e-006 4.61976e+001 7.82660e-002 13
153 192 9.30703e-006 4.61976e+001 7.82652e-002 13
154 195 8.88313e-006 4.61975e+001 7.82645e-002 13
155 198 8.50524e-006 4.61975e+001 7.82639e-002 13
156 201 8.16579e-006 4.61975e+001 7.82632e-002 13
157 204 7.85884e-006 4.61974e+001 7.82626e-002 13
158 207 7.57964e-006 4.61974e+001 7.82620e-002 13
159 210 7.32436e-006 4.61974e+001 7.82615e-002 13
160 213 7.08986e-006 4.61974e+001 7.82609e-002 13
161 216 6.87354e-006 4.61973e+001 7.82604e-002 13
162 219 6.67323e-006 4.61973e+001 7.82598e-002 13
163 222 6.48710e-006 4.61973e+001 7.82593e-002 13
164 225 6.31360e-006 4.61973e+001 7.82588e-002 13
165 228 6.15142e-006 4.61972e+001 7.82584e-002 13
166 231 5.99940e-006 4.61972e+001 7.82579e-002 13
167 234 5.85655e-006 4.61972e+001 7.82574e-002 13
168 237 5.72203e-006 4.61972e+001 7.82570e-002 13
169 240 5.59506e-006 4.61972e+001 7.82565e-002 13
170 243 5.47501e-006 4.61971e+001 7.82561e-002 13
171 246 5.36127e-006 4.61971e+001 7.82557e-002 13
172 249 5.25333e-006 4.61971e+001 7.82553e-002 13
173 252 5.15073e-006 4.61971e+001 7.82549e-002 13
Generating Plot...
(13 prototypes)
Multinomial Logistic Regression
nInstances = 200;
nVars = 10;
nClasses = 5;
X = [ones(nInstances,1) randn(nInstances,nVars-1)];
w = randn(nVars,nClasses-1).*(rand(nVars,nClasses-1)>.5);
[junk y] = max(X*[w zeros(nVars,1)],[],2);
w_init = zeros(nVars,nClasses-1);
w_init = w_init(:);
funObj = @(w)SoftmaxLoss2(w,X,y,nClasses);
lambda = 1*ones(nVars,nClasses-1);
lambda(1,:) = 0;
lambda = lambda(:);
fprintf('\nComputing Maximum Likelihood Multinomial Logistic Regression Coefficients\n');
mfOptions.Method = 'newton';
wMLR_ML = minFunc(funObj,w_init,mfOptions);
wMLR_ML = reshape(wMLR_ML,nVars,nClasses-1);
fprintf('\nComputing L2-Regularized Multinomial Logistic Regression Coefficients...\n');
funObjL2 = @(w)penalizedL2(w,funObj,lambda);
wMLR_L2 = minFunc(funObjL2,w_init,mfOptions);
wMLR_L2 = reshape(wMLR_L2,nVars,nClasses-1);
fprintf('\nComputing L1-Regularized Multinomial Logistic Regression Coefficients...\n');
wMLR_L1 = L1GeneralProjection(funObj,w_init,lambda);
wMLR_L1 = reshape(wMLR_L1,nVars,nClasses-1);
fprintf('\nComputing Elastic-Net Multinomial Logistic Regression Coefficients...\n');
wMLR_L1L2 = L1GeneralProjection(funObjL2,w_init,lambda);
wMLR_L1L2 = reshape(wMLR_L1L2,nVars,nClasses-1);
fprintf('Number of Features Selected by Maximum Likelihood Multinomial Logistic classifier: %d (out of %d)\n',nnz(wMLR_ML(2:end,:)),(nVars-1)*(nClasses-1));
fprintf('Number of Features Selected by L2-regualrized Multinomial Logistic classifier: %d (out of %d)\n',nnz(wMLR_L2(2:end,:)),(nVars-1)*(nClasses-1));
fprintf('Number of Features Selected by L1-regualrized Multinomial Logistic classifier: %d (out of %d)\n',nnz(wMLR_L1(2:end,:)),(nVars-1)*(nClasses-1));
fprintf('Number of Features Selected by Elastic-Net Multinomial Logistic classifier: %d (out of %d)\n',nnz(wMLR_L1L2(2:end,:)),(nVars-1)*(nClasses-1));
figure(fig);fig=fig+1;
clf;hold on;
subplot(2,2,1);
stem(wMLR_ML(:),'r');
title('Maximum Likelihood Multinomial Logistic');
subplot(2,2,2);
stem(wMLR_L2(:),'b');
title('L2-Regularized Multinomial Logistic');
subplot(2,2,3);
stem(wMLR_L1(:),'g');
title('L1-Regularized Multinomial Logistic');
subplot(2,2,4);
stem(wMLR_L1L2(:),'c');
title('Elastic-Net Multinomial Logistic');
[junk yhat] = max(X*[wMLR_L1 zeros(nVars,1)],[],2);
fprintf('Classification error rate on training data for L1-regularied Multinomial Logistic: %.2f\n',sum(y ~= yhat)/length(y));
pause;
Computing Maximum Likelihood Multinomial Logistic Regression Coefficients
Iteration FunEvals Step Length Function Val Opt Cond
1 2 1.00000e+000 1.06342e+002 1.36577e+002
2 3 1.00000e+000 6.08441e+001 6.17547e+001
3 4 1.00000e+000 3.70207e+001 2.58713e+001
4 5 1.00000e+000 2.20104e+001 1.21785e+001
5 6 1.00000e+000 1.24345e+001 6.09354e+000
6 7 1.00000e+000 6.23839e+000 2.84310e+000
7 8 1.00000e+000 2.64782e+000 1.13831e+000
8 9 1.00000e+000 1.02777e+000 4.40040e-001
9 10 1.00000e+000 3.87893e-001 1.76195e-001
10 11 1.00000e+000 1.44865e-001 6.78270e-002
11 12 1.00000e+000 5.38119e-002 2.55016e-002
12 13 1.00000e+000 1.99274e-002 9.48513e-003
13 14 1.00000e+000 7.36567e-003 3.51259e-003
14 15 1.00000e+000 2.71935e-003 1.29824e-003
15 16 1.00000e+000 1.00317e-003 4.79307e-004
16 17 1.00000e+000 3.69859e-004 1.76823e-004
17 18 1.00000e+000 1.36302e-004 6.51886e-005
18 19 1.00000e+000 5.02131e-005 2.40180e-005
19 20 1.00000e+000 1.84932e-005 8.84437e-006
Optimality Condition below TolFun
Computing L2-Regularized Multinomial Logistic Regression Coefficients...
Iteration FunEvals Step Length Function Val Opt Cond
1 2 1.00000e+000 1.19781e+002 1.21324e+002
2 3 1.00000e+000 9.60352e+001 3.35980e+001
3 4 1.00000e+000 9.27307e+001 5.17383e+000
4 5 1.00000e+000 9.26337e+001 1.65478e-001
5 6 1.00000e+000 9.26336e+001 1.78162e-004
Directional Derivative below TolX
Computing L1-Regularized Multinomial Logistic Regression Coefficients...
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 1.24605e+002 1.67365e+002 29
2 3 1.00000e+000 1.09212e+002 9.87535e+001 37
3 4 1.00000e+000 1.00762e+002 6.66051e+001 35
4 5 1.00000e+000 9.50435e+001 5.06619e+001 35
5 6 1.00000e+000 7.71720e+001 3.45132e+001 28
6 7 1.00000e+000 7.57075e+001 2.25620e+001 32
7 8 1.00000e+000 6.99643e+001 6.89722e+000 30
8 9 1.00000e+000 6.93636e+001 1.51431e+001 29
9 10 1.00000e+000 6.89224e+001 8.66161e-001 29
10 11 1.00000e+000 6.89182e+001 1.04492e-002 29
11 12 1.00000e+000 6.89182e+001 5.25248e-003 29
Directional Derivative too small
Computing Elastic-Net Multinomial Logistic Regression Coefficients...
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 1.37238e+002 1.57646e+002 31
2 3 1.00000e+000 1.25495e+002 8.82512e+001 38
3 4 1.00000e+000 1.20237e+002 5.44033e+001 37
4 5 1.00000e+000 1.17370e+002 3.67001e+001 36
5 6 1.00000e+000 1.12869e+002 9.93453e+000 32
6 7 1.00000e+000 1.12622e+002 4.51382e-001 31
7 8 1.00000e+000 1.12621e+002 1.12040e-003 31
Directional Derivative too small
Number of Features Selected by Maximum Likelihood Multinomial Logistic classifier: 36 (out of 36)
Number of Features Selected by L2-regualrized Multinomial Logistic classifier: 36 (out of 36)
Number of Features Selected by L1-regualrized Multinomial Logistic classifier: 25 (out of 36)
Number of Features Selected by Elastic-Net Multinomial Logistic classifier: 27 (out of 36)
Classification error rate on training data for L1-regularied Multinomial Logistic: 0.01
Compressed Sensing
figure(fig);fig=fig+1;
nVars = 256;
nNonZero = 5;
x = zeros(nVars,1);
x(ceil(rand(nNonZero,1)*nVars)) = 1;
subplot(2,2,1);
imshow(reshape(x,[16 16]),[]);
title(sprintf('Original Signal (%d elements, %d nonZero)',nVars,nNonZero));
nMeasurements = 49;
phi = rand(nMeasurements,nVars);
subplot(2,2,2);
imshow(phi,[]);
title('Measurement Matrix');
y = phi*x;
subplot(2,2,3);
imshow(reshape(y,[7 7]),[]);
title(sprintf('Measurements (%d elements)',nMeasurements));
funObj = @(f)GaussianLoss(f,phi,y);
f = zeros(nVars,1);
options = [];
for lambda = 10.^[1:-1:-2]
f = L1GeneralProjection(funObj,f,lambda*ones(nVars,1),options);
residual = norm(phi*f-y)
reconstructionError = norm(f-x)
subplot(2,2,4);
imshow(reshape(f,[16 16]),[]);
title(sprintf('Reconstructed (lambda = %.2f, error = %.2f)',lambda,reconstructionError));
pause;
end
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 6.32969e+001 3.37636e+003 192
2 4 1.88652e-001 5.67908e+001 9.69973e+002 153
3 6 5.41639e-002 5.62888e+001 4.87318e+002 153
4 8 7.16056e-002 5.59121e+001 2.04678e+002 148
5 9 1.00000e+000 5.48125e+001 8.83802e+002 105
6 10 1.00000e+000 5.25594e+001 3.46065e+002 86
7 11 1.00000e+000 5.17248e+001 2.34471e+002 70
8 12 1.00000e+000 5.11952e+001 1.57213e+002 56
9 13 1.00000e+000 5.06777e+001 3.94767e+001 55
10 14 1.00000e+000 5.03550e+001 3.08609e+001 52
11 15 1.00000e+000 5.00851e+001 3.14352e+001 49
12 18 3.75952e-006 5.00845e+001 3.14351e+001 49
13 21 3.70363e-006 5.00839e+001 3.09286e+001 48
14 23 1.00000e-003 5.00333e+001 6.51705e+001 45
15 25 1.00000e-003 5.00203e+001 6.45109e+001 44
16 27 1.00000e-003 5.00072e+001 6.44463e+001 44
17 29 1.00000e-003 4.99942e+001 6.43819e+001 44
18 31 1.00000e-003 4.99811e+001 6.43175e+001 44
19 33 1.00000e-003 4.99681e+001 6.42532e+001 44
20 35 1.00000e-003 4.99551e+001 6.41889e+001 44
21 37 1.00000e-003 4.99422e+001 6.41248e+001 44
22 39 1.00000e-003 4.99293e+001 6.40606e+001 44
23 41 1.00000e-003 4.99164e+001 6.39966e+001 44
24 43 1.00000e-003 4.99035e+001 6.39326e+001 44
25 45 1.00000e-003 4.98906e+001 6.38686e+001 44
26 47 1.00000e-003 4.98778e+001 6.38048e+001 44
27 49 1.00000e-003 4.98650e+001 6.37410e+001 44
28 51 1.00000e-003 4.98530e+001 6.23687e+001 43
29 53 1.00000e-003 4.98416e+001 6.23063e+001 43
30 55 1.00000e-003 4.98303e+001 6.05795e+001 42
31 57 1.14027e-003 4.98186e+001 6.05104e+001 42
32 59 1.13768e-003 4.98069e+001 6.04415e+001 42
33 61 1.13509e-003 4.97953e+001 6.03729e+001 42
34 63 1.13252e-003 4.97837e+001 6.03046e+001 42
35 65 1.12996e-003 4.97722e+001 6.02364e+001 42
36 67 1.12742e-003 4.97608e+001 6.01685e+001 42
37 69 1.12488e-003 4.97494e+001 6.01008e+001 42
38 71 1.12235e-003 4.97380e+001 6.00334e+001 42
39 73 1.11984e-003 4.97267e+001 5.99661e+001 42
40 75 1.11734e-003 4.97155e+001 5.98991e+001 42
41 77 1.11485e-003 4.97043e+001 5.98324e+001 42
42 79 1.11237e-003 4.96931e+001 5.97658e+001 42
43 81 1.10990e-003 4.96820e+001 5.96995e+001 42
44 83 1.10744e-003 4.96714e+001 5.84490e+001 41
45 85 1.25672e-003 4.96600e+001 5.83756e+001 41
46 87 1.25357e-003 4.96486e+001 5.83024e+001 41
47 89 1.25043e-003 4.96381e+001 5.66004e+001 40
48 91 2.76440e-003 4.96169e+001 5.64440e+001 40
49 93 2.74919e-003 4.95960e+001 5.62888e+001 40
50 95 2.73415e-003 4.95753e+001 5.61349e+001 40
51 97 2.71927e-003 4.95548e+001 5.59822e+001 40
52 99 2.70455e-003 4.95353e+001 5.42686e+001 39
53 101 2.54530e-003 4.95182e+001 5.41304e+001 39
54 103 2.53240e-003 4.95036e+001 5.32850e+001 38
55 105 3.99677e-003 4.94803e+001 5.17439e+001 37
56 107 4.53913e-003 4.94541e+001 5.02717e+001 36
57 109 4.66562e-003 4.94274e+001 4.84279e+001 35
58 111 5.62175e-003 4.93960e+001 4.63936e+001 34
59 113 7.17574e-003 4.93572e+001 4.60607e+001 34
60 115 7.07377e-003 4.93195e+001 4.57349e+001 34
61 117 6.97467e-003 4.92829e+001 4.54159e+001 34
62 119 6.87832e-003 4.92473e+001 4.51035e+001 34
63 121 6.78460e-003 4.92126e+001 4.47975e+001 34
64 123 6.69342e-003 4.91789e+001 4.44977e+001 34
65 125 6.60466e-003 4.91461e+001 4.42038e+001 34
66 127 6.51823e-003 4.91147e+001 4.30530e+001 33
67 129 8.17757e-003 4.90789e+001 4.28190e+001 32
68 131 1.00391e-002 4.90346e+001 4.23892e+001 32
69 133 9.84023e-003 4.89944e+001 3.92713e+001 30
70 135 1.33120e-002 4.89434e+001 3.87485e+001 30
71 137 1.29634e-002 4.88951e+001 3.82462e+001 30
72 139 1.26328e-002 4.88492e+001 3.77630e+001 30
73 141 1.23187e-002 4.88078e+001 3.61559e+001 29
74 143 1.68594e-002 4.87526e+001 3.55463e+001 29
75 145 1.63021e-002 4.87053e+001 3.52587e+001 28
76 147 1.95701e-002 4.86484e+001 3.45687e+001 28
77 149 1.88213e-002 4.85970e+001 3.29516e+001 27
78 151 2.00219e-002 4.85593e+001 3.50463e+001 25
79 153 7.63672e-002 4.84192e+001 3.54549e+001 24
80 155 6.54619e-002 4.83061e+001 3.31339e+001 24
81 157 5.74139e-002 4.82190e+001 3.12316e+001 24
82 159 5.11906e-002 4.81550e+001 2.88533e+001 22
83 161 8.48077e-002 4.80571e+001 2.64063e+001 22
84 163 7.14905e-002 4.79875e+001 2.45185e+001 22
85 165 6.19494e-002 4.79352e+001 2.29996e+001 22
86 167 5.47409e-002 4.78944e+001 2.17406e+001 22
87 169 4.90850e-002 4.78616e+001 1.95901e+001 21
88 171 4.78488e-002 4.78338e+001 1.86528e+001 21
89 173 4.35098e-002 4.78107e+001 1.78412e+001 21
90 175 3.99107e-002 4.77914e+001 1.61053e+001 20
91 177 5.76292e-002 4.77688e+001 1.51772e+001 20
92 179 5.13867e-002 4.77526e+001 1.43688e+001 19
93 181 7.52988e-002 4.77322e+001 1.32869e+001 19
94 183 6.48008e-002 4.77177e+001 1.19826e+001 18
95 185 1.22698e-001 4.76960e+001 1.02827e+001 17
96 187 2.62733e-001 4.76704e+001 1.11722e+001 16
97 188 1.00000e+000 4.76376e+001 8.85229e+000 14
98 189 1.00000e+000 4.76307e+001 3.74218e-001 14
99 190 1.00000e+000 4.76238e+001 4.01986e+000 14
100 191 1.00000e+000 4.76224e+001 2.98428e-013 14
Solution Found
residual =
1.5420
reconstructionError =
0.6501
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 6.67405e+000 2.66880e+003 147
2 4 2.62417e-001 5.50568e+000 1.60709e+002 78
3 6 8.38829e-002 5.47517e+000 3.39994e+001 70
4 7 1.00000e+000 5.43512e+000 1.35056e+002 56
5 9 7.59189e-002 5.39396e+000 3.23431e+001 65
6 10 1.00000e+000 5.34967e+000 1.35610e+002 42
7 12 5.15013e-002 5.33525e+000 6.26252e+001 57
8 14 6.87997e-002 5.32548e+000 2.59036e+001 60
9 16 2.48393e-001 5.31049e+000 8.11850e+000 46
10 18 1.00000e-003 5.30947e+000 7.67580e+000 42
11 20 1.24344e-003 5.30848e+000 7.66625e+000 42
12 22 1.24035e-003 5.30750e+000 7.65675e+000 42
13 24 1.23727e-003 5.30652e+000 7.64727e+000 42
14 26 1.23421e-003 5.30554e+000 7.48334e+000 41
15 28 1.26206e-003 5.30455e+000 7.47389e+000 41
16 30 1.25888e-003 5.30357e+000 7.46449e+000 41
17 32 1.25571e-003 5.30258e+000 7.45511e+000 41
18 34 1.25256e-003 5.30161e+000 7.44577e+000 41
19 36 1.24942e-003 5.30063e+000 7.43647e+000 41
20 38 1.24630e-003 5.29967e+000 7.42720e+000 41
21 40 1.24319e-003 5.29871e+000 7.41797e+000 41
22 42 1.24010e-003 5.29775e+000 7.40877e+000 41
23 44 1.23703e-003 5.29680e+000 7.39961e+000 41
24 46 1.23397e-003 5.29585e+000 7.32985e+000 40
25 48 1.84373e-003 5.29449e+000 7.31634e+000 40
26 50 1.83694e-003 5.29315e+000 7.30290e+000 40
27 52 1.83019e-003 5.29181e+000 7.28954e+000 40
28 54 1.82349e-003 5.29047e+000 7.28023e+000 39
29 56 1.81672e-003 5.28912e+000 7.25690e+000 38
30 58 7.45418e-003 5.28408e+000 7.28646e+000 38
31 60 7.33774e-003 5.27928e+000 7.65888e+000 36
32 62 7.11392e-002 5.23697e+000 7.35326e+000 34
33 64 1.24015e-001 5.17561e+000 7.18541e+000 30
34 65 1.00000e+000 5.10232e+000 1.91370e+002 16
35 67 1.90880e-001 4.99405e+000 2.41960e+001 26
36 70 5.18887e-003 4.99116e+000 1.27875e+001 17
37 72 7.29053e-002 4.98157e+000 8.56340e+000 9
38 74 5.57548e-002 4.98135e+000 3.78648e+000 9
39 76 1.24508e-001 4.98117e+000 1.30576e+000 14
40 78 3.43421e-003 4.98110e+000 1.05763e+000 17
41 80 3.40712e-003 4.98102e+000 6.78023e-001 20
42 82 1.22765e-002 4.98089e+000 6.84599e-001 18
43 84 3.41401e-002 4.98065e+000 6.75208e-001 14
44 85 1.00000e+000 4.97820e+000 1.75024e+000 14
45 86 1.00000e+000 4.97793e+000 1.42109e-013 14
Solution Found
residual =
0.1542
reconstructionError =
0.0654
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 5.17093e-001 2.53120e+002 125
2 4 2.55456e-001 5.05198e-001 1.34295e+001 60
3 6 6.60781e-002 5.04969e-001 4.99936e+000 59
4 8 1.16150e-001 5.04796e-001 1.73094e+000 58
5 9 1.00000e+000 5.04022e-001 1.16514e+000 47
6 11 1.00000e-003 5.04009e-001 1.16398e+000 47
7 13 1.00000e-003 5.03997e-001 1.13563e+000 46
8 15 1.00000e-003 5.03986e-001 1.10547e+000 45
9 17 1.40546e-003 5.03973e-001 1.10392e+000 45
10 19 1.40152e-003 5.03959e-001 1.10237e+000 45
11 21 1.39759e-003 5.03945e-001 1.05069e+000 43
12 23 1.77510e-003 5.03929e-001 1.04883e+000 43
13 25 1.76880e-003 5.03912e-001 1.04697e+000 43
14 27 1.76255e-003 5.03896e-001 1.04513e+000 43
15 29 1.75635e-003 5.03879e-001 1.04329e+000 43
16 31 1.75018e-003 5.03863e-001 1.04147e+000 43
17 33 1.74407e-003 5.03847e-001 1.03965e+000 43
18 35 1.73799e-003 5.03831e-001 9.78965e-001 39
19 37 1.81208e-002 5.03684e-001 9.48224e-001 37
20 39 9.01010e-002 5.03029e-001 8.94912e-001 32
21 41 2.96753e-001 5.01477e-001 1.17446e+000 26
22 42 1.00000e+000 5.00690e-001 4.79431e+000 15
23 44 1.97877e-001 4.99945e-001 7.82301e-001 13
24 46 9.06503e-002 4.99903e-001 2.35445e-001 8
25 47 1.00000e+000 4.99873e-001 2.78799e-002 8
26 49 1.00372e-001 4.99871e-001 3.05133e-002 10
27 50 1.00000e+000 4.99867e-001 3.10119e-003 11
Directional Derivative too small
residual =
0.0154
reconstructionError =
0.0065
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 1.00000e+000 5.00708e-002 3.10617e+000 15
2 4 4.84366e-002 5.00664e-002 1.55617e+000 15
3 6 4.91144e-002 5.00639e-002 6.11083e-001 14
4 8 8.94352e-002 5.00617e-002 1.86774e-001 13
5 9 1.00000e+000 5.00556e-002 7.92249e-001 13
6 11 1.65491e-001 5.00521e-002 2.60883e-001 17
7 12 1.00000e+000 5.00441e-002 7.62646e-002 24
8 14 1.59240e-002 5.00428e-002 7.50502e-002 24
9 16 1.54196e-002 5.00415e-002 7.38929e-002 22
10 18 1.49466e-002 5.00404e-002 7.27885e-002 20
11 20 1.45021e-002 5.00393e-002 7.17329e-002 19
12 22 1.40835e-002 5.00382e-002 7.07226e-002 19
13 24 1.36887e-002 5.00372e-002 6.97545e-002 18
Change in Objective below optTol
residual =
0.0066
reconstructionError =
0.0053
Chain-structured conditional random field
load wordData.mat
lambda = 1;
options.order = -1;
[w,v_start,v_end,v] = crfChain_initWeights(nFeatures,nStates,'zeros');
featureStart = cumsum([1 nFeatures(1:end)]);
sentences = crfChain_initSentences(y);
nSentences = size(sentences,1);
maxSentenceLength = 1+max(sentences(:,2)-sentences(:,1));
fprintf('Training chain-structured CRF\n');
wv = [w(:);v_start;v_end;v(:)];
wv = L1GeneralProjection(@crfChain_loss,wv,lambda*ones(size(wv)),options,X,y,nStates,nFeatures,featureStart,sentences);
[w,v_start,v_end,v] = crfChain_splitWeights(wv,featureStart,nStates);
trainErr = 0;
trainZ = 0;
yhat = zeros(size(y));
for s = 1:nSentences
y_s = y(sentences(s,1):sentences(s,2));
[nodePot,edgePot]=crfChain_makePotentials(X,w,v_start,v_end,v,nFeatures,featureStart,sentences,s);
[nodeBel,edgeBel,logZ] = crfChain_infer(nodePot,edgePot);
[junk yhat(sentences(s,1):sentences(s,2))] = max(nodeBel,[],2);
end
trainErrRate = sum(y~=yhat)/length(y)
figure(fig);fig=fig+1;
imagesc([y yhat]);
colormap gray
title('True sequence (left), sequence of marginally most likely states (right)');
figure(fig);fig=fig+1;
subplot(1,2,1);
imagesc(w);
title('CRF Feature Potentials');
subplot(1,2,2);
imagesc([v_start' 0;v v_end]);
title('CRF Transition Potentials');
pause;
Training chain-structured CRF
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 2.33945e-004 1.16381e+003 1.98402e+003 78
2 3 1.00000e+000 7.20044e+002 1.36342e+003 74
3 4 1.00000e+000 6.08935e+002 8.18662e+002 80
4 5 1.00000e+000 5.34749e+002 5.60844e+002 80
5 6 1.00000e+000 5.25309e+002 1.05327e+003 80
6 7 1.00000e+000 4.94016e+002 3.00315e+002 80
7 8 1.00000e+000 4.90192e+002 1.65073e+002 80
8 9 1.00000e+000 4.87184e+002 1.72200e+002 80
9 10 1.00000e+000 4.83194e+002 1.92707e+002 80
10 11 1.00000e+000 4.78099e+002 1.99136e+002 80
11 12 1.00000e+000 4.72206e+002 1.31610e+002 78
12 13 1.00000e+000 4.70295e+002 9.35521e+001 79
13 14 1.00000e+000 4.68486e+002 1.08810e+002 72
14 15 1.00000e+000 4.66622e+002 7.00045e+001 70
15 16 1.00000e+000 4.65089e+002 7.28681e+001 64
16 17 1.00000e+000 4.64204e+002 6.69368e+001 62
17 18 1.00000e+000 4.63494e+002 4.90332e+001 67
18 19 1.00000e+000 4.62041e+002 5.33944e+001 65
19 20 1.00000e+000 4.61455e+002 4.30296e+001 63
20 22 3.12085e-001 4.61361e+002 3.80585e+001 66
21 23 1.00000e+000 4.60997e+002 2.47402e+001 65
22 24 1.00000e+000 4.60804e+002 2.65894e+001 67
23 25 1.00000e+000 4.60389e+002 2.84687e+001 66
24 26 1.00000e+000 4.60071e+002 3.01407e+001 63
25 27 1.00000e+000 4.59832e+002 2.48094e+001 62
26 28 1.00000e+000 4.59552e+002 2.48566e+001 61
27 29 1.00000e+000 4.59502e+002 2.73500e+001 62
28 30 1.00000e+000 4.59318e+002 2.55255e+001 59
29 31 1.00000e+000 4.59172e+002 1.42763e+001 59
30 32 1.00000e+000 4.59075e+002 1.00320e+001 59
31 33 1.00000e+000 4.59027e+002 9.27870e+000 59
32 34 1.00000e+000 4.58997e+002 1.47076e+001 58
33 35 1.00000e+000 4.58950e+002 8.21100e+000 58
34 36 1.00000e+000 4.58931e+002 5.58002e+000 58
35 37 1.00000e+000 4.58909e+002 4.26716e+000 57
36 38 1.00000e+000 4.58901e+002 4.20469e+000 57
37 39 1.00000e+000 4.58885e+002 3.79034e+000 57
38 40 1.00000e+000 4.58873e+002 3.41395e+000 58
39 41 1.00000e+000 4.58857e+002 3.71607e+000 56
40 42 1.00000e+000 4.58847e+002 3.37825e+000 57
41 43 1.00000e+000 4.58838e+002 2.86222e+000 57
42 44 1.00000e+000 4.58829e+002 1.95657e+000 57
43 45 1.00000e+000 4.58824e+002 1.62453e+000 57
44 46 1.00000e+000 4.58820e+002 1.50561e+000 57
45 47 1.00000e+000 4.58817e+002 1.22613e+000 58
46 48 1.00000e+000 4.58816e+002 8.16550e-001 58
47 49 1.00000e+000 4.58815e+002 5.33149e-001 58
48 50 1.00000e+000 4.58814e+002 4.50694e-001 58
49 51 1.00000e+000 4.58814e+002 3.97548e-001 59
50 52 1.00000e+000 4.58814e+002 3.00911e-001 58
51 53 1.00000e+000 4.58814e+002 1.89587e-001 58
52 54 1.00000e+000 4.58814e+002 1.40255e-001 58
53 55 1.00000e+000 4.58814e+002 1.44677e-001 58
54 56 1.00000e+000 4.58814e+002 1.22503e-001 58
55 57 1.00000e+000 4.58814e+002 1.03012e-001 58
56 58 1.00000e+000 4.58813e+002 8.24169e-002 58
57 59 1.00000e+000 4.58813e+002 6.36173e-002 58
58 60 1.00000e+000 4.58813e+002 5.58530e-002 57
59 61 1.00000e+000 4.58813e+002 5.68639e-002 57
60 62 1.00000e+000 4.58813e+002 3.89743e-002 57
61 63 1.00000e+000 4.58813e+002 3.24658e-002 57
62 64 1.00000e+000 4.58813e+002 2.78042e-002 57
63 65 1.00000e+000 4.58813e+002 2.08939e-002 57
64 66 1.00000e+000 4.58813e+002 1.39595e-002 57
Change in Objective below optTol
trainErrRate =
0.1030
Graphical Lasso
lambda = .05;
nNodes = 10;
adj = triu(rand(nNodes) > .75,1);
adj = setdiag(adj+adj',1);
P = randn(nNodes).*adj;
P = (P+P')/2;
tau = 1;
X = P + tau*eye(nNodes);
while ~ispd(X)
tau = tau*2;
X = P + tau*eye(nNodes);
end
mu = randn(nNodes,1);
C = inv(X);
R = chol(C)';
X = zeros(nInstances,nNodes);
for i = 1:nInstances
X(i,:) = (mu + R*randn(nNodes,1))';
end
X = standardizeCols(X);
sigma_emp = cov(X);
nonZero = find(ones(nNodes));
funObj = @(x)sparsePrecisionObj(x,nNodes,nonZero,sigma_emp);
Kfull = eye(nNodes);
fprintf('Fitting full Gaussian graphical model\n');
Kfull(nonZero) = minFunc(funObj,Kfull(nonZero),options);
funObj = @(x)sparsePrecisionObj(x,nNodes,nonZero,sigma_emp);
Ksparse = eye(nNodes);
fprintf('Fitting sparse Gaussian graphical model\n');
Ksparse(nonZero) = L1GeneralProjection(funObj,Ksparse(nonZero),lambda*ones(nNodes*nNodes,1),options);
figure(fig);fig=fig+1;
subplot(1,2,1);
imagesc(Kfull);
title('Estimated Full Precision Matrix');
subplot(1,2,2);
imagesc(Ksparse);
title('Estimated Sparse Precision Matrix');
pause;
Fitting full Gaussian graphical model
Iteration FunEvals Step Length Function Val Opt Cond
1 2 1.45585e-001 9.85485e+000 5.86310e+000
2 3 1.00000e+000 9.53402e+000 4.82722e+000
3 4 1.00000e+000 9.46806e+000 1.39930e+000
4 5 1.00000e+000 9.45418e+000 9.48458e-001
5 6 1.00000e+000 9.44120e+000 3.37560e-001
6 7 1.00000e+000 9.44009e+000 2.47006e-001
7 8 1.00000e+000 9.43969e+000 6.67221e-002
8 9 1.00000e+000 9.43963e+000 3.37366e-002
9 10 1.00000e+000 9.43962e+000 1.89929e-002
10 11 1.00000e+000 9.43962e+000 1.71478e-002
Function Value changing by less than TolFun
Fitting sparse Gaussian graphical model
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 5.90310e-002 1.04653e+001 3.73156e+000 52
2 3 1.00000e+000 1.02538e+001 2.03923e+000 54
3 4 1.00000e+000 1.02315e+001 6.81129e-001 52
4 5 1.00000e+000 1.02249e+001 3.81131e-001 50
5 6 1.00000e+000 1.02206e+001 2.79254e-001 50
6 7 1.00000e+000 1.02184e+001 1.36684e-001 50
7 8 1.00000e+000 1.02181e+001 3.63234e-002 50
8 9 1.00000e+000 1.02181e+001 1.80400e-002 50
9 10 1.00000e+000 1.02181e+001 1.05722e-002 50
10 11 1.00000e+000 1.02181e+001 2.88609e-003 50
Directional Derivative too small
Markov Random Field Structure Learning
lambda = 5;
nInstances = 500;
nNodes = 10;
edgeDensity = .5;
nStates = 2;
ising = 1;
tied = 0;
useMex = 1;
y = UGM_generate(nInstances,0,nNodes,edgeDensity,nStates,ising,tied);
adjInit = fixed_Full(nNodes);
edgeStruct = UGM_makeEdgeStruct(adjInit,nStates,useMex);
Xnode = ones(nInstances,1,nNodes);
Xedge = ones(nInstances,1,edgeStruct.nEdges);
infoStruct = UGM_makeMRFInfoStruct(edgeStruct,ising,tied);
[nodeWeights,edgeWeights] = UGM_initWeights(infoStruct,@zeros);
weights_init = [nodeWeights(:);edgeWeights(:)];
nodePenalty = zeros(size(nodeWeights));
edgePenalty = ones(size(edgeWeights));
regularizer = lambda*[nodePenalty(:);edgePenalty(:)];
inferFunc = @UGM_Infer_Exact;
options.order = -1;
funObj = @(weights)UGM_MRFLoss(weights,y,edgeStruct,infoStruct,inferFunc);
[weights] = L1GeneralProjection(funObj,weights_init,regularizer,options);
[nodeWeights,edgeWeights] = UGM_splitWeights(weights,infoStruct);
adjFinal = zeros(nNodes);
for e = 1:edgeStruct.nEdges
if edgeWeights(:,:,e) ~= 0
n1 = edgeStruct.edgeEnds(e,1);
n2 = edgeStruct.edgeEnds(e,2);
adjFinal(n1,n2) = 1;
adjFinal(n2,n1) = 1;
end
end
figure(fig);fig=fig+1;
drawGraph(adjFinal);
title('Estimated Structure');
pause
Exact Sampling
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 9.38086e-004 3.46202e+003 2.71832e+002 36
2 3 1.00000e+000 3.44897e+003 3.00925e+001 39
3 4 1.00000e+000 3.44889e+003 1.13428e+001 39
4 5 1.00000e+000 3.44887e+003 3.75467e+000 36
5 6 1.00000e+000 3.44887e+003 1.23886e+000 36
6 7 1.00000e+000 3.44887e+003 2.90550e-001 36
7 8 1.00000e+000 3.44887e+003 1.32574e-001 36
8 9 1.00000e+000 3.44887e+003 2.82932e-002 36
Directional Derivative too small
neato version 1.8.10 (11/26/02)
Neural Network with Sparse Connections
fig = 100;
lambda = 1;
options.maxIter = 100;
options.adjustStep = 2;
options.order = -1;
options.corrections = 10;
nInstances = 200;
nVars = 1;
[X,y] = makeData('regressionNonlinear',nInstances,nVars);
X = [ones(nInstances,1) X];
nVars = nVars+1;
nHidden = [9 9];
nParams = nVars*nHidden(1);
for h = 2:length(nHidden);
nParams = nParams+nHidden(h-1)*nHidden(h);
end
nParams = nParams+nHidden(end);
funObj = @(weights)MLPregressionLoss_efficient(weights,X,y,nHidden);
fprintf('Training neural network for regression...\n');
lambdaL2 = 1e-3;
wMLP = randn(nParams,1);
while 1
w_old = wMLP;
wMLP = L1GeneralProjection(@penalizedL2,wMLP,lambda*ones(nParams,1),options,funObj,lambdaL2);
if norm(w_old-wMLP,inf) < 1e-4
break;
end
end
figure(fig);hold on;fig=fig+1;
Xtest = [-5:.05:5]';
Xtest = [ones(size(Xtest,1),1) Xtest];
yhat = MLPregressionPredict_efficient(wMLP,Xtest,nHidden);
plot(X(:,2),y,'.');
h=plot(Xtest(:,2),yhat,'g-');
set(h,'LineWidth',3);
legend({'Data','Neural Net'});
inputWeights = reshape(wMLP(1:nVars*nHidden(1)),nVars,nHidden(1));
offset = nVars*nHidden(1);
for h = 2:length(nHidden)
hiddenWeights{h-1} = reshape(wMLP(offset+1:offset+nHidden(h-1)*nHidden(h)),nHidden(h-1),nHidden(h));
offset = offset+nHidden(h-1)*nHidden(h);
end
outputWeights = wMLP(offset+1:offset+nHidden(end));
adj = zeros(nVars+sum(nHidden)+1);
for i = 1:nVars
for j = 1:nHidden(1)
if abs(inputWeights(i,j)) > 1e-4
adj(i,nVars+j) = 1;
end
end
end
for h = 1:length(nHidden)-1
for i = 1:nHidden(h)
for j = 1:nHidden(h+1)
if abs(hiddenWeights{h}(i,j)) > 1e-4
adj(nVars+sum(nHidden(1:h-1))+i,nVars+sum(nHidden(1:h))+j) = 1;
end
end
end
end
for i = 1:nHidden(end)
if abs(outputWeights(i)) > 1e-4
adj(nVars+sum(nHidden(1:end-1))+i,end) = 1;
end
end
labels = cell(length(adj),1);
for i = 1:nVars
labels{i,1} = sprintf('x_%d',i);
end
for h = 1:length(nHidden)
i = i + 1;
labels{i,1} = sprintf('b_%d',h);
for j = 2:nHidden(h)
i = i + 1;
labels{i,1} = sprintf('h_%d_%d',h,j);
end
end
labels{end,1} = 'y';
figure(fig);fig=fig+1;
drawGraph(adj,'labels',labels);
title('Neural Network');
Training neural network for regression...
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 4.08454e-005 1.66399e+003 1.71232e+004 108
2 3 2.52862e-001 1.00335e+003 1.29615e+004 108
3 4 5.97206e-001 2.03436e+002 3.10219e+003 108
4 5 1.00000e+000 1.48343e+002 1.57656e+003 108
5 6 1.00000e+000 1.29778e+002 1.26715e+003 108
6 7 1.00000e+000 1.17143e+002 1.08173e+003 108
7 8 1.00000e+000 1.04244e+002 7.30853e+002 108
8 9 1.00000e+000 9.50537e+001 7.95011e+002 108
9 10 1.00000e+000 9.24759e+001 3.66325e+002 108
10 11 1.00000e+000 9.19439e+001 2.84383e+002 108
11 12 5.17151e-001 9.10800e+001 2.23816e+002 107
12 13 9.32764e-001 8.99777e+001 2.47885e+002 106
13 14 1.00000e+000 8.92123e+001 2.65738e+002 106
14 15 5.36641e-001 8.78337e+001 2.33402e+002 107
15 16 2.52683e-001 8.53195e+001 3.14874e+002 107
16 18 1.38973e-001 8.49813e+001 3.58974e+002 107
17 19 2.46741e-001 8.43699e+001 3.21151e+002 104
18 20 2.85410e-001 8.33804e+001 2.52978e+002 103
19 21 4.90292e-001 8.19016e+001 2.75304e+002 103
20 22 1.00000e+000 8.05908e+001 5.46381e+002 101
21 23 1.00000e+000 7.91651e+001 3.97523e+002 100
22 24 7.15273e-001 7.81898e+001 5.11782e+002 99
23 25 4.14569e-001 7.64417e+001 3.62020e+002 100
24 26 1.81255e-001 7.34832e+001 5.24595e+002 99
25 27 5.72451e-002 6.85788e+001 3.71559e+002 94
26 30 4.67337e-002 6.70163e+001 6.99421e+002 96
27 31 9.80043e-002 6.56872e+001 7.68464e+002 92
28 32 5.05177e-001 6.33272e+001 6.37148e+002 97
29 34 1.26761e-001 6.24018e+001 4.12499e+002 95
30 35 9.11939e-003 6.06937e+001 4.30920e+002 97
31 36 5.38560e-001 5.74050e+001 4.25871e+002 94
32 37 3.74300e-001 5.21468e+001 3.65631e+002 92
33 39 1.21256e-001 5.08666e+001 8.39182e+002 89
34 41 4.83267e-003 4.81954e+001 1.88034e+002 87
35 43 1.07140e-001 4.76213e+001 1.58935e+002 85
36 44 4.22558e-001 4.66548e+001 2.35714e+002 84
37 45 1.97953e-001 4.56239e+001 5.37933e+002 86
38 46 2.29977e-001 4.44456e+001 5.13380e+002 84
39 48 1.74231e-002 4.43593e+001 4.88886e+002 90
40 49 2.25336e-001 4.41918e+001 4.79604e+002 90
41 50 5.31930e-002 4.38707e+001 4.46179e+002 91
42 51 2.77264e-002 4.31998e+001 4.09090e+002 90
43 52 1.81037e-002 4.19045e+001 3.43385e+002 85
44 54 3.02735e-003 4.17482e+001 3.29268e+002 84
45 55 9.13888e-003 4.14894e+001 2.95899e+002 82
46 56 4.00484e-001 4.11241e+001 4.26944e+002 82
47 57 1.00000e+000 4.08143e+001 2.10435e+002 83
48 58 1.00000e+000 4.04783e+001 2.87600e+002 85
49 59 9.70613e-001 3.98754e+001 2.82166e+002 85
50 61 2.48895e-002 3.96233e+001 5.69654e+002 85
51 62 1.43034e-001 3.91405e+001 5.81576e+002 84
52 63 1.57817e-001 3.84015e+001 7.01776e+002 87
53 64 1.63222e-001 3.73901e+001 6.30279e+002 85
54 65 1.00000e+000 3.64252e+001 4.21443e+002 83
55 66 1.00000e+000 3.55989e+001 2.67763e+002 84
56 67 1.00000e+000 3.51374e+001 3.82437e+002 84
57 68 4.62467e-001 3.44718e+001 2.93623e+002 84
58 70 1.70140e-002 3.44316e+001 2.50598e+002 84
59 71 1.26753e-001 3.43537e+001 1.76858e+002 85
60 72 1.30875e-001 3.42595e+001 2.31240e+002 85
61 73 4.32812e-001 3.40831e+001 1.99399e+002 85
62 74 2.47575e-001 3.37929e+001 2.57576e+002 85
63 75 4.14560e-001 3.35868e+001 3.28637e+002 84
64 76 4.35779e-001 3.34944e+001 4.35433e+002 82
65 77 5.55860e-001 3.33774e+001 2.53332e+002 82
66 78 9.58784e-001 3.32470e+001 1.46054e+002 82
67 79 1.00000e+000 3.30816e+001 1.24965e+002 79
68 81 1.46147e-001 3.30200e+001 1.08513e+002 79
69 83 1.29157e-002 3.30088e+001 1.28700e+002 80
70 84 1.65075e-001 3.29888e+001 1.29266e+002 79
71 85 3.37064e-001 3.29528e+001 1.36387e+002 79
72 86 3.37613e-001 3.28902e+001 1.75095e+002 77
73 87 4.36002e-001 3.28118e+001 2.50447e+002 73
74 88 7.36930e-001 3.26553e+001 2.52217e+002 72
75 89 1.68489e-001 3.23569e+001 2.06094e+002 70
76 90 2.38551e-004 3.21319e+001 4.27290e+002 67
77 91 2.12840e-001 3.17552e+001 1.44851e+002 64
78 93 1.62859e-004 3.17199e+001 9.51875e+001 67
79 95 5.07493e-003 3.17127e+001 1.33888e+002 67
80 96 4.09366e-002 3.16993e+001 1.25562e+002 69
81 97 4.54370e-002 3.16741e+001 1.24499e+002 68
82 98 2.58963e-001 3.16213e+001 7.56610e+001 67
83 100 4.37665e-002 3.16127e+001 8.31989e+001 67
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 3 1.07536e-004 3.06758e+001 7.73049e+001 68
2 4 1.00000e+000 3.06701e+001 7.60698e+001 68
3 5 1.03909e-002 3.06597e+001 7.20003e+001 68
4 6 4.65149e-002 3.06396e+001 8.63388e+001 69
5 7 9.07622e-002 3.06010e+001 1.05639e+002 67
6 8 1.14559e-001 3.05271e+001 1.46932e+002 66
7 9 1.54628e-001 3.03839e+001 1.70512e+002 64
8 10 1.55718e-001 3.01186e+001 2.06197e+002 64
9 11 2.97934e-001 2.96687e+001 1.92771e+002 61
10 12 1.00000e+000 2.94280e+001 4.63163e+002 64
11 13 2.36481e-001 2.89171e+001 2.28541e+002 59
12 15 5.54445e-002 2.88302e+001 1.62964e+002 62
13 16 6.13835e-001 2.87540e+001 2.53371e+002 61
14 17 1.30602e-001 2.87048e+001 3.02964e+002 59
15 18 7.21798e-001 2.86401e+001 1.52993e+002 59
16 19 1.00000e+000 2.85771e+001 1.18399e+002 60
17 20 4.00924e-001 2.84662e+001 1.45841e+002 63
18 21 2.40363e-001 2.82587e+001 2.00038e+002 61
19 22 9.46841e-002 2.78472e+001 2.54639e+002 61
20 23 2.01034e-001 2.70093e+001 2.19070e+002 58
21 24 4.90716e-001 2.63892e+001 5.17019e+002 52
22 26 1.29011e-001 2.61191e+001 2.91807e+002 54
23 27 1.00000e+000 2.58701e+001 1.37808e+002 53
24 28 4.05948e-001 2.57035e+001 3.48978e+002 54
25 29 4.25614e-001 2.55841e+001 4.55805e+002 57
26 30 6.90363e-001 2.53924e+001 2.85384e+002 56
27 31 9.92747e-001 2.51367e+001 1.68748e+002 56
28 32 1.00000e+000 2.48275e+001 1.05994e+002 57
29 33 1.00000e+000 2.45753e+001 3.02538e+002 56
30 34 9.54657e-001 2.45199e+001 2.14651e+002 58
31 35 1.85112e-001 2.43844e+001 1.22127e+002 57
32 36 1.00000e+000 2.43101e+001 1.18560e+002 57
33 38 3.37798e-001 2.42885e+001 1.26249e+002 57
34 39 7.46695e-002 2.42456e+001 1.03387e+002 57
35 40 1.94785e-001 2.41624e+001 9.24802e+001 57
36 41 1.02360e-001 2.40029e+001 8.75036e+001 56
37 42 1.58134e-003 2.37711e+001 1.54836e+002 56
38 43 1.55917e-001 2.33346e+001 1.32115e+002 58
39 44 2.77198e-001 2.26173e+001 2.09178e+002 57
40 45 7.42628e-001 2.15245e+001 3.46250e+002 54
41 46 3.60661e-001 1.98938e+001 1.23137e+002 53
42 48 4.08952e-002 1.97003e+001 1.45951e+002 52
43 49 2.14540e-001 1.93576e+001 2.57516e+002 52
44 50 2.37453e-001 1.87704e+001 3.87315e+002 51
45 51 3.18426e-001 1.87505e+001 5.28775e+002 47
46 52 3.33343e-002 1.87137e+001 5.22436e+002 47
47 53 4.10484e-003 1.86417e+001 5.00320e+002 47
48 54 9.11807e-002 1.85031e+001 4.57745e+002 47
49 55 1.90652e-001 1.82480e+001 3.69677e+002 47
50 56 4.72343e-001 1.78493e+001 1.91246e+002 47
51 57 1.00000e+000 1.76175e+001 6.82726e+001 47
52 58 1.00000e+000 1.75328e+001 8.69867e+001 48
53 59 1.67993e-001 1.73797e+001 1.19063e+002 47
54 60 1.98615e-001 1.71194e+001 1.62904e+002 45
55 61 2.79348e-001 1.66295e+001 2.10557e+002 43
56 62 1.10795e-001 1.57750e+001 1.25429e+002 39
57 64 1.62693e-002 1.54303e+001 9.02005e+001 36
58 65 1.00000e+000 1.53932e+001 1.27413e+002 34
59 66 1.20019e-001 1.53212e+001 1.20340e+002 34
60 67 1.01058e-001 1.51853e+001 1.08741e+002 32
61 68 2.17364e-001 1.49413e+001 9.04620e+001 31
62 69 5.04484e-001 1.45672e+001 5.51266e+001 31
63 70 1.00000e+000 1.43247e+001 5.63406e+001 27
64 72 2.53926e-001 1.43031e+001 4.06079e+001 27
65 73 2.44722e-001 1.42746e+001 3.81260e+001 27
66 75 3.83608e-001 1.42539e+001 3.21800e+001 29
67 76 4.44835e-001 1.42297e+001 6.72063e+001 29
68 77 5.11741e-001 1.41911e+001 5.40609e+001 29
69 78 9.47180e-003 1.41330e+001 7.42053e+001 29
70 79 3.01455e-001 1.40266e+001 3.73137e+001 28
71 80 5.61738e-001 1.38906e+001 5.33092e+001 26
72 81 4.97924e-001 1.37833e+001 7.38791e+001 26
73 82 1.00000e+000 1.37402e+001 4.56364e+001 26
74 83 6.08917e-001 1.36726e+001 3.20825e+001 26
75 84 5.25466e-001 1.35549e+001 2.59510e+001 26
76 85 4.67184e-001 1.34162e+001 5.16278e+001 23
77 86 1.91116e-001 1.33446e+001 3.30587e+001 22
78 87 8.42883e-001 1.33119e+001 5.93610e+001 20
79 88 4.73404e-001 1.32662e+001 3.43860e+001 21
80 89 7.98102e-001 1.32332e+001 6.09486e+001 20
81 91 3.41788e-002 1.32284e+001 5.62429e+001 20
82 92 1.70890e-001 1.32193e+001 4.71092e+001 20
83 93 2.45983e-001 1.32024e+001 3.80574e+001 20
84 94 2.49547e-001 1.31713e+001 2.88845e+001 20
85 95 3.01065e-001 1.31154e+001 1.83619e+001 20
86 96 3.89242e-001 1.30250e+001 2.56584e+001 19
87 97 3.68283e-001 1.28878e+001 6.38066e+001 20
88 98 6.25152e-001 1.26904e+001 7.15051e+001 19
89 99 9.96793e-001 1.23908e+001 6.83905e+001 19
90 101 2.98183e-001 1.21663e+001 1.27093e+002 19
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 3 9.03594e-004 1.17522e+001 6.53132e+001 22
2 4 1.00000e+000 1.16995e+001 4.15744e+001 22
3 5 6.66297e-001 1.16132e+001 3.60660e+001 22
4 6 7.62774e-001 1.14930e+001 4.98337e+001 20
5 7 1.00000e+000 1.14041e+001 4.36747e+001 19
6 8 1.00000e+000 1.13479e+001 3.46269e+001 20
7 9 1.00000e+000 1.13117e+001 1.41730e+001 20
8 10 1.00000e+000 1.12980e+001 2.02746e+001 20
9 11 8.26326e-001 1.12816e+001 3.26751e+001 20
10 12 1.00000e+000 1.12626e+001 3.47297e+001 20
11 13 1.86242e-002 1.12246e+001 3.54299e+001 20
12 14 1.46120e-002 1.11902e+001 5.22158e+001 19
13 15 2.57320e-001 1.11154e+001 3.96377e+001 19
14 16 1.00000e+000 1.10374e+001 2.70724e+001 19
15 17 1.27105e-001 1.08862e+001 1.32041e+001 20
16 18 1.82984e-001 1.06693e+001 6.43909e+001 18
17 19 2.43323e-002 1.06120e+001 1.34191e+002 18
18 20 1.65706e-001 1.04966e+001 1.20558e+002 19
19 21 1.73504e-002 1.02877e+001 9.83791e+001 18
20 22 1.05317e-001 1.00352e+001 6.20218e+001 18
21 23 1.00000e+000 9.96155e+000 3.01799e+001 17
22 24 3.93692e-001 9.83874e+000 2.12657e+001 17
23 25 4.48508e-001 9.61969e+000 2.22099e+001 17
24 26 4.20657e-001 9.35154e+000 6.52605e+001 17
25 27 1.00000e+000 9.27032e+000 3.87604e+001 17
26 28 5.94776e-001 9.16259e+000 3.09195e+001 17
27 29 1.00000e+000 9.11248e+000 3.62042e+001 16
28 30 6.83971e-001 9.06411e+000 1.91525e+001 16
29 31 1.00000e+000 9.03655e+000 1.53773e+001 15
30 32 9.81408e-001 9.02472e+000 2.32680e+001 13
31 33 1.58726e-001 9.00427e+000 2.09866e+001 13
32 34 4.35020e-001 8.97043e+000 1.31044e+001 13
33 35 6.07463e-001 8.91462e+000 1.09445e+001 13
34 36 6.68401e-001 8.88985e+000 3.17047e+001 13
35 37 2.61270e-001 8.84621e+000 1.61115e+001 14
36 38 9.49961e-001 8.79071e+000 6.30979e+000 14
37 39 1.00000e+000 8.77531e+000 5.34204e+000 14
38 40 1.00000e+000 8.75194e+000 5.35294e+000 14
39 41 3.58205e-001 8.72234e+000 8.42289e+000 13
40 42 1.00000e+000 8.67960e+000 1.14441e+001 13
41 43 6.31418e-001 8.60597e+000 1.87559e+001 13
42 45 7.18936e-002 8.59871e+000 1.84286e+001 12
43 46 1.57292e-001 8.58609e+000 1.55558e+001 12
44 47 4.73652e-001 8.56375e+000 9.60160e+000 12
45 48 7.27435e-001 8.53481e+000 4.43424e+000 12
46 49 1.00000e+000 8.52226e+000 1.17542e+001 12
47 51 4.81573e-001 8.51633e+000 3.67885e+000 11
48 52 1.00000e+000 8.51152e+000 7.16176e+000 11
49 53 1.00000e+000 8.50722e+000 6.52305e+000 11
50 54 4.36462e-001 8.50006e+000 5.40309e+000 11
51 55 7.92375e-001 8.49044e+000 4.40335e+000 11
52 56 1.00000e+000 8.48308e+000 6.26498e+000 11
53 57 6.57668e-001 8.47074e+000 8.34989e+000 12
54 58 5.60297e-001 8.45056e+000 1.23004e+001 12
55 59 6.85624e-001 8.42009e+000 1.73051e+001 12
56 60 9.29007e-001 8.37537e+000 2.49483e+001 12
57 61 9.03902e-001 8.30738e+000 2.58755e+001 12
58 62 9.06679e-001 8.23225e+000 2.08733e+001 12
59 63 8.68046e-001 8.15356e+000 3.56009e+001 12
60 65 4.07735e-001 8.12568e+000 1.78993e+001 12
61 66 5.38783e-001 8.07484e+000 1.17190e+001 11
62 68 2.90512e-002 8.06118e+000 3.07318e+001 11
63 69 5.58009e-001 8.03483e+000 2.73159e+001 11
64 70 2.65597e-001 7.98694e+000 2.20684e+001 12
65 71 4.39313e-001 7.90530e+000 1.62930e+001 12
66 72 6.19246e-001 7.80928e+000 1.85278e+001 10
67 74 1.21193e-001 7.78574e+000 7.51937e+000 10
68 75 2.24679e-001 7.74412e+000 1.08510e+001 10
69 76 2.12518e-001 7.66935e+000 1.88511e+001 10
70 79 2.85135e-003 7.66920e+000 1.71830e+001 9
71 80 3.11165e-003 7.66889e+000 1.71256e+001 9
72 81 1.78558e-003 7.66827e+000 1.70848e+001 9
73 82 1.76170e-003 7.66702e+000 1.70234e+001 9
74 83 2.24138e-003 7.66453e+000 1.69218e+001 9
75 84 3.24083e-003 7.65948e+000 1.66898e+001 9
76 85 5.57484e-003 7.64920e+000 1.61559e+001 9
77 86 1.10666e-002 7.62860e+000 1.54240e+001 9
78 87 1.81293e-002 7.58764e+000 1.44022e+001 9
79 88 2.82158e-002 7.50669e+000 1.22874e+001 9
80 89 5.10397e-002 7.35197e+000 7.77795e+000 9
81 91 1.98346e-002 7.34731e+000 7.97539e+000 8
82 92 2.27395e-001 7.33915e+000 6.85242e+000 8
83 93 4.36680e-001 7.32583e+000 6.87428e+000 8
84 94 7.36301e-001 7.30593e+000 6.44983e+000 8
85 95 8.86338e-001 7.27306e+000 6.16532e+000 9
86 96 5.27135e-001 7.21798e+000 5.55159e+000 9
87 97 7.86421e-001 7.14016e+000 1.26454e+001 9
88 98 1.00000e+000 7.08100e+000 2.15033e+001 9
89 99 5.47121e-001 7.05512e+000 2.65243e+001 7
90 100 8.33976e-002 7.00888e+000 1.54164e+001 7
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 3 1.39892e-003 6.98309e+000 1.40446e+001 7
2 4 1.00000e+000 6.96724e+000 1.35566e+001 7
3 5 1.47089e-001 6.93764e+000 1.23876e+001 7
4 6 2.89100e-001 6.89281e+000 9.21934e+000 7
5 7 9.51489e-001 6.84144e+000 5.35762e+000 7
6 8 1.00000e+000 6.82286e+000 5.94187e+000 7
7 9 1.00000e+000 6.81591e+000 5.73728e+000 7
8 10 8.65101e-001 6.80248e+000 5.54801e+000 7
9 11 9.65406e-002 6.77650e+000 5.98975e+000 7
10 12 1.11991e-001 6.72677e+000 8.08649e+000 7
11 13 1.62992e-001 6.63445e+000 1.23136e+001 7
12 14 2.96157e-001 6.56294e+000 2.12972e+001 6
13 15 1.00000e+000 6.53087e+000 5.79239e+000 6
14 16 1.00000e+000 6.51513e+000 4.89303e+000 6
15 17 3.86673e-001 6.48828e+000 5.97784e+000 6
16 18 5.75406e-001 6.44923e+000 1.03135e+001 6
17 19 9.81584e-001 6.40602e+000 9.13449e+000 6
18 20 1.00000e+000 6.40581e+000 1.46947e+001 6
19 21 4.21019e-003 6.40539e+000 1.45964e+001 6
20 22 1.27321e-002 6.40454e+000 1.44280e+001 6
21 23 2.41122e-002 6.40287e+000 1.40912e+001 6
22 24 4.75682e-002 6.39961e+000 1.34407e+001 6
23 25 9.93039e-002 6.39342e+000 1.21298e+001 6
24 26 2.27729e-001 6.38241e+000 9.41237e+000 6
25 27 6.56465e-001 6.36746e+000 3.33953e+000 6
26 28 1.00000e+000 6.36466e+000 7.37002e-001 6
27 29 1.00000e+000 6.36411e+000 8.38322e-001 6
28 30 3.37952e-001 6.36312e+000 8.90398e-001 6
29 31 1.33378e-001 6.36126e+000 7.13079e-001 6
30 32 1.91337e-001 6.35808e+000 9.11820e-001 6
31 33 5.71655e-001 6.35454e+000 2.79281e+000 6
32 34 1.00000e+000 6.35335e+000 1.11353e+000 6
33 35 1.00000e+000 6.35288e+000 6.56511e-001 6
34 36 1.00000e+000 6.35255e+000 1.52031e-001 6
35 37 1.00000e+000 6.35253e+000 6.57991e-002 6
36 38 1.00000e+000 6.35252e+000 1.22323e-001 6
37 39 8.33746e-001 6.35252e+000 4.63133e-002 6
38 40 1.00000e+000 6.35252e+000 1.94276e-002 6
Change in Objective below optTol
Iteration FunEvals Step Length Function Val Opt Cond Non-Zero
1 2 4.62925e-003 6.35252e+000 2.26678e-002 6
Change in Objective below optTol
neato version 1.8.10 (11/26/02)
Deep Neural Network with Sparse Connections
fig = 1000;
lambda = 1;
options.maxIter = 100;
options.adjustStep = 2;
options.order = -1;
options.corrections = 10;
options.verbose = 0;
nInstances = 200;
nVars = 1;
[X,y] = makeData('regressionNonlinear2',nInstances,nVars);
X = [ones(nInstances,1) X];
nVars = nVars+1;
nHidden = [9 9 9 9 9 9 9 9 9];
nParams = nVars*nHidden(1);
for h = 2:length(nHidden);
nParams = nParams+nHidden(h-1)*nHidden(h);
end
nParams = nParams+nHidden(end);
funObj = @(weights)MLPregressionLoss_efficient(weights,X,y,nHidden);
fprintf('Training neural network for regression...\n');
lambdaL2 = 1e-3;
wMLP = randn(nParams,1);
for i = 1:1000
fprintf('Training:');
w_old = wMLP;
wMLP = L1GeneralProjection(@penalizedL2,wMLP,lambda*ones(nParams,1),options,funObj,lambdaL2);
fprintf(' (nnz = %d, max change = %f)\n',nnz(wMLP),norm(w_old-wMLP,inf));
if norm(w_old-wMLP,inf) < 1e-5
break;
end
end
figure(fig);hold on;fig=fig+1;
Xtest = [-5:.05:5]';
Xtest = [ones(size(Xtest,1),1) Xtest];
yhat = MLPregressionPredict_efficient(wMLP,Xtest,nHidden);
plot(X(:,2),y,'.');
h=plot(Xtest(:,2),yhat,'g-');
set(h,'LineWidth',3);
legend({'Data','Deep Neural Net'});
inputWeights = reshape(wMLP(1:nVars*nHidden(1)),nVars,nHidden(1));
offset = nVars*nHidden(1);
for h = 2:length(nHidden)
hiddenWeights{h-1} = reshape(wMLP(offset+1:offset+nHidden(h-1)*nHidden(h)),nHidden(h-1),nHidden(h));
offset = offset+nHidden(h-1)*nHidden(h);
end
outputWeights = wMLP(offset+1:offset+nHidden(end));
adj = zeros(nVars+sum(nHidden)+1);
for i = 1:nVars
for j = 1:nHidden(1)
if abs(inputWeights(i,j)) > 1e-4
adj(i,nVars+j) = 1;
end
end
end
for h = 1:length(nHidden)-1
for i = 1:nHidden(h)
for j = 1:nHidden(h+1)
if abs(hiddenWeights{h}(i,j)) > 1e-4
adj(nVars+sum(nHidden(1:h-1))+i,nVars+sum(nHidden(1:h))+j) = 1;
end
end
end
end
for i = 1:nHidden(end)
if abs(outputWeights(i)) > 1e-4
adj(nVars+sum(nHidden(1:end-1))+i,end) = 1;
end
end
labels = cell(length(adj),1);
for i = 1:nVars
labels{i,1} = sprintf('x_%d',i);
end
for h = 1:length(nHidden)
i = i + 1;
labels{i,1} = sprintf('b_%d',h);
for j = 2:nHidden(h)
i = i + 1;
labels{i,1} = sprintf('h_%d_%d',h,j);
end
end
labels{end,1} = 'y';
figure(fig);fig=fig+1;
drawGraph(adj,'labels',labels);
title('Neural Network');
Training neural network for regression...
Training: (nnz = 674, max change = 0.852624)
Training: (nnz = 672, max change = 0.733786)
Training: (nnz = 669, max change = 0.113227)
Training: (nnz = 667, max change = 0.084462)
Training: (nnz = 666, max change = 0.057579)
Training: (nnz = 664, max change = 0.049427)
Training: (nnz = 661, max change = 0.088832)
Training: (nnz = 662, max change = 0.050118)
Training: (nnz = 657, max change = 0.068962)
Training: (nnz = 654, max change = 0.103723)
Training: (nnz = 652, max change = 0.098402)
Training: (nnz = 650, max change = 0.072713)
Training: (nnz = 649, max change = 0.051067)
Training: (nnz = 645, max change = 0.178320)
Training: (nnz = 645, max change = 0.055595)
Training: (nnz = 641, max change = 0.082746)
Training: (nnz = 640, max change = 0.095307)
Training: (nnz = 638, max change = 0.097814)
Training: (nnz = 638, max change = 0.104997)
Training: (nnz = 634, max change = 0.096488)
Training: (nnz = 639, max change = 0.081371)
Training: (nnz = 630, max change = 0.105067)
Training: (nnz = 626, max change = 0.083035)
Training: (nnz = 629, max change = 0.112420)
Training: (nnz = 623, max change = 0.054118)
Training: (nnz = 623, max change = 0.077413)
Training: (nnz = 618, max change = 0.054774)
Training: (nnz = 620, max change = 0.050815)
Training: (nnz = 614, max change = 0.027933)
Training: (nnz = 625, max change = 0.162325)
Training: (nnz = 620, max change = 0.048202)
Training: (nnz = 611, max change = 0.049195)
Training: (nnz = 606, max change = 0.062228)
Training: (nnz = 606, max change = 0.038837)
Training: (nnz = 602, max change = 0.025493)
Training: (nnz = 596, max change = 0.035176)
Training: (nnz = 597, max change = 0.038042)
Training: (nnz = 596, max change = 0.038937)
Training: (nnz = 600, max change = 0.028449)
Training: (nnz = 597, max change = 0.055782)
Training: (nnz = 600, max change = 0.028711)
Training: (nnz = 597, max change = 0.060109)
Training: (nnz = 597, max change = 0.026089)
Training: (nnz = 591, max change = 0.076818)
Training: (nnz = 587, max change = 0.040271)
Training: (nnz = 581, max change = 0.077309)
Training: (nnz = 580, max change = 0.038111)
Training: (nnz = 577, max change = 0.115166)
Training: (nnz = 575, max change = 0.050735)
Training: (nnz = 568, max change = 0.057974)
Training: (nnz = 567, max change = 0.052128)
Training: (nnz = 562, max change = 0.040689)
Training: (nnz = 563, max change = 0.038760)
Training: (nnz = 554, max change = 0.050392)
Training: (nnz = 564, max change = 0.048297)
Training: (nnz = 556, max change = 0.053398)
Training: (nnz = 556, max change = 0.052321)
Training: (nnz = 548, max change = 0.102344)
Training: (nnz = 545, max change = 0.053279)
Training: (nnz = 542, max change = 0.078188)
Training: (nnz = 542, max change = 0.060599)
Training: (nnz = 530, max change = 0.047609)
Training: (nnz = 532, max change = 0.045560)
Training: (nnz = 539, max change = 0.110536)
Training: (nnz = 529, max change = 0.029767)
Training: (nnz = 525, max change = 0.032484)
Training: (nnz = 521, max change = 0.026707)
Training: (nnz = 513, max change = 0.031644)
Training: (nnz = 516, max change = 0.025712)
Training: (nnz = 517, max change = 0.051039)
Training: (nnz = 517, max change = 0.023640)
Training: (nnz = 504, max change = 0.072083)
Training: (nnz = 509, max change = 0.027579)
Training: (nnz = 492, max change = 0.074874)
Training: (nnz = 492, max change = 0.032212)
Training: (nnz = 496, max change = 0.038962)
Training: (nnz = 478, max change = 0.019628)
Training: (nnz = 477, max change = 0.036758)
Training: (nnz = 478, max change = 0.019326)
Training: (nnz = 478, max change = 0.076023)
Training: (nnz = 478, max change = 0.021171)
Training: (nnz = 464, max change = 0.031861)
Training: (nnz = 463, max change = 0.022064)
Training: (nnz = 461, max change = 0.044318)
Training: (nnz = 451, max change = 0.021028)
Training: (nnz = 445, max change = 0.030780)
Training: (nnz = 435, max change = 0.017932)
Training: (nnz = 435, max change = 0.043201)
Training: (nnz = 432, max change = 0.023933)
Training: (nnz = 427, max change = 0.035083)
Training: (nnz = 432, max change = 0.025096)
Training: (nnz = 427, max change = 0.024891)
Training: (nnz = 417, max change = 0.019443)
Training: (nnz = 427, max change = 0.034074)
Training: (nnz = 417, max change = 0.031912)
Training: (nnz = 414, max change = 0.028549)
Training: (nnz = 408, max change = 0.029243)
Training: (nnz = 407, max change = 0.021303)
Training: (nnz = 394, max change = 0.017433)
Training: (nnz = 398, max change = 0.025931)
Training: (nnz = 404, max change = 0.018039)
Training: (nnz = 400, max change = 0.026320)
Training: (nnz = 391, max change = 0.020066)
Training: (nnz = 385, max change = 0.021791)
Training: (nnz = 386, max change = 0.017118)
Training: (nnz = 390, max change = 0.040676)
Training: (nnz = 398, max change = 0.013432)
Training: (nnz = 393, max change = 0.045648)
Training: (nnz = 386, max change = 0.017121)
Training: (nnz = 392, max change = 0.023984)
Training: (nnz = 381, max change = 0.021884)
Training: (nnz = 377, max change = 0.015260)
Training: (nnz = 375, max change = 0.011128)
Training: (nnz = 381, max change = 0.018601)
Training: (nnz = 374, max change = 0.016093)
Training: (nnz = 265, max change = 0.000096)
Training: (nnz = 383, max change = 0.020544)
Training: (nnz = 382, max change = 0.011710)
Training: (nnz = 376, max change = 0.019077)
Training: (nnz = 384, max change = 0.016089)
Training: (nnz = 371, max change = 0.018400)
Training: (nnz = 377, max change = 0.013654)
Training: (nnz = 369, max change = 0.012276)
Training: (nnz = 361, max change = 0.017523)
Training: (nnz = 363, max change = 0.013958)
Training: (nnz = 370, max change = 0.034601)
Training: (nnz = 372, max change = 0.021576)
Training: (nnz = 354, max change = 0.017333)
Training: (nnz = 359, max change = 0.021075)
Training: (nnz = 354, max change = 0.015693)
Training: (nnz = 354, max change = 0.017387)
Training: (nnz = 363, max change = 0.022056)
Training: (nnz = 360, max change = 0.015068)
Training: (nnz = 352, max change = 0.020701)
Training: (nnz = 350, max change = 0.011955)
Training: (nnz = 358, max change = 0.045111)
Training: (nnz = 343, max change = 0.009718)
Training: (nnz = 360, max change = 0.036942)
Training: (nnz = 250, max change = 0.000096)
Training: (nnz = 354, max change = 0.025623)
Training: (nnz = 352, max change = 0.016378)
Training: (nnz = 344, max change = 0.020825)
Training: (nnz = 323, max change = 0.012811)
Training: (nnz = 327, max change = 0.017319)
Training: (nnz = 331, max change = 0.018284)
Training: (nnz = 326, max change = 0.019329)
Training: (nnz = 333, max change = 0.032580)
Training: (nnz = 218, max change = 0.000082)
Training: (nnz = 336, max change = 0.011392)
Training: (nnz = 322, max change = 0.022577)
Training: (nnz = 318, max change = 0.022012)
Training: (nnz = 315, max change = 0.014922)
Training: (nnz = 316, max change = 0.017028)
Training: (nnz = 317, max change = 0.019373)
Training: (nnz = 305, max change = 0.018385)
Training: (nnz = 321, max change = 0.015137)
Training: (nnz = 303, max change = 0.013903)
Training: (nnz = 305, max change = 0.013153)
Training: (nnz = 311, max change = 0.016180)
Training: (nnz = 308, max change = 0.005912)
Training: (nnz = 320, max change = 0.071235)
Training: (nnz = 320, max change = 0.012887)
Training: (nnz = 309, max change = 0.030543)
Training: (nnz = 303, max change = 0.020426)
Training: (nnz = 296, max change = 0.015518)
Training: (nnz = 294, max change = 0.020771)
Training: (nnz = 293, max change = 0.013933)
Training: (nnz = 285, max change = 0.011330)
Training: (nnz = 285, max change = 0.019366)
Training: (nnz = 284, max change = 0.010768)
Training: (nnz = 271, max change = 0.016416)
Training: (nnz = 277, max change = 0.014074)
Training: (nnz = 287, max change = 0.012413)
Training: (nnz = 181, max change = 0.000099)
Training: (nnz = 273, max change = 0.014298)
Training: (nnz = 270, max change = 0.013969)
Training: (nnz = 272, max change = 0.011162)
Training: (nnz = 274, max change = 0.015193)
Training: (nnz = 270, max change = 0.012508)
Training: (nnz = 271, max change = 0.017530)
Training: (nnz = 262, max change = 0.011797)
Training: (nnz = 161, max change = 0.000100)
Training: (nnz = 270, max change = 0.023663)
Training: (nnz = 263, max change = 0.009626)
Training: (nnz = 261, max change = 0.032543)
Training: (nnz = 264, max change = 0.010716)
Training: (nnz = 265, max change = 0.020293)
Training: (nnz = 170, max change = 0.000097)
Training: (nnz = 269, max change = 0.008514)
Training: (nnz = 263, max change = 0.027545)
Training: (nnz = 256, max change = 0.008161)
Training: (nnz = 259, max change = 0.051472)
Training: (nnz = 254, max change = 0.009542)
Training: (nnz = 242, max change = 0.015740)
Training: (nnz = 260, max change = 0.011242)
Training: (nnz = 236, max change = 0.018000)
Training: (nnz = 238, max change = 0.005891)
Training: (nnz = 248, max change = 0.045964)
Training: (nnz = 247, max change = 0.011811)
Training: (nnz = 248, max change = 0.042366)
Training: (nnz = 243, max change = 0.011197)
Training: (nnz = 228, max change = 0.029193)
Training: (nnz = 232, max change = 0.010828)
Training: (nnz = 228, max change = 0.036810)
Training: (nnz = 216, max change = 0.010987)
Training: (nnz = 231, max change = 0.033983)
Training: (nnz = 223, max change = 0.011559)
Training: (nnz = 223, max change = 0.029825)
Training: (nnz = 218, max change = 0.014230)
Training: (nnz = 217, max change = 0.034016)
Training: (nnz = 213, max change = 0.013610)
Training: (nnz = 214, max change = 0.063275)
Training: (nnz = 211, max change = 0.018152)
Training: (nnz = 128, max change = 0.000099)
Training: (nnz = 220, max change = 0.034018)
Training: (nnz = 212, max change = 0.009567)
Training: (nnz = 206, max change = 0.014882)
Training: (nnz = 203, max change = 0.007489)
Training: (nnz = 206, max change = 0.044506)
Training: (nnz = 205, max change = 0.006341)
Training: (nnz = 203, max change = 0.018613)
Training: (nnz = 202, max change = 0.006019)
Training: (nnz = 201, max change = 0.023723)
Training: (nnz = 202, max change = 0.010747)
Training: (nnz = 195, max change = 0.013776)
Training: (nnz = 197, max change = 0.024188)
Training: (nnz = 201, max change = 0.009171)
Training: (nnz = 199, max change = 0.019755)
Training: (nnz = 202, max change = 0.008910)
Training: (nnz = 196, max change = 0.016387)
Training: (nnz = 190, max change = 0.011747)
Training: (nnz = 195, max change = 0.014255)
Training: (nnz = 188, max change = 0.007746)
Training: (nnz = 120, max change = 0.000093)
Training: (nnz = 196, max change = 0.010112)
Training: (nnz = 191, max change = 0.006648)
Training: (nnz = 115, max change = 0.000100)
Training: (nnz = 208, max change = 0.080113)
Training: (nnz = 195, max change = 0.007551)
Training: (nnz = 192, max change = 0.024511)
Training: (nnz = 118, max change = 0.000097)
Training: (nnz = 188, max change = 0.010954)
Training: (nnz = 119, max change = 0.000098)
Training: (nnz = 189, max change = 0.010616)
Training: (nnz = 187, max change = 0.010480)
Training: (nnz = 187, max change = 0.011920)
Training: (nnz = 187, max change = 0.011501)
Training: (nnz = 187, max change = 0.013678)
Training: (nnz = 188, max change = 0.007433)
Training: (nnz = 114, max change = 0.000098)
neato version 1.8.10 (11/26/02)
