[Kaeldric]

Hello.
Recently i developed a a linear regression algorithm and I wanted to test it plotting it's learning curves.
I took some datasets from UCI (https://archive.ics.uci.edu/ml/datasets.html) and started draw some curves.
After a while, for some combinations of parameters, the curve of the learning set and the cross validation set one crossed each other!
After that they seems to continue their journey: the learning curve continued to raise and the CV curve continued to go down. 

I studied that those curves should saturate to two near values or (in high BIAS cases) they should saturate to distant values ... but never cross each other. 

Does anyone know if this is a normal behavior?

Thank you. 

[keghn]

 Are talking about something like a  Neural network fits noisy standing sine wave:

[ranch vermin]

All I know about this,  thats sorta like it,  is take reading x as the address,  and plot reading y - and youve got a transfer array,  the base building block of computer memory systems.

I like it,  but I havent ever "evolved" it, or anything, i just make direct assignments in my work.


[EDIT]   ur symantics might be completely non correlating at all,  that would cause that behaviour

u at least need some kind of proportional relationship,   like texture density and z distance, or it wont work at all.


thanks keghn for that video, i wasnt quite sure what i was reading...
[EDIT]

[keghn]

 My notes, Cross Validation:

[Kaeldric]

My notes, Cross Validation:

Thank you, it was very instructive. It is, by the way, almost the same procedure I used.
The only difference id that I didn't use folds. I extrapolated a training set and a cross validation set from the original data set and plotted the MSE from the two sets as a function of the number of examples of the training set used in the learning step.
The only thing that seemed strange to me is that after a particular value of the examples the algorithm calculate better hypothesis  from the cross validation set than the training set.
I ask myself if this is possible behavior or if there is something wrong in the procedure I implemented.

[Kaeldric]

I elaborated a different set of examples and I plotted "better" curves this time.
Is there possible that I found a couple of data set that I wasn't able to model with a simple polynomial regression?

By the way, this time I notice that increasing the degree of the polynomial  the MSE of the training set increases. I expected to see it falling because the polynomial is overfitting the data.

Do you think that is there something wrong or it's me too much confident on the theory?

[Korrelan]

Hi Kaeldric

Is the linear regression algorithm you are using based closely on an existing methodology or is a ‘new’ approach?

You extrapolated you initial training set, then got better results with the same algorithm and a second extrapolated/ elaborated set, to me this points to an error in the training data, or the operational range of the algorithm.

If you are trying to improve on an existing method, I would suggest you first run your own extrapolated training data through an existing known algorithm just to check the validity of data sets.

 

[keghn]

After that they seems to continue their journey: the learning curve continued to raise and the CV curve continued to go down. 

 The your regression line should go travel in between the high bumps, points, and low bump of the original line.

Least squares:   
https://en.wikipedia.org/wiki/Least_squares     

 I like to use a traveling window algorithm and find the average sum within the window.
Window Sliding Technique: 
https://www.geeksforgeeks.org/window-sliding-technique/ 

[Kaeldric]

Hi korrelan.
I'm using a simple Octave implementation of linear regression based on gradient descent methodology to minimize the cost function.
It is a prototype I used to get a machine learning certification recently and I am quite sure it is doing is job.
But you are right ... maybe I try to check the implementation again or try something more "tested".
Can you suggest me a known implementation I can use?

Thank you.

[Kaeldric]

The your regression line should go travel in between the high bumps, points, and low bump of the original line.

I'm very sorry but I don't understand the meaning, probably because my little broken english. Can you explain please?

By the way I think I used the same approach you suggested, in fact my cost function has the form:

C = (1/2m)*sum_m(h(x)-y)²

where m=the number of elements in the set
h(x) the hypothesis given input element x
y the real value I should get with x

[keghn]

 You are using the "root mean square" equation?
 Might be better if you use the "least mean" equation?

The in between line of the high and low points. Or in between the high and low points of a wavy line. The fitted line: 
https://en.wikipedia.org/wiki/Linear_least_squares_(mathematics)#/media/File:Linear_least_squares_example2.svg 


Linear Regression - Least Squares Criterion Part 1:


 I like smoothing algorithms. 

Savitsky-Golay Smoothing: 
http://www.ipredict.it/methods/SavitskyGolay/ 


Smoothing spline: 
https://www.google.com/search?q=Smoothing+spline&client=ubuntu&source=lnms&tbm=isch&sa=X&ved=0ahUKEwjAxd6MrKjZAhUKqFQKHfBtDAE4ChD8BQgLKAI&biw=1195&bih=614#imgrc=VeRT3ekf2RU3MM:

[Kaeldric]

You are using the "root mean square" equation?
 Might be better if you use the "least mean" equation?

Obviously this didn't change the general curve plot, but the values give me a more intuition about the real error rate.
Thank you for the suggestion!