[ 3 / biz / cgl / ck / diy / fa / ic / jp / lit / sci / vr / vt ] [ index / top / reports ] [ become a patron ] [ status ]
2023-11: Warosu is now out of extended maintenance.

/sci/ - Science & Math

View post   
File: 21 KB, 300x300, lillian-gish-9312548-1-402.jpg [View same] [iqdb] [saucenao] [google]
9688057 No.9688057 [Reply] [Original]

Why don't ML people just perform meta-ML experiments where they use ML to learn new ML techniques? Am I oversimplifying it?

Sometimes you'll see a mention of a new technique where they combine a certain type of layer with some mathematical linkage of parameters between the layers, and these things seem like the kinds of techniques that could easily be found through brute force.

Just rent an AWS cluster (or Amazon themselves could even do this), generate a "template" of potential ML techniques, then just execute them all and explore the space of parameter tuning and layer transformations, then heuristically determine which methods seem promising based on classification or scoring results and send a report off daily to the researchers

Is this being done?

>> No.9688059

>ML
not science or math

>> No.9688061

>>9688059
How is it not math retard

>> No.9688111

>>9688057
My understanding is that it can be done in theory but is completely impractical, because the universe of "potential ML techniques" is subject to the curse of dimensionality (ML techniques are essentially mathematical formulas, hence this space would resemble the space of syntax trees) and the halting problem puts a fundamental limit the effectiveness of regularization (using ML techniques or otherwise).

Though I don't have any formal ML background and am basically talking out of my ass here, so I'd be happy to be proven wrong.

>> No.9688240

>>9688057
Why don't you study ML and figure it out yourself?

I'm studying ML and I'm wondering why on earth you'd want to run an ML algorithm to figure out which ML technique to use. This is something that you're supposed to decide. You know, the act of designing involves making informed decisions.

>> No.9688246

>>9688240
>I'm studying math and I'm wondering why on earth you'd want to run a computer algorithm to figure out the answers to math problems. This is something that you're supposed to do yourself. You know, the act of performing mathematics involves making calculations

>> No.9688275

>>9688057
>Why don't ML people just perform meta-ML experiments
>where they use ML to learn new ML techniques?
>Am I oversimplifying it?
Ya think?! Dduuuhh

>> No.9688301

>>9688057
Real answer: It is theoretically possible. Practically, there is a limit on the data and computing resources available, and there is also the problem of overfitting the validation set: You pick some approach, train on the training set, and then check how good you perform on the validation set. If you do this multiple times for multiple times with different approaches, you might find one which is in general not very useful, but (by chance) performs well on the validation set, and, because of that, overestimate the approach. And mixing up training/validation set and trying out each approach multiple times does not help: This would just cause you to find the best approach based on the union of both sets, but still not necessarily a generally good approach