/sci/ - Science & Math

>>9391698
Hobbyist statistician here.

is statistics a viable career path

>>9391744
very

>>9391751
Better than engineering?

>>9391756
you should do what you enjoy more anon. after spending years studying sth you don't like just because another anon told you to do so is frustrating

>>9391756
As a career, yes. As a degree, maybe.

>>9391698
My presence almost surely converges to a value equivalent to here. Somebody put me out of my fucking misery

18 yr old retard that is taking AP Statistics reporting in

https://en.wikipedia.org/wiki/Multicollinearity

Always remember not to regress on variables with a highly negative correlation.

>>9391802
A high positive correlation would be just as bad

>>9391779
Remember that residual plots and scatter plots are NOT the same thing

>>9391698
undergrad here
Tell me GLMs get better after a bit

>>9391779
Tip: Companies love hiring college students and grads with experience, but NOT grads without experience. Take an internship in your desired field while in college.

>>9391758
>sth
I thought you had mistyped shit and then realized this was urban gorilla shorthand for something. You have a keyboard in front of you, use it. If you're on a phone you're getting raped by the ukrainian mystery dick scripts, and you should shoo shoo in any case.

>>9391698
I'm the economics/stats guy here, hoping to get into the MS in Stat program after I finish up my last couple of courses.

>>9394355
Same poster here, is anyone doing any undergrad research with a faculty member? If so, how did you get into it? I'd like to try to get some experience before applying for my masters.

>>9394366
I managed to do some by just asking the professor

>>9391802
Just use ridge, lasso, or PCR. You don't always need OLS.

>>9394355
>>9394366
Make sure your linear algebra is on point and review your calculus before starting your program.

>>9394678
Thanks, will do. I've been reviewing calculus but I still need to take linear algebra. I've an e-book on it though so I'll start working through it.

I should probably brush up on R as well I'm guessing.

>>9394658
I'll give that a shot, I still need to get a few classes out of the way first though.

>>9391779
enjoy your "how to read an ANOVA table" class

>>9394678
But muh gauss-markov theorem

>>9391756
Engineering is for brainlets only

>>9394678
Well,
The 'first' group could arive anytime, but when it does, the other group has 15 minutes to arive. 15 minutes being 25% of one hour; so 0.25?

Had my first stats class this semestre, was pretty fun. Shame that I need to wait until next autumn to take some more so I will just take Probability II, Stochastic Processes and applied stochastic processes(insurance math) and mathematical modelling this year.
In the autumn I will be able to take Linear Stats Models, Statistical Interference Theory, Analysis of Categorical Data and Financial Math, so it will be pretty lit semestre desu

>>9395278
Nice Anon! I'm still in my first year but I can't wait to take those classes too

>>9394045
What does this even mean?

>>9394678
This guy knows how to get work done and knows that first year graduate stats classes are basically Calculus VII with endless algebraic tricks that only extend to R^2 (and if you don't use them you won't have enough time to complete the exam).

>>9394678
No undergrad understands linear algebra. The undergraduate linear algebra course is less worse than useless at most universities. The beauty is best appreciated via analysis/PDEs and the special case of Hilbert spaces. The intuition is always geometric. The proofs are always algebraic. The historical motivation is always optimization. No more than 1 of these approaches are taught well in an of these are taught well given department; usually <1.

>>9394678
What happens when the response variable is discrete? (This Anon knows the answer.) Regression theory is a crutch for lazy departments at this point. Tibshirani & Hastie (ESL) should be the first year bible by now. That would require professors actually holding office hours and writing decent homework and exams because - you know - the material is hard.

>>9394787
In industry R/Python is 20/80 overall but varies by industry. SAS basically pharma. If you see Matlab then the managers are old and probably learned either stats in an engineering school. If you see Stata or SPSS or JMP run for the hills.

>>9395836
>What happens when the response variable is discrete?
Discrete how? Dichotomous: logistic regression, counts: Poisson regression, categorical: classification/clustering.
>This Anon knows the answer.
Congratulations faggot. You're not the only statistician on this board.
>the material is hard
That depends on the department's goal. Most are just plug and chug MS programs that use Minitab, unfortunately. Theory is not heavily emphasized at the MS level any more.
>In industry R/Python is 20/80 overall but varies by industry. SAS basically pharma.
SAS is also used at big companies and government with institutional inertia. Matlab still has some pretty cool uses. Agree about the Stata/SPSS though.

>>9395836
>What does this even mean?
https://en.wikipedia.org/wiki/Generalized_linear_model

>>9394678

>>9397269
Holy shit, that's genius. How do you think of that?

>>9397274
>taking the difference between two uniformly distributed random variables is genius

Is GLS regression only ever used in panel data or is it useful under other circumstances?

>>9397274
Introduction to Probability Theory. Lots of homework and exam problems were like that, but usually harder (involving multivariable calculus, integrating a joint PDF over the area of interest).

Just finished a class on Mathematical Statistics(MLE, confidence intervals, hypothesis testing, regression, anova, nonparametric). Where do I go from here?

>>9397274
see >>9397310 and >>9397320

Looks like I need to retire that IQ test finally, now that someone has solved it.

>>9394045
What is the difficulty that glms are giving you?

>>9397317
>panel data
Can you elaborate on this?

>>9397353
What was the textbook?

>>9397353
>Mathematical Statistics
>(MLE, confidence intervals, hypothesis testing, regression, anova, nonparametric)
Weird, that sounds like a non-mathematical statistics class. Math stat classes usually cover foundational stuff like measure theory.

>>9397353
le chi squared xDDD

>>9397368
Hogg

>>9397369
I know measure theory, so I think that teaching it in a stats class is kind of silly, but I know most stat people don't reach that far in the math sequence.

>>9397364
not sure i interpreted the question correctly

>>9397389
>Hogg
To continue studying math stat at the next level, then you want Casella and Berger's Statistical Inference (solutions to most of the problems are available online).

If you want a good linear models book, then look for Rencher's Linear Models in Statistics (mostly theory but has all solutions in the back of the book), but a more applied book would be Kutner's Applied Linear Regression Models.

If you've taken analysis and linear algebra (not that baby level augment matrix with zero vector and solve. A real linear algebra course) then look at The Elements of Statistical Learning (free pdf on authors website and full solutions elsewhere). If your math is not really there, then the authors also made a free book titled Introduction to Statistical Learning with R. The difference between the two is similar to baby Rudin and papa Rudin.

Any other questions, or even what are you trying to accomplish by studying?

>>9397393
Not going to comment on that one. You already killed one of my IQ tests today.

>>9397430
I got my degree in mathematics, and I want to learn more applied mathematics, so thats why I took mathematical statistics. The mathematical background for statistics is something that I don't need to worry about. I suppose that I can go start learning about machine learning, since that's where all the money is now.

>>9397364
https://en.wikipedia.org/wiki/Panel_data

>>9397432
wouldn't you need to know the percentage of ten-digit phone numbers that don't connect anywhere

>>9397476
No. The level of this problem is one that I would not expect a MS degree holder to solve. Hint: use Bayesian methods.

>>9397494
but then it being the correct number tells you nothing about whether or not the line is busy and P(line busy | right #) and P(line busy) cancel out and your just left with P(right # | line busy) = P(right #) = 50%

am i supposed to put an uninformative prior on holden's 50% sureness because he's a dumb twat or should i collect some data on how often dumb twats type the wrong number into their phones

>>9391770
just restrict your modem to a domain excluding this one. you'll be fine

>>9397548
P(correct # | busy signal) = [ P(busy signal | correct #) * P(correct #) ] / P(busy signal) = [ .01 * 0.5 ] / .01 = .5

I got the same answer as you. This question is flawed.

>>9397364
I've gotta work with GLMs for a project but have almost no practical knowledge on them since my prof is a theory spouting retard
>im no brainlet i swer

statistics are fake shit

Statisics undegrad here as well. Loving this shit. Banging bitches and banging Bayes.

>>9398356
I suggest taking a look at "Statistical Rethinking" by Richard McElreath. It's very example-driven and there's a great chapter on GLMs. Also the author has a sense of humor which is always nice.

Does anybody else feel these feels?

>Can't do a pen and paper statistics test to save my life
>Can do programming much better than my classmates
>Get shitty marks
>End up in a good job, because machine learning is more useful to businesses than rote learning the UMVUE derivation for a gamma distribution.

In my linear algebra course I learned to do least squares using transposed matrices
In my statistics course I learned to do it using sums of squared errors

I tried using both methods on the same problem and I did not get the same answer, although the line of best fit was very similar. Are these methods intrinsically different, or did I fuck up something? I calculated all the sums directly in the expression so there was no rounding error

>>9394045

LOL

thats pretty much me rite now

Question for y'all.

I'm trying to draw a graph that looks like the Standard Normal Distribution when it's plotted on a graph where the x-axis is logarithmic, instead of linear. (Sorry if I'm abusing terminology, I have very little background in maths.)

The idea I'm going for would put 10,000 at the mean, 100,000 at +1 standard deviation, 1,000,000 at +2 standard deviations, and so on. In the opposite direction, I'd have 1,000 at -1 standard deviation, 100 at -2 standard deviations, and so on.

Is there a function like this? If so, what would it look like, or how would you derive one?

Happy to answer any clarifying questions, or talk about my project if anyone's interested. Thanks in advance for any suggestions.

>pic unrelated

>>9394045

wait what are GLMs lol I was looking at the pic fml

Is Stan the way to go for Bayesian modeling?

>>9398678
he doesn't use the superior WINBUGS >>>/g/

>>9398720
>doesn't even use hybrid Monte Carlo
>Windows only
Nah.

>>9398486
If the logarithm of x is normally distributed x follows something called a (creatively named) lognormal distribution.

[math]
\texttt{> rnorm(1, c(0, 100))} \\
\texttt{[1] 0.1432776} \\
\texttt{> rnorm(2, c(0, 100))} \\
\texttt{[1] -0.2667315 99.2165480} \\
\texttt{> rnorm(3, c(0, 100))} \\
\texttt{[1] -0.9371833 99.1430490 -0.5432836}
[/math]

Who the hell designed R.

>>9398759
So it alternates between simulating the first and second components of the mean vector?
I don't see why that's particularly odd. What would you want it to do?

>>9398826
>What would you want it to do?
Something that doesn't feel completely arbitrary. Why allow for vector arguments to the mean in the first place? I could easily see someone unaware of rmvnorm thinking they were going to get an iid sequence of samples from a multivariate normal with mean (0, 100) out of that.

>>9398826
>>9398759
maybe it's a holdover from a time before apply and replicate existed?

I could also see doing something like
x <- rnorm(400,c(10,50,100,200))
x <- matrix(x,ncol=4,byrow=TRUE)

if I wanted to make a matrix with simulated data with one mean per column

>>9398759
r is annoying until you try and use python for similar use cases

>>9398759
>t.doesn't know about pseudorandom number generators or how they work

>>9398913
Agreed. I would take Matlab over Python any day of the week.

>>9398966
anon's complaint was how it was switching between the two specified means, which your snippet doesn't even address

>>9398913
> tfw the more I study math&stats the more I love R

>>9399080
it's so comfy for anything related to data analysis too

>>9398742
>lognormal distribution
This is exactly what I was looking for, thank you very much.

>>9398434
>Can do programming much better than my classmates

what do you mean by "programming" in context of statistics

>>9399080
>>9399081
How can anyone hate R?
It's so interactive it's like playing a video game.

>>9399463
Plenty of languages have integrated console/IDE things like RStudio.

Hi, literal statistics brainlet here, going to study pure maths next year and taking statistics in 2019 (can study 2 careers at the same time)

I know how discrete and continuous random variables work, I know how to work with some moments (basic ones) and I understand the first half of Meyer's introductory prob/stat, but statistical inference is just too much for me

What are some good introductory books on the subject? Videos, tips, anything can be helpful

>>9399548
need this too

It depends on how well you know probability theory, the way my school taught statistics was a proper introduction to probability theory (without really going too much into the measure theory), and then a formal look at statistical inference. It was divided into 2 courses (the second course being statistical inference), and both covered the book: "Mathematical Statistics and Data Analysis" by John Rice.

I highly recommend this book, the first half is all probability theory, so you can fill in the gaps you are missing for that, and the second half covers a formal theory of statistical inference that (importantly) ALSO covers non-parametric inference, which is a vital for any statistician to know about today. I am pretty sure there is a PDF of this book floating around on the internet somewhere

for those of you getting into it ann anastasi is thee late goddess of stats. get any edition of her classic textbook and you will pick it up.

>>9391698
Quality Control Manager here, you called?

>>9400459
DAMN! need this! but cant find pdf

Did anyone had to do robocode? If not, it basically is a field with fighting tanks. I want to model their locations. Where should I look for?

>>9400698
Kalman filters.

>>9391698
>Book cooking and overfitting data: the field

>>9400826
Why do undergrads call everything a field. It's so pompous. Just call it a subject.

>>9395836
Killer, I'm way more familiar with Python's syntax so that's very good to hear. Still want to get somewhat proficient at R though.

>>9396816
>Counts: Poisson regression
No. Binomial or bust.

>>9395258
wrong. if the first group arrives after 0:45, the overlap is discarded

>>9400943
Go away brainlet
http://www.math.usu.edu/jrstevens/biostat/PoissonNB.pdf

>>9400991
I'm not sure what that's supposed to tell me. Yes, in some cases the negative binomial is also preferable to the Poisson.

Is your complaint that I should have said negative binomial or binomial or bust? That seems like just a longer way of saying the same thing.

The only time you would use a poisson is if you lacked the information required to use a better distribution. And, ideally, you should probably shoot for Bernoulli, and just model the individual trials, but I won't go so far as to say Bernoulli or bust. Getting trial level data can be challenging, so I'll let you pass with a binomial.

>>9398434
>Can't do a pen and paper statistics test to save my life

I knew this feel until I got a very old fasioned prof this last semester who had tenure and wasn't afraid to humiliate you in front of the whole class if you fucked up - I had to learn out of fear

>>9400776
>>9400698
Will it work with several tanks, such that the probability of two tanks being close is very small?

>>9401165
I don't know. I have no idea what robocode is or what kind of information you have access to.

>>9395836
>If you see Stata or SPSS or JMP run for the hills.
Do people actually use JMP professionally? I have undergrads use it when they need to do stats but don't know R, but I try to get them off JMP as quick as I can.

>>9401311
I am guessing that you have never worded in industry?

>>9400826
Its hilarious how people assume that just because you CAN do something unscrupulous with the knowledge you get from a field, that all practitioners of that field do those unscrupulous things

>All chemists make bombs and drugs
>All engineers make weapons systems

Almost done with my Senior year and took AP Stat last year, god-tier course. Love it but what kind of jobs could I get related to statistics? I already have a stable 30$ per hour '''''''CS''''''' job doing Node and Angular programming, so should I even consider a career in stat or just keep doing this?

>>9401514
Stay in stats, do a CS minor, take a few CS upper-level electives in addition to that, and go into machine learning

>>9401546
is stats really that stable? thought that was one of those degrees that you needed to go to graduate school for

>>9401553
Do you want to do interesting work? You'll have to go to graduate school anyway.

If you just want to be a code monkey making 60$/hr starting, just do a CS major and the minimal math/stat requirements associated with it.

>>9401514
honestly to have that job in HS is impresive, i would keep it for a while

The statistics do not work

Has anyone here read An Intermediate Course in Probability by Allan Gut?
t. taking Probability II this upcoming semestre and it uses that book

Honest question lads. I took a college statistics course in high school, but it didn't transfer to the correct stats course for my major. Therefore, I took an applied stats course to fulfill the requirement. I have now taken two statistics courses. I have also taken courses in discrete structures, AI, data science, computer vision, and machine learning. HOWEVER, I have not officially taken an Intro to Probability course. Let's be real /sci/, have I basically covered 90% of the topics in an Intro to Probability course so I can jump to Intermediate Probability?

By the way, I've also taken calc 3, linear, and differential equations.

>>9401678
"Intro to Probability" and "Intermediate Probability" are just names, because probability can be taught at different levels. You should look up the textbook used in these courses and see what you learn.

>>9401621
Hack. All he does is complain about the limitations of statistics while contributing nothing of value. The only people who think hes actually smart are statslets who get impressed when he spouts stats terminology while acting indignant

>>9401621
ye cannae no nuffin: stats edition

stats babby here

so yesterday i thought i'd do a real basic statistic and collected some data during an NBA game

I counted every shot for one team and put each shot in 2 categories

Shots Under 5 seconds / Shots over 5 seconds.

there was a total of 72 shots.

12.5 shots were made with 5 seconds or less on the shot clock

34 shots were made with more than 5 seconds on the shot clock

58 total shots were attempted with 5 seconds over the shot clock

14 total shots were attempted with 5 seconds or less on the shot clock

12.5/72 = 17.3%

34/72 = 47.2%

does this statistic sound correct, like the way i collected data and such?

>>9401866
You seem to have a very limited view of what statistics is about. Might I recommend reading a book.

>>9401895
Daily reminder statistics is just a dumbed down version of the vastly superior computer science sibling, data science.

>>9401866
Great work! You're on the right track :) However, how did you get the 12.5 shots (made with 5 seconds or less)? Not sure I understand 0.5 shots. Also, your statistics are slightly off.

12.5/72 translated to english means "there were 12.5 of the total shots made"

34/72 translated to english means "there were 34 of the total shots made"

If you want stats to indicate success rate of shots under/over 5 seconds, you should calculate:

12.5 / 14 , which is translated as "12.5 of the 14 shots attempted with 5 seconds or less were successfully made"

and

34/58, which means "34 of the 58 shots attempted with 5 seconds or more were made"

Also, you should figure out which category the shot belongs if there's exactly 5 seconds left on the clock.

>>9401910

>>9401427
He's implying that it's a routine expectation in the field, whereas most chemists are not expected to make heroin and bombs as a matter of course.
Look at quantitative finance for chrissakes, or the big data engine behind the hillary campaign.

>>9391698
I have a question that I am too lazy to do myself in order to piss off /pol/
>IF the average IQ of whites is 100
>IF the average IQ of blacks is 85
>IF both have a standard bell curve distribution
Then:
In a room with 100 random whites and 100 random blacks, what would be the racial makeup of the 100 SMARTEST people in that room?

>>9402181
Well, what are the standard deviations for black and white IQ scores?

What does /sci/ think of this book? I've read through the first twelve or so chapters and have done some exercises. I've gotten a lot of out it.
>t. cs/applied mathlet

>>9391698
>Gaussian distribution
kek, nothing personal, kid
>>9402181
Do Jews count as white?

>>9397476
I'm confused by the phone being busy part. How does that help you tell if the number is correct or not?

>>9402374
fuck, meant to reply to this one >>9397432

>>9397432
Isn't it just 50%?

>>9400505
libgen.io (library genesis) has pdfs
http://libgen.io/book/index.php?md5=41119197FE08494D8BA7DDEA173F4474

Anybody interested you can also take Stats from Open University in UK online for a fraction of the cost of reg university http://www.open.ac.uk/courses/qualifications/q36

Then go pay for a MSc or pursue PhD

>>9397432
bump
I need to know the fucking answer

>>9394355
Data collector here, we make that shit up and your educcation and function is socciety is hopeless and useless.

>>9401910
Honest question, what's the difference between data science and statistics?
Every data science and machine learning book I read seems to be 90% statistics and the related theory to it.

>>9403418
Not a big difference, Machine Learning is just statistics, all the major PhDs involved in ML and Neural Net research are Stats PhDs.

So it's basically a statistician that can program and script data set 'learning' so a program can make deductions from said data set.

Anybody here study Econometrics or Quantitative Finance? Is statistics a big part of the discipline?

>>9403485
Yes. Econometrics is basically Statistics.