/sci/ - Science & Math

(Continued)

Technical notes:

- "Mention" is defined as a post's comment containing the board's abbreviation immediately inside two forward slashes (e.g., "/pol/"). I might be missing some mentions, but I can't really automatically determine whether a comment is about /a/ or just used the word "a", and I'm not counting them by hand. A quick ctrl+f of /sci/'s catalog before posting this shows that twice is the word 'sci' used without slashes in OP comments compared to 27 times with slashes.

- I try collect around 7 pm EST.

- /f/ is a bitch that doesn't act nice when running through my capture script, and I don't care to deal with the red-headed step child of 4chan. I stopped collecting it a week ago and advise you to take any of its statistics with a grain of salt.

- I used R for the analysis and the ggplot2 package for the graph.

Well that's sort of cool I guess.

Send this to moot.

Neat.

You might want to try out some of the classifiers from Orange for python and see what pops out. The new Mathematica might have some useful tools too.

>/sci/
>Highest /x/ rate

>/pol/
not suprised
>/g/
somewhat suprised

/m/ and /f/ are mistakenly counted on /soc/

>most mentions of [s4s] come from /sci/, /b/, /po/
>most mentions of /sci/ come from [s4s] and /3/

intredasting....

>/sci/ mentioning /lit/ more than any other board
>/lit/ not reciprocating
fuckers

>>6929190
/pol/'s like the 2nd most populat board on 4chan. of course it gets mentioned a lot... just like /b/ gets mentioned a lot.

all this chart shows is popularity of each board. more popular the board, more mentions of that board on all other boards.

>>6929200

speaking of which, OP ought to include "[s4s]" in the search rather than just "/s4s/"

>>6929099
>/fit/ seems to have talked about /3/ a lot. The highest rate of mentions is /soc/ talking about /m/
I don't get it

Can you just pull up a few examples of either? This has to be some kind of error.

>>6929208
that's not why /pol/ is most mentioned

>>6929208
No, it shows the effect of "go back to >>>/pol/" type comments.

>>6929221
In /soc/ they often do the 23/m/(dunno about the third one) basically to shorten their sex age and so on. So /m/ for male and /f/ for female has been mixed up as mentions of the board.

anymore stats/graphs?

>>6929105
Switch your "mentioning" criteria to a regex like \b(/boardname/)\b so that urls to unrelated stuff aren't caught.

>>6929099
Everyone hates /pol/, /b/, /g/, /v/, and /x/, /fit/ so it makes sense that they get mentioned a lot. Similarly everyone sends people to /s/, /r/, and /d/ so it makes sense that they get mentioned a lot as well. Nothing out of the ordinary there.

Similarly the only boards that actively sit around mentioning every other board are /pol/, /r9k/, /s4s/, and /vg/. I can understand the first three since they're literally containment boards for cancer (well except /s4s/, they're nice guys) but I'm totally surprised about /vg/.

>>6929221
>>6929240

OP here. This seems to have been, in fact, the reason. it's why /soc/'s second most talked about board came up as /f/.

>>6929208
People don't include the names of each board they visit in their comments. This could show cross-boarding, though.

>>6929183
I do plan to learn about textual analysis, but I'm starting slow.

>>6929252
>pol is a containment board for cancer
>vg isnt

talk about bias

you should include hits for things like pol trv tv vp wsg
without the slashes, you get the idea but unique ones would stand out if they're inside whitespace
try to gather some sort of context but have it easy to exclude from the main hits
think about just with english you could do some easy things to see how a or an was used in a sentence
A sentence though is also hard to grab so it gets worse and worse
but more interesting thats what I would do before you get bored with the project

>>6929260
bullshit. it's a huge board and only /b/ is bigger.

>>6929265
/vg/ is a containment board for cancer too

you are an idiot.

>>6929282
All boards other than /sci/ are containment boards for cancer.

>>6929282
I always thought /vg/ was like /v/ but for more obscure videogames or something. I went there now and saw it was a board solely dedicated to generals.

>>6929282
>containment board
/sci/ is a containment board for autistic nutjobs

>>6929285
/sci/ doubles as a containment board for pseudo-intellectuals and pseudo-philosophers, also for kids who want to appear smart and aren't even one of those two

>>6929260
>>6929282

hahaha, you're right. I was confusing it with /vr/. I agree, /vg/ is a containment board.

>>6929301
Though I'm sure you fit that category you really belong on /lit/.

>>6929291
That's /r9k/.

>>6929241
Here are the first ones in case anyone's not seen them.

>>6929309
>That's /r9k/.
have you seen the results of "let's take an autism test" threads? at least 60% of /sci/ posters seem to be autistic and 10% of those are highly autistic.

>>6929323
people on /sci/ pretend to be autistic because they think autists have higher IQs

>>6929322

did you email these to moot?

you should

>>6929323
The only reason to take a retarded "test" on the internet is to see how ridiculous you can make your score.

>>6929329
<span class="math">protip% The IQ threads are only there for shitposting. They keep all of the cross-boarders out of the rest of the board.[/spoiler]

>>6929245
Actually, I had decided to not use boundaries with the pattern since I wanted to capture urls. Otherwise, things like >>6929231 would've gone unnoticed.

Also, I like the idea that this thread will skew any runs I do on today's posts.

So how would a static approximation of the most shit board be?

I figure you take the 100 most common non-dictionary words per board and count the frequency of each per post. The boards that use these memes most often should be the shittier.

>>6929329
>people on /sci/ pretend to be autistic because they think autists have higher IQs
lol.. yeah.. keep telling yourself that, kid.

just remember the next time you're arguing with someone on here, they probably can't even leave their basement on their own without their mom accompanying them.

>>6929353
/hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/ /hm/

OP you might be able to deal with those spurious /m/ and /f/ hits if you alter your search so it looks for at least one whitespace outside the slash, so you pick up things like /sci/entists or co/ck/

>>6929362
or just ignore if there is a numeral preceding the first '/', and only in the case of /f/ and /m/

>>6929362
although i think you'd have to construct separate grep() calls to take care of whitespace-before and whitespace-after cases, and make sure you're looking for a trailing/leading non-whitespace character so you don't double-dip on the whitespace-before-and-after cases...

nevermind, this is quickly getting way too complicated

>>6929373
oh, yeah, that's a much better solution

>>6929362
>>6929374
It might be a good idea to run both a really limited filter and the regular filter on the site at the same time. Do this for about a week. Then compare the results between both. The difference may be negligible in which case it would be a total waste of time to worry about the affect of unrelated links.

After all, it's not like we're doing anything fancy with these statistics besides gaining a general overall feel of things.

>>6929331
>>6929322
if i can make a suggestion? i love that you're using a gradated background, more people need to do that, but with how closely you've packed the bars in this plot it's visually alittle confusing when you get to the labels and try to figure out which one is which - i keep wanting to match up the borders in the label section to the borders in the graph.

some things I think would help:
>make the gradations in the background not be hte same width as the bars in the plot
>increasing your graph width so you have more room for the eyes to play around in
>change the color of the label box region so it isn't the same color as the background

>>6929255
look on the bright side, you just gave us good hard data that there's more men than women on /soc/

next graph, which board posts ayy lmao the most

>>6929424
I'd assume it's /sp/.

>>6929427
i would have guessed /b/, then /x/

>>6929361
You'll only count once, but thanks for caring.

>>6929356
This was my original goal with the median replies per thread. I thought longer threads meant better discussions, but that was silly.

/sci/ needs the ability to share data with text files. I'm not confident in my LaTeX to make a matrix.

>>6929438
Long threads are often the result of trolling, image and link dumps (going on for days in slower boards), or in the case of /vg/ just cancerous generals.

search for "/boardname/ here" and "from /boardname/" or similar

get an approximation of people "visiting" other boards, and where they visit

>>6929415
No, there are just more instances of someone claiming to be a man than instances of someone claiming to be a woman. We don't know much about the actual distribution.

"Do you really think someone would do that? Just go on the internet and tell lies?"

>>6929362

But then I'd miss fa/tg/uy.

I could just do "(?<!\d)/board/".

Also, here are the words containing "Hitler" from the week of 11/19 to 11/25:
shitlers
s_panzer_division_hitlerjugend
ss_panzer_division_hitlerjugend
picohitlers
painting_by_adolf_hitler_expected_t
shitlery
senhitlers
adolf_hitler_2000

4chan needs an analytics tool like Google has with Google labs. It'd be entertaining and informative.

>>6929472
Or maybe it was Google trends? I don't remember.

>>6929466
This is why you need machine learning. You can teach it cases so it can intelligently count references to a board.

>>6929466
so are you grabbing image names, subject and name field too?

>>6929466
treat "_" as whitespace

>>6929424
All of this is with the same week's data, since I don't feel like juggling more in RAM.

>>6929472
I'm quite certain given a very simple self-learning shitposting algorithm most board posts can be completely replaced by bots without anyone noticing.

>>6929496
This only shows which boards have the most posters. You'd need relative frequencies instead. Divide by the total number of posts on each board in the given time range.

>>6929496
wow ty, good job to /sci/ keeping it under 50

>>6929496
Actually this is a real interesting way to study the spread of memes over boards. Maybe one can make schemes on how and to what degree certain board shares memes and stuff.

>>6929498
That might actually be a good way to increase revenue on a website. Automate shitposting to make the community feel larger and more active so as to draw in more actual pageviews... Holy shit I can't believed it took me so long to realise why YouTube is so retarded. God damn it Google.

>>6929486
I am grabbing everything I can from the API. If it's given, I put it in the row for that post. If not, it's NA.

Here's the API documentation: https://github.com/4chan/4chan-API

>>6929494
I just make those word lists to see how creative people are. I'd rather catch a URL than miss something funny. Especially since URLS can be funny.

>>6929504
This one's for you.

>>6929481
seconding machine learning

/d/

there you go /sci/

>>6929509
>>6929527

goddammit /sci/

>>6929540
ayy lmao

>>6929540
>we're even higher than /x/
ayy

I should have known that I don't need to know LaTeX. There's a package that automatically creates LaTeX expressions of tables. Here are the patterns I used to search for memes:
LOL = '^lol$',
meme = '(?<![a-z])meme(?![a-z])',
crazy_8 = '(?<![[:alnum:]])(gr|r|b|m|str)8(?![[:alnum:]])',
feels = '(?<![a-z])feels|that\\s*\\S*\\s*feel(?![a-z])',
shiggy = '(?<![a-z])ishygddt|shiggy|shigy(?![a-z])',
bane = '^(.*\n)?bane\\??(\n.*)?$', # On it's own line
math_phd = '\\$300[,\\.]?(000|k)\\s*start',
hitler = 'hitler', # Can't ignore creativity, like robohitler
kek = 'kek', # I don't think this shows up in any actual word
hypnotoad = 'hypno\\s*toad',
year_of_our_lord = '^(.*\n)?>((the )?year( of (our|the) lord)? )?\\d+\\s*(\\+\\s*\\d+)(\n.*)?$',
fedora = 'fedora'

Results in the next few posts...

\begin{table}[ht]
\centering
\begin{tabular}{rlrrrrrrrrrrrr}
\hline
& board short & LOL & meme & crazy 8 m8 & feels & shiggy & bane & math\_phd & hitler & kek & hypnotoad & year of our lord & fedora \\
\hline
1 & 3 & 2 & 3 & 24 & 14 & 0 & 0 & 0 & 2 & 19 & 0 & 0 & 1 \\
2 & a & 3 & 199 & 129 & 372 & 5 & 1 & 0 & 52 & 136 & 0 & 0 & 36 \\
3 & adv & 0 & 2 & 58 & 279 & 2 & 0 & 0 & 1 & 31 & 0 & 0 & 15 \\
4 & an & 1 & 5 & 12 & 22 & 1 & 0 & 0 & 4 & 23 & 0 & 0 & 7 \\
5 & asp & 2 & 10 & 50 & 66 & 3 & 0 & 0 & 2 & 51 & 0 & 0 & 20 \\
6 & b & 8 & 59 & 197 & 151 & 3 & 0 & 0 & 69 & 424 & 1 & 0 & 46 \\
7 & biz & 0 & 3 & 35 & 49 & 0 & 0 & 0 & 4 & 61 & 0 & 0 & 4 \\
8 & c & 0 & 4 & 7 & 21 & 0 & 0 & 0 & 2 & 3 & 0 & 0 & 1 \\
9 & cgl & 0 & 15 & 39 & 152 & 3 & 0 & 0 & 5 & 83 & 0 & 0 & 18 \\
10 & ck & 0 & 141 & 75 & 80 & 1 & 0 & 0 & 6 & 59 & 0 & 0 & 42 \\
11 & cm & 2 & 2 & 2 & 32 & 1 & 0 & 0 & 1 & 7 & 0 & 0 & 3 \\
12 & co & 0 & 65 & 126 & 399 & 0 & 2 & 0 & 63 & 144 & 1 & 0 & 37 \\
13 & d & 2 & 10 & 27 & 128 & 0 & 0 & 0 & 5 & 14 & 0 & 0 & 4 \\
14 & diy & 0 & 2 & 14 & 23 & 2 & 0 & 0 & 0 & 13 & 0 & 0 & 4 \\
15 & e & 0 & 3 & 5 & 18 & 0 & 0 & 0 & 1 & 4 & 0 & 0 & 0 \\
16 & f & 0 & 3 & 4 & 18 & 0 & 0 & 0 & 1 & 4 & 0 & 0 & 0 \\
17 & fa & 4 & 155 & 276 & 120 & 6 & 3 & 0 & 21 & 151 & 0 & 0 & 88 \\
18 & fit & 7 & 123 & 431 & 492 & 18 & 0 & 0 & 30 & 439 & 0 & 0 & 114 \\
19 & g & 3 & 106 & 262 & 182 & 18 & 1 & 0 & 13 & 223 & 0 & 0 & 143 \\
20 & gd & 0 & 1 & 8 & 22 & 1 & 0 & 0 & 1 & 13 & 0 & 0 & 1 \\
\hline
\end{tabular}
\end{table}

Forgot the tags originally
<span class="math">
\begin{table}[ht]
\centering
\begin{tabular}{rlrrrrrrrrrrrr}
\hline
& board short & LOL & meme & crazy 8 m8 & feels & shiggy & bane & math\_phd & hitler & kek & hypnotoad & year of our lord & fedora \\
\hline
1 & 3 & 2 & 3 & 24 & 14 & 0 & 0 & 0 & 2 & 19 & 0 & 0 & 1 \\
2 & a & 3 & 199 & 129 & 372 & 5 & 1 & 0 & 52 & 136 & 0 & 0 & 36 \\
3 & adv & 0 & 2 & 58 & 279 & 2 & 0 & 0 & 1 & 31 & 0 & 0 & 15 \\
4 & an & 1 & 5 & 12 & 22 & 1 & 0 & 0 & 4 & 23 & 0 & 0 & 7 \\
5 & asp & 2 & 10 & 50 & 66 & 3 & 0 & 0 & 2 & 51 & 0 & 0 & 20 \\
6 & b & 8 & 59 & 197 & 151 & 3 & 0 & 0 & 69 & 424 & 1 & 0 & 46 \\
7 & biz & 0 & 3 & 35 & 49 & 0 & 0 & 0 & 4 & 61 & 0 & 0 & 4 \\
8 & c & 0 & 4 & 7 & 21 & 0 & 0 & 0 & 2 & 3 & 0 & 0 & 1 \\
9 & cgl & 0 & 15 & 39 & 152 & 3 & 0 & 0 & 5 & 83 & 0 & 0 & 18 \\
10 & ck & 0 & 141 & 75 & 80 & 1 & 0 & 0 & 6 & 59 & 0 & 0 & 42 \\
11 & cm & 2 & 2 & 2 & 32 & 1 & 0 & 0 & 1 & 7 & 0 & 0 & 3 \\
12 & co & 0 & 65 & 126 & 399 & 0 & 2 & 0 & 63 & 144 & 1 & 0 & 37 \\
13 & d & 2 & 10 & 27 & 128 & 0 & 0 & 0 & 5 & 14 & 0 & 0 & 4 \\
14 & diy & 0 & 2 & 14 & 23 & 2 & 0 & 0 & 0 & 13 & 0 & 0 & 4 \\
15 & e & 0 & 3 & 5 & 18 & 0 & 0 & 0 & 1 & 4 & 0 & 0 & 0 \\
16 & f & 0 & 3 & 4 & 18 & 0 & 0 & 0 & 1 & 4 & 0 & 0 & 0 \\
17 & fa & 4 & 155 & 276 & 120 & 6 & 3 & 0 & 21 & 151 & 0 & 0 & 88 \\
18 & fit & 7 & 123 & 431 & 492 & 18 & 0 & 0 & 30 & 439 & 0 & 0 & 114 \\
19 & g & 3 & 106 & 262 & 182 & 18 & 1 & 0 & 13 & 223 & 0 & 0 & 143 \\
20 & gd & 0 & 1 & 8 & 22 & 1 & 0 & 0 & 1 & 13 & 0 & 0 & 1 \\
\hline
\end{tabular}
\end{table}[/spoiler]

>>6929588
>>6929582
Huh. Does /sci/ not do tables?

>>6929582
>>6929588
you are an utter failure

>>6929593
/sci/ doesn't do a lot of shit.

can you give the percentage of greentext vs regular text per board or is that too much?

>>6929600
I could. I'm still trying to clean up all the HTML text in the comment fields. So not tonight, but it is doable.

>>6929498

Sure, kid.

I wanted to leave you all with a list of the different types of fags 4channers call one another, but that list was 3,059 words long. So here are 50 chosen at random:

poorfag
showfags
zorofags
conefag
jonfags
warriorfag
dickfag
fairyfags
teamfag
faggotfaggot
milfags
smartfags
widowfag
robinfag
seiyuufag
niheifag
voidfag
vlcfag
islandfags
lolifag
pancakefags
fresnofags
helfag
waacfag
totallynotsamefag
dieselfag
exaltedfag
nukigefags
manxfags
gagfags
minnesotafag
writefaggots
danishfag
waacfags
harukafag
insiderfag
elsafag
uglyfag
stuckfag
twitterfags
gaiafags
runefag
everyfaggot
lazyfag
misakafag
anniefag
electronicsfags
craftfaggot
autistfag
sausefag

>>6929584
>>6929597
>>6929593
>>6929588

mathjax and jsmath don't implement the tabular environment because it's a textmode environment and not a mathmode environment (I don't know if any of the web based LaTeX interpreters do it).

You can remove the table and centering tags, switch tabular tag to an array tag, and remove all of the hline tags. The \hline (horizontal line) tags won't work inside an array in jsmath, but they should work in an array in mathjax.

We should e-mail moot and tell him to upgrade /sci/ from jsmath to mathjax (more or less the successor to jsmath since jsmath is now stable/unmaintained).

I'll repost a fixed version of your table in the next post.

>>6929621
>vlcfag
>widowfag
>electronicsfags
>craftfaggot
The common type of fags really are telling of the board culture.

>>6929624

On a sidenote, the warosu.org/sci/ archive is already on mathjax.

<span class="math">\begin{array}{rlrrrrrrrrrrrr}
& board short & LOL & meme & crazy 8 m8 & feels & shiggy & bane & math\_phd & hitler & kek & hypnotoad & year of our lord & fedora \\
1 & 3 & 2 & 3 & 24 & 14 & 0 & 0 & 0 & 2 & 19 & 0 & 0 & 1 \\
2 & a & 3 & 199 & 129 & 372 & 5 & 1 & 0 & 52 & 136 & 0 & 0 & 36 \\
3 & adv & 0 & 2 & 58 & 279 & 2 & 0 & 0 & 1 & 31 & 0 & 0 & 15 \\
4 & an & 1 & 5 & 12 & 22 & 1 & 0 & 0 & 4 & 23 & 0 & 0 & 7 \\
5 & asp & 2 & 10 & 50 & 66 & 3 & 0 & 0 & 2 & 51 & 0 & 0 & 20 \\
6 & b & 8 & 59 & 197 & 151 & 3 & 0 & 0 & 69 & 424 & 1 & 0 & 46 \\
7 & biz & 0 & 3 & 35 & 49 & 0 & 0 & 0 & 4 & 61 & 0 & 0 & 4 \\
8 & c & 0 & 4 & 7 & 21 & 0 & 0 & 0 & 2 & 3 & 0 & 0 & 1 \\
9 & cgl & 0 & 15 & 39 & 152 & 3 & 0 & 0 & 5 & 83 & 0 & 0 & 18 \\
10 & ck & 0 & 141 & 75 & 80 & 1 & 0 & 0 & 6 & 59 & 0 & 0 & 42 \\
11 & cm & 2 & 2 & 2 & 32 & 1 & 0 & 0 & 1 & 7 & 0 & 0 & 3 \\
12 & co & 0 & 65 & 126 & 399 & 0 & 2 & 0 & 63 & 144 & 1 & 0 & 37 \\
13 & d & 2 & 10 & 27 & 128 & 0 & 0 & 0 & 5 & 14 & 0 & 0 & 4 \\
14 & diy & 0 & 2 & 14 & 23 & 2 & 0 & 0 & 0 & 13 & 0 & 0 & 4 \\
15 & e & 0 & 3 & 5 & 18 & 0 & 0 & 0 & 1 & 4 & 0 & 0 & 0 \\
16 & f & 0 & 3 & 4 & 18 & 0 & 0 & 0 & 1 & 4 & 0 & 0 & 0 \\
17 & fa & 4 & 155 & 276 & 120 & 6 & 3 & 0 & 21 & 151 & 0 & 0 & 88 \\
18 & fit & 7 & 123 & 431 & 492 & 18 & 0 & 0 & 30 & 439 & 0 & 0 & 114 \\
19 & g & 3 & 106 & 262 & 182 & 18 & 1 & 0 & 13 & 223 & 0 & 0 & 143 \\
20 & gd & 0 & 1 & 8 & 22 & 1 & 0 & 0 & 1 & 13 & 0 & 0 & 1 \\
\end{array}[/spoiler]

>>6929527
AYYY

>>6929628
huh, that's weird. It works in the jsmath lab, also it seems that warosu's mathjax implementation is currently broken as well.

http://www.math.union.edu/~dpvc/jsmath/jsMath-lab.html

>>6929621
you should throw the whole list up on pastebin

>tfw we truly rule all the boards
It's good to be king

>>6929641
this

>>6929641
http://pastebin.com/iWEH0P7m

>>6929648
>rule
That is a strange way to describe what a cancer does.

>>6929621
Don't forget Eurofag
or maybe europoor

>>6929496
ayyy lamo

>>6929658
OP is a funnynuggetfag

>>6929648
>/pol/ goes to other boards to talk about /pol/
>The stats now show that other boards are constantly talking about /pol/
One thing's for sure, you guys don't know how to analyze stats.

>>6929681
who goes to other boards to talk about /pol/ in number able to effect the stats?

I'm currently putting all the data I've collected into a dropbox folder, along with the scripts I wrote to create the data tables. I figured other people might want to play around too, and likely would do better than me. To open and use them, you need R, as well as the rjson and data.table packages. R is free (as in beer and speech), so I don't feel compelled to write everything to a csv. It'll be about 30 minutes until everything's synced (nearly a gig), but here are the links to the stuff in my drop box:

4chan Data:
https://www.dropbox.com/sh/njxtqsnvysrgrme/AABkqEH4sHY92ko7mnk8dbJRa?dl=0

All Thread Table Create.R
https://www.dropbox.com/s/324xjjpgknp96d8/All%20Thread%20Table%20Create.R?dl=0

json_4mat.R
https://www.dropbox.com/s/xoprd3qogg2tbbo/json_4mat.R?dl=0

json_to_chanrow.R
https://www.dropbox.com/s/x0zd6prj8tisa1v/json_to_chanrow.R?dl=0

read_thread.R
https://www.dropbox.com/s/l97oblezdpkxsq9/read_thread.R?dl=0

thread_list_to_table.R
https://www.dropbox.com/s/suuyx7i17jfvsd3/thread_list_to_table.R?dl=0

I make no claims as to my ability to write efficient/unbuggy scripts or comments at any level of proficiency. "All Thread Table Create.R" requires all the other R scripts to work, and it will throw warnings for 404 since there's a lag between getting the thread URL and the thread's JSON. So some threads may disappear in that time. At least, that's what I think happens.

>>6929252
>>6929260
>>6929282
>>6929285
>>6929291
>>6929301
>>6929309
yfw every board is a containment board

>>6929684
/pol/ does, check your reading skills man

>>6929681

>you don't know how to analyse stats
>muh feelings say it's because of /pol/ going on other boards

>>6929681
This is why people call you stupid, /pol/

>>6929721
Well it's certainly not because anyone likes you... I mean heck, /pol/s been said in this board alone like 10 times, and here you shitting up /sci/. If you guys would stay in your own board instead of coming here to disprove global warming or whatever other stupid shit you guys do the mentions would pretty much drop to zero.

>>6929309
Nope, /lit/ is a good board. Smart people who want to discuss philosophy go there, while dumb people go to /sci/ and /x/.

>>6929697
No, /pol/'s shitposting spreads, it is hardy a containment board. And then you have Normie who get fed up of nigger/kike spam and say >Back to /pol/

/pol/ really has the worst shitposters.

>>6929721
>Said by a /pol/esmoker as they were in the process of going to another board to talk about /pol/.

>>6929741
lol'd

>>6929739
>MUH FEELINGS MUHFUGGA

This is awesome. I tip my fedora to you op

>>6929208
iirc it's the third most popular one
/b/
/v/
/pol/

>>6929329
Yeah but therein lies the rub.

It's not really possible to fake being autistic, because if you think being "autistic" is in any way cool or desirable, you probably ARE at least slightly autistic.

>>6929730
Are you seriously implying me making one comment about /pol/ is going to throw off this statistic?

>>6929099

/fit/itizen here

/3/ is used often when sharing stats. For example
> "I lift 2/3/4 plaets"

This does show that /fit/ is filled with skeletons though.

>>6930021
Thank you for explaining. I guess that'll get weeded out along with the /soc/ stuff.

>>6929658
thanks man, you rock

is there any specific order to this?

>>6930069
Order it appears in the data table. So if there is any order, it would be by the board it was on alphabetically and then the post number numerically.

>>6929692
After looking into it, the only board that would give 404s to thread data requests is /b/. Delayed thread pruning exists for all other boards.

Now I just need to find out how to get the post numbers of archived threads. Then I'd be able to capture every post except on /b/ or /f/.

>>6930079
do you scrape any kind of frequency along with the names? it would be interesting to see some kind of word cloud for each board's -fag names

>>6929692
is there actually anything in those .Rdata files? I tried loading one and I'm not seeing any objects actually loaded into my workspace

>>6930109
nevermind, i'm a fucking idiot

>>6929692
>>6930109
You might need the 'data.table' package to get them to do certain things. It's a wonderful package that (I think) uses hash tables for subsets and tries to copy objects as rarely as possible. It makes everything faster and easier to read.

>>6930163
nah, you formatted them as a plain data table just fine, i was just being a moron.

nice job the scripts, by the way.

>>6930163
also i hope you don't mind but i'm curious enough about per-board usage and frequency that i'm writing up a script to compile that info

>>6930195
I shared because I wanted others to look into what they find interesting. Just remember to remove sticky threads. I'll throw my take on that into the dropbox.

>>6930095
I tried scraping first, and that's hell. 4chan can be an html beast. So I use the nicely structured (except for /f/, apparently) JSON API. I have all the entire comment from each copllected post (including html tags), so here you go:

http://pastebin.com/HD5khyDi

This is for the most recent week, since I reopened my R session.

>>6930195
>>6930205

Here are my functions:

post_frequency & remove_stickies.R
https://www.dropbox.com/s/mgfu6bmnk6mnpur/post_frequency%20%26%20remove_stickies.R?dl=0

>>6930209
ah! you rock, thanks.

This is very cool. Thanks for making it/sharing the data.

>>6930329
HAPPENING

the /soc/ -> /m/ connection is due to a/s/l threads

>>6930329
i love you moot

>>6930329
OH SHI-

>PhD in R
>any 4chan job I want
>$0k starting

>>6930448
Hot Pockets aren't a satisfactory form of compensation for you?

>>6930454
Do I look like a cat?

>>6930329
>>6930329
Hey Moot did you ever get that letter from /mlp/?

>>6930463
>Do I look like a cat?
You tell me.

>>6930473
Janny looks like cat

OP, if you ever update this data, you should try to account for URLs that commonly have "slash letter slash" in them like dropbox, exhentai, etc. And if possible, try to account for /soc/'s 18/m/cali posts lol.

pchea right, let's suck moot's amazing jew dick guys

>>6930484
>Buying this image
What's it like to live in retard land?

>>6930484

>>6930486
>What's it like to live in retard land?
PUT YOUR CAPCODE BACK ON JEWT

>>6930488
Fuck off moot you fucking kike.

>>6930486
amazing reply buddy, moot's then-gf works for silverstring and is a big thing there and he wanted to make a deal for some shitty social network as an evolution of 4chan, so he didn't want the bad name from gamergate

if you didn't know this by know you probably are too retarded to even consider it right now

>>6930488

>>6930488
you know how to play it cool abusing your position easily

no way to take you on, then. it's fine knowing you're awful and the people who mattered realized long ago. the board is filled to the brim with children and complete retards who want to belong in the internet club 4chan became

you probably like it better this way anyway

>>6930489
>>6930488
oy moot, how cum you don't post so much anymore? Also,I can't say these stats really surprise me, although /g/ being so popular is rather unexpected. Guess hot pockets are back in season.

>>6930494
moot can /v/ have dubs back

>>6930496
fuck off then.
no one will miss you

>>6930491
>>>/pol/

>>6929189
shocker

>>6930502
>I-I hope m-moot will n-notice me n-now....

>>6930488
Moot you fucking jew, because of you and your nazi mods, every single board has sjw sleeper agents in every thread.

You ruined the community we formed and tried to change it into something its not. You sold out and now the last bastion of free speech is gone. 4chan was literally the only place that could stand up to the social justice warriors on tumblr.

And now look at them, they're raiding every website possible with their shit and ruining every scene. They are doxxing and threatening innocent people and then they're defended all across the mass media.

look at what you've done. Was selling out to gawker worth it you fucking jew bastard?
You look away a lot of peoples homes, this was our only place to go.

That's an awful lot of effort for something you're not even going to publish.
Interesting though.

>>6930509
I haven't laughed this hard in awhile, thank you anon.

>>6930488
moot kill yourself
love the ghost of snacks
also,
>>>>>/new/

>>6930509
That's so much bullshit. There's 10 trolls that use SJW shit for every 1 SJW out there

>>6930329
moot pls tell Scooby to come back to /fit/, you're the only one powerful enough to convince him...

>>6930509
Is it just me or does /sci/ have the highest concentration of schizophrenics?
Complaining about SJW pandering is one thing, but all this? Wat?

>>6930509
Did some "ex-mod on /d/" tell you that?

>>6930519
>That's so much bullshit. There's 10 trolls that use SJW shit for every 1 SJW out there
The people whining about whiny people are themselves whinier.

It's the ultimate irony. Also a tongue-twister?

>>6930522
Welcome to a day in my life/what 4chan has become/what people actually believe.

>>6930525
moot please make a safe for work /r/ board
/r/ is just /b/tards asking source for generic tumblr porn gif

>>6930497
/g/ is pretty much just a consumer electronic's board. Not much about tech there anymore

>what ssd should I get
>what's your fav microsoft product
>take a screenshot of your desktop
>take a screenshot of your phone
>take a screenshot of your browser
>you're inferior because of your monitor
>take a photo of your custom keyboard
literally just read it off there right now

Bring back /prog/!
I admit, I stopped visiting much, but at one point it had a really good community.

>>6930527
>>6930525
Uhuh, tell me more about how threads about this shit aren't constantly deleted.

Tell me more about people NOT being banned for having a non politically correct opinion.

Fucking jew, worst shitstain on planet earth. You ruined a whole community.

>>6930522
>whaaat? oh my god i live under a rock whaaat!?

>>6930527
>children please help foster my delusion, make me feel better, I know I don't need thinking people when i have retards like you to follow me and say nice things!

a lot of /hm/ in this thread

/quest/ when

And History allow on /sci/ please

>>6930540
fuck it must be hard being moot, everyone that uses something you created hates you since I can remember.

Just want you to know moot, this thing you made had a huge effect on me in my childhood. It was probably for the worse, but a significant one effect none the less.

>>6930545
>haha im superior to everyone look at me! haha

>>6930527
Well, it is the cream of the crop of the insane.

>>6929099
Why is /hm mentioned so much here? Is science full of closet homosexuals?

>>6930549
>you're great moot! you're also bad, but you're great in a bad way! hehe!

i can keep going on, see the shit you pander to?

>>6930329

>>6930544
The threads were probably deleted from /v/ because video games.
Threads were probably deleted from other boards because moot is afraid of raids these days for some reason.

>>6930554
I've never seen it referenced.
Because engineers browse /sci/ maybe?

>>6930549
Maybe he shouldn't be a fucking faggot that sells out, removes mods and then enforces a shit tier rhetoric approach by just deleting any decenting opinion. Then maybe people wouldn't fucking hate him as much.

A good leader is never known, and is never recognized. That way we have the freedom to believe in the cause fully without any doubts.

>>6930554
Because it's a common meme here that /hm/ is the "homework" board, hence people who post their homework are told to go >>>/hm/.

Like the /sci/ equivalent of "get >>>/out/"

>>6930557
threads were deleted all around for nonsensical bullshit, for being non politically correct, for bothering his interests to sell and his sjw gf/friends

it's not just the retards who were obviously spamming

>>6930329
Do you love science and math?

>>6930566
we all love science, we're NERDS! haha GEEK is the new COOL!

>>6930549
>Just want you to know moot, this thing you made had a huge effect on me in my childhood.
Me too.
Although almost certainly for the better.
If I wasn't here I'd be on Reddit or some other cancerous backwater.

>>6930572
and more internet club shit

MORE INTERNET CLUB SHIT

4CHAN ISNT SUPPOSED TO BE YOUR FUCKING CLUB TO HAVE SHIT FLINGING FIGHTS WITH OTHER INTERNET CLUBS LIKE REDDIT FUCKING SHIT EATING MORON

what the fuck did you do to 4chan moot?

>>6930578
are you mentally ill, or medicated by any chance?

>>6930580
you're the water that makes up that cancerous backwater you mention, fucking brat

>>6930578
It's not a club.
It's just a website I visit where everybody has an equal say and almost nobody attention whores.
Far better than other sites that ban you for insulting people or encourage you to parrot the common opinion so you can collect worthless points.

Quit being butthurt that your favourite non-4chan website is cancer.

>>6930578
Excuse me, sir, but you seem to have acquired an injury on your anus. It must be very painful.

>>6930578
you sound new m8
we don't have accounts and this place is for free speech.

take a fucking chill pill, you sound /megabutthurt/

>>6930528
Also /a/ needs a sticky directing to the board wiki, it will not stop all retards from making rec threads and it will cause shitposting at first but it will help some people too stupid to google it
That combined with sfw /r/ and people will have no excuses to make rec threads on /a/, or other boards anymore and no one will feel bad if their threads get shitposted

>>6930578
>>6930581
Calm down my fellow Redditor.

>>6930583
you can't be this dense

the point is 4chan is shit just like reddit and you're not supposed to have a feeling of belonging, you do not talk about it, you don't feel "welcome", you are fucking anonymous

i shouldn't have to explain the rules to you and what they mean. you just like the 4chan club and came to be edgy and cool.

>>6930205

>>6930586
see >>6930587
also fuck you

>>6930587
I never said I felt like I belonged.
What the shit are you snorting?

>>6930589
forgot to mention that's a logarithmic scale.

>>6930591
I said calm down

>>6930595
i said fuck you.

enjoy your club with other shitheads that think just like you and the crowd mentality of believing you're going against the crowd while stile gathering and parroting the same shit, which is "we're not parroting shit like reddit lol!"

no one can revert what happened to this shithole

>>6930589
Is this a representation of what -fag words are said on 4chan and how common they are?
Pretty good

>>6930599
Calm down my man.

>>6930599
Why did you assume I was being offensive when I said you were a redditor? I was serious
You should honestly calm down

>>6930527
hey moot tell your mods to stop posting CP on /sp/

Goddammit a perfectly good thread ruined

>>6930608
w-what?

>>6930615
/sp/ mods use spambots that post CP in off-topic threads to discourage off-topic posting.

even if it's not the mods work, theres still a big spam/cp problem on /sp/ that only occurs in off-topic threads

>>6930561
maybe he just doesn't like you.

we are the abused pitbulls

one day you will let us outside, and will immediately regret it

>>6930527
Put wordfilters on /int/

>>6930527
>what people actually believe
confirmed for blue pilled retard. no no, people aren't being manipulated from kindergarten to believe in social marxism and degeneracy

n-n-nothing to s-s-see here

>>6930658
/pol/ please go

>>6930484
>/pol/ cannot take a joke

You guys are no fun at all.

>>6930488
I regret driving all the way to Atlanta just to meet you last year.

>>6930666
Nice satan trips

>>6930527
I will always love you moot.

>>6930488
>>6930489
>>6930527
What the fuck you were doing at XOXO Fest though? That's all I want to know

>>6930527
moot, have you taken care of the Goodra shitposter on /vp/ yet?

>>6930672
Being a SJW shill as usual.

>>6930672
>What the fuck you were doing at XOXO Fest though? That's all I want to know
Because I spoke in 2012 (https://www.youtube.com/watch?v=BxRNRi7bOSI)), like Portland, and enjoyed the previous two iterations.

>>6930679
>Being a SJW shill as usual.
Er, nevermind—what he said. Shills gotta shill, ya know.

>>6930689
I LOVE YOU NO MATTER WHAT YOU DO MOOT

>>6930689
You get a lot of shit. But you don't deserve it. I love you moot.

>>6930689
shut up newfag

>>6930689
How come /pol/ has a billion wordfilters all of a sudden?

Just curious.

>>6930689
have you ever thought of using those who follow you for any purpose?

your one of the most influential people alive in a way, and yet you dont capitalize on it. We clearly dont make you money, or bring much happiness. Why not just use the tide for fun?

>>6930689
loyal consumers have to loyal comsume

How the fuck does /g/ get mentioned so much still?

>>6930693
>>6930696
fuck you faggots how about you post something with a meaning instead of sucking his dick and making a million "I loevy ou <33) posts? Not that he will respond to anything else

>>6930698
Fun things are fun.

>>6930698
This actually has my interest.

Also what are all the exact words filtered?

>>6930704
I have nothing to say other than what a qt.3.14 he is

Eat shit nigger

>>6930708
No one gives a shit
>>>/lgbt/

>>6930689
>TFW I baited moot into replying to me

>>6930689
Off topic, but do you got any idea on how to stop people with VPNs from shitting up the place? There's like 5 dudes on /v/ who literally do nothing but shitpost all day every day and evade every ban given to them with free VPNs. If it ain't the weird dude who is into that one Pokemon, it's Ruggarell and his umpteenth variation on "What video games do moms play?" It's a serious problem, and I can't even filter most of it out because they keep changing their bait around.

>>6930707
N**gger is Cra**r

GTKRWN is now GTK Love and tolerance now.

Censor is enrich

jew is happy merchant and jewess is happy merchantess.

/pol/ is tumblr and tumblr is /pol/

And cuck is cluck.

Those are the ones I have figured out so far. Pretty funny.

>>6930698
MUH FREEDOM

>>6930689
Did you get lot of shit from other people when the celeb leaks happens on /b/?

>>6930713
moot censored the word cuck?

musta hit a little close to home for him

>>6930689
Hey moot, is it true you don't go to /co/ / scan over /co/ that often?

>>6930712
>Off topic, but do you got any idea on how to stop people with VPNs from shitting up the place?
There's really no way to deal with it other than what we already do, which is to [range]ban proxies, etc.

>>6930488
Fucking seriously, tumblr gifs?
How the unmighty have fallen

>>6930719
are range/location bans permanent or simply a temporary thing because there might be some innocent users who get range-banned in the process.

>>6930719
Why don't you call the ragebans?

>>6930726
>there might be some innocent users who get range-banned in the proce
I can confirm this happening. Before I lived in Canada, I was in India (inb4 immigrant jokes), and I remember getting banned randomly one time on /g/ (inb4 microsoft jokes) with the reason being ''proxy''.

People from other countries browse 4chan too!

>>6930719
hey moot, why do people hate fun here (4chan, not /sci/ specifically)?

>>6930742
you can browse when using a proxy (was not always the case) you may not be able too post though
I've ran into that many times on mobile network (some /sp/-fag that loves to shitpost)