Week 2, day 1
HW2: Data Visualization
Hao-Chien Wang
Some variables
n: number of times a word is used in a source of text.n_shift: normalizedn. Since the numbers of words in total from different sources are different, it must be normalized.exNum: Number of sentence examples provided bydictionary.com
I use the words collected (with crawlers) from three of the largest forums in stackexchange (arqade, askubuntu and English grammar & usage) and look up those words in dictionary.com and count the number of example sentences using the crawler. First, I load the data and try to plot some simple graph:
library(dplyr)##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, union#English
data_eng = read.csv("~/Documents/summerproj/data_science_programming/week_1/day2/dictionary/dictionary_com_output_english_20000.csv", header = TRUE )
data_eng = filter(data_eng, exNum >= 0)
data_eng$n_shift = ( (data_eng$n - mean(data_eng$n) ) / sd(data_eng$n) ) * 0.2 + 1
data_eng$n = ( (data_eng$n - mean(data_eng$n) ) / sd(data_eng$n) )
data_eng$word_source = "english"
#arqade
data_arqade = read.csv("~/Documents/summerproj/data_science_programming/week_1/day2/dictionary/dictionary_com_output_gaming.csv", header = TRUE )
data_arqade = filter(data_arqade, exNum >= 0)
data_arqade$n_shift = ( (data_arqade$n - mean(data_arqade$n) ) / sd(data_arqade$n) ) * 0.2 + 1
data_arqade$n = ( (data_arqade$n - mean(data_arqade$n) ) / sd(data_arqade$n) )
data_arqade$word_source = "arqade"
#askubuntu
data_ubuntu = read.csv("~/Documents/summerproj/data_science_programming/week_1/day2/dictionary/dictionary_com_output_askubuntu.csv", header = TRUE )
data_ubuntu = filter(data_ubuntu, exNum >= 0)
data_ubuntu$n_shift = ( (data_ubuntu$n - mean(data_ubuntu$n) ) / sd(data_ubuntu$n) ) * 0.2 + 1
data_ubuntu$n = ( (data_ubuntu$n - mean(data_ubuntu$n) ) / sd(data_ubuntu$n) )
data_ubuntu$word_source = "askubuntu"
#combine
data = rbind(data_ubuntu, data_eng, data_arqade)
head(data)## X word n exNum n_shift word_source
## 1 1 the 137.63091 18 28.52618 askubuntu
## 2 2 to 88.42767 24 18.68553 askubuntu
## 3 3 and 53.62696 18 11.72539 askubuntu
## 4 4 a 52.40956 34 11.48191 askubuntu
## 5 5 i 51.77989 2 11.35598 askubuntu
## 6 6 you 47.04576 8 10.40915 askubuntu#plot
library(ggplot2)
ggplot(data, aes( x = exp( - 5 * n_shift ), y = exNum, color = word_source)) + geom_point()
I use e^{-5 \times n\_shift} as the x-axis to rescale n so I can have a better view. It can be noticed that there’s a horizontal line, while most of the dots are above the line, which means most of the words have at least some numbers of examples. I want to find that line, so I plot the count of different exNum:
ggplot(data, aes( x = exNum, )) + geom_bar() + facet_grid(word_source ~ .)
The result is not so satisfying because the number of words with exNum = 0 is overwhelmingly large. I print them out to find the reason:
head(filter(data, exNum == 0, word_source == "askubuntu"), n = 20)## X word n exNum n_shift word_source
## 1 118 doesn't 3.1174005 0 1.623480 askubuntu
## 2 151 i've 2.4134422 0 1.482688 askubuntu
## 3 212 http 1.7429120 0 1.348582 askubuntu
## 4 226 ctrl 1.6660053 0 1.333201 askubuntu
## 5 228 can't 1.6504929 0 1.330099 askubuntu
## 6 245 didn't 1.5049819 0 1.300996 askubuntu
## 7 358 isn't 0.9884836 0 1.197697 askubuntu
## 8 439 i'd 0.7730575 0 1.154611 askubuntu
## 9 499 usr 0.6620671 0 1.132413 askubuntu
## 10 532 i'll 0.6190256 0 1.123805 askubuntu
## 11 537 cpu 0.6124711 0 1.122494 askubuntu
## 12 713 haven't 0.4121204 0 1.082424 askubuntu
## 13 820 releases 0.3354322 0 1.067086 askubuntu
## 14 856 unix 0.3177349 0 1.063547 askubuntu
## 15 860 couldn't 0.3166425 0 1.063329 askubuntu
## 16 905 php 0.2917352 0 1.058347 askubuntu
## 17 926 uninstall 0.2827773 0 1.056555 askubuntu
## 18 959 wasn't 0.2670464 0 1.053409 askubuntu
## 19 996 hdd 0.2519710 0 1.050394 askubuntu
## 20 1075 shouldn't 0.2216016 0 1.044320 askubuntuhead(filter(data, exNum == 0, word_source == "english"), n = 20)## X word n exNum n_shift word_source
## 1 131 doesn't 4.0350277 0 1.807006 english
## 2 151 i've 3.3936282 0 1.678726 english
## 3 200 can't 2.3939633 0 1.478793 english
## 4 212 i'd 2.2550629 0 1.451013 english
## 5 213 isn't 2.2529760 0 1.450595 english
## 6 309 didn't 1.5605613 0 1.312112 english
## 7 443 i'll 1.0369605 0 1.207392 english
## 8 455 http 0.9831627 0 1.196633 english
## 9 464 they're 0.9588145 0 1.191763 english
## 10 756 wasn't 0.5697081 0 1.113942 english
## 11 858 haven't 0.4885477 0 1.097710 english
## 12 878 couldn't 0.4713880 0 1.094278 english
## 13 1294 shouldn't 0.2842552 0 1.056851 english
## 14 1365 pre 0.2698782 0 1.053976 english
## 15 1457 webster's 0.2478490 0 1.049570 english
## 16 1529 ing 0.2337039 0 1.046741 english
## 17 1620 cf 0.2156167 0 1.043123 english
## 18 1935 odo 0.1694712 0 1.033894 english
## 19 2043 downvote 0.1560217 0 1.031204 english
## 20 2175 hasn't 0.1434998 0 1.028700 englishhead(filter(data, exNum == 0, word_source == "arqade"), n = 20)## X word n exNum n_shift word_source
## 1 81 i've 3.9459622 0 1.789192 arqade
## 2 121 can't 2.6753765 0 1.535075 arqade
## 3 123 doesn't 2.6277880 0 1.525558 arqade
## 4 229 didn't 1.4474279 0 1.289486 arqade
## 5 263 isn't 1.2771546 0 1.255431 arqade
## 6 301 i'd 1.1085317 0 1.221706 arqade
## 7 333 haven't 0.9679668 0 1.193593 arqade
## 8 338 minecraft 0.9533877 0 1.190678 arqade
## 9 344 i'll 0.9393587 0 1.187872 arqade
## 10 373 they're 0.8571103 0 1.171422 arqade
## 11 517 http 0.5993622 0 1.119872 arqade
## 12 795 dps 0.3614196 0 1.072284 arqade
## 13 806 wasn't 0.3556430 0 1.071129 arqade
## 14 915 couldn't 0.2995271 0 1.059905 arqade
## 15 1163 pre 0.2205796 0 1.044116 arqade
## 16 1173 shouldn't 0.2183790 0 1.043676 arqade
## 17 1180 they'll 0.2172787 0 1.043456 arqade
## 18 1239 cooldown 0.2065507 0 1.041310 arqade
## 19 1358 it'll 0.1823438 0 1.036469 arqade
## 20 1361 civ 0.1817936 0 1.036359 arqadeAs we can see, there are several cases in the words with exNum = 0:
- common english words that doesn’t have a dictionary page, e.g. doesn’t, isn’t.
- words in the original html files that was not filtered out during the crawling process, e.g. http, www.
- words related to specific data source, e.g. mincraft(arqade), usr(askubuntu), unix(askubuntu)
Since we seldom look up those kinds of words in the dictionary, I filtered them out and plot again:
data = filter(data, exNum != 0)
ggplot(data, aes( x = exNum, )) + geom_bar() + facet_grid(word_source ~ .)
#zoom in at smaller exNum
ggplot(filter(data,exNum < 25), aes( x = exNum)) + geom_bar() + facet_grid(word_source ~ .)
Here, we can see the difference between different data sources. Most words in askubuntu and arqade have at least 10 examples, while there is a number of words in the english forum that have 5 ~ 10 examples. However, this doesn’t necessarily means that users in the english forum use words that are more rarely used, as shown in the next part.
Lastly, I examine the the behavior of exNum when n become small, since we normally look up the dictionary for less used words. I calculate the count of words which exNum < 5.
#use new data from the English forum in stackexchange since I want all n to be positive and unscaled again
data_new = read.csv("~/Documents/summerproj/data_science_programming/week_1/day2/dictionary/dictionary_com_output_english_20000.csv", header = TRUE )
data_new = filter(data_new, exNum > 0)
#rescale n
data_new$log_n = floor(log(data_new$n)/0.3)*0.3
#head(arrange(select( data_new,exNum , n_cut, exNum, word), n_cut, word) , n = 100)
#count the number of data which has less than 5 examples
data_n = filter( data_new, exNum < 5 ) %>%
count( log_n, sort=TRUE )
#count the total number of data
data_n_total = data_new %>%
count( log_n, sort=TRUE )
#calculate the ratio. The higher the ratio is, the more words with this value of n have less than 5 examples
n_ratio = merge(data_n, data_n_total, by = "log_n")
n_ratio$ratio = n_ratio$n.x / n_ratio$n.y
head(n_ratio, n = 20)## log_n n.x n.y ratio
## 1 0.0 370 1592 0.232412060
## 2 0.6 1690 9606 0.175931709
## 3 0.9 946 5997 0.157745539
## 4 1.2 597 4424 0.134945750
## 5 1.5 717 6199 0.115663817
## 6 1.8 410 4254 0.096379878
## 7 2.1 386 4399 0.087747215
## 8 2.4 254 3233 0.078564800
## 9 2.7 269 4475 0.060111732
## 10 3.0 197 3493 0.056398511
## 11 3.3 134 3003 0.044622045
## 12 3.6 102 2884 0.035367545
## 13 3.9 62 2318 0.026747196
## 14 4.2 62 2059 0.030111705
## 15 4.5 45 1696 0.026533019
## 16 4.8 24 1432 0.016759777
## 17 5.1 19 1203 0.015793849
## 18 5.4 12 1001 0.011988012
## 19 5.7 7 773 0.009055627
## 20 6.0 9 676 0.013313609#plot
ggplot(n_ratio, aes( x = log_n, y = ratio)) + geom_line() 
The ratio is larger when n is either very large or very small, which indicates that when a word is commonly used or very rarely used, there is a larger chance that you will find fewer examples in dictionary.com. Therefore, we can conclude that the dictionary has attempted to add more examples to the words that are not so commonly used, but it is still difficult to find examples for those rare words.