{"33": {"Name": "test 1", "Details": "test 1 ", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"33.1": {"Name": "test 1", "Details": "test 1 ", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "33", "snippets": [{"Details": "", "PreExCode": "", "Query": "#Lets create 3 vectors with title, author and year.\ncolor <- c('Red','Blue','Yellow','Green')\n\n#Lets look at how the created vectors look.\ncolor", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "33", "snippets": {}}, "32": {"Name": "Test section", "Details": "

Test section

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "32", "snippets": [{"Details": "test ", "PreExCode": "", "Query": "#Lets create 3 vectors with title, author and year.\ncolor <- c('Red','Blue','Yellow','Green')\n\n#Lets look at how the created vectors look.\ncolor", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "#Lets create 3 vectors with title, author and year. \ncolor <- c('Red','Blue','Yellow','Green') \n#Lets look at how the created vectors look. \ncolor", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "0": {"Name": "Data101 Active Textbook", "Details": "

\n \n

Tomasz Imielinski

Rutgers University

\n \n

Abstract

Active textbook allows readers to interact with the book content. This interaction takes place through executable snippets of code as well as testing facilities such as question roulette and short coding assignments . Thus, an active textbook engages the student in so many more ways than a traditional book. It invites students to code and test themselves as part of the textbook.

Key concepts of active Textbook

\n
Snippets
\n
\n
Test yourself
\n
\n
Data Puzzles and Coding Roulette
\n

\n\n

The most important aspect of this interactive book are the interactive code chunks for running code, which are powered by a minimal version of Datacamp\u2019s learning interface called \u201cDatacamp Light\u201d. For more info visit Datacamp Light

\n
Note: This book uses Datacamp Light for supporting runnable code chunks. In case the code chunks do not connect to run-time session, please copy the code and run in RStudio. Also, please report if facing this issue to the instructor via email.
\n

Acknowledgement: This textbook would not have been created without extensive help from Devarsh Shah, Param Shah and Aditya Sehgal

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "0", "snippets": {}}, "1": {"Name": "Introduction", "Details": "The objective of this textbook is to provide you with the shortest path to exploring your data, visualizing it, forming hypotheses and validating and defending them. In other words, to introduce you to data science. We call it an active textbook, since students can interact with the book by running and modifying snippets of R code. Students can also test themselves using Query and Code Roulette - on questions and simple coding tasks. Thus, an active textbook interacts with its reader, helps to run code and also asks students questions and gives them simple coding tasks. Concepts which we discuss in the book are widely covered on the web with countless youtube video tutorials. We briefly introduce the basic concepts here as well, but our focus is on problem solving and simple coding. In other words, honing the skills to do something with the data, not just talk about it. To learn a concept you have to know how to code it. To put it bluntly, no coding, no learning!\n\nActive textbook is data-centric. We stress that prior to data analysis and exploration, you have to know your data. Paraphrasing the famous saying that real estate is about \u201clocation, location and location\u201d, data science is about data, data and data. Using numerous data sets we guide the student through the process of getting acquainted with their data. We call these data sets - data puzzles, since each of them is synthetically created and has hidden patterns embedded in the process of data generation. For each of our data puzzles we show the process of getting familiar with the data beginning from simple scripts called queries and proceeding through as-hoc hypothesis testing as well as Bayesian reasoning. As we said, each data puzzle hides interesting and non-trivial patterns which are to be discovered. This process of discovery makes data science similar to the work of a detective. Discoveries range from grading methods of Professor Moody, factors influencing quality of a party, voter profiles in local town elections or quality of sleep determinants.\n\nGiven a data set, you want to be able to make any plot you wish, find plots which show something actionable and interesting, explore data by slicing and dicing it and finally present your results in a statistically convincing manner, perhaps in a colorful and visually appealing way. Finally, you will be able to apply some basic machine learning methods to build, train and test prediction models. All of this will be accomplished in a succinct and crisp way using a small subset of R instructions.\n\nWe assume no prior programming background. We will teach you as little R as necessary to achieve the goals of this book: explore the data, visualize it, verify hypotheses and build prediction models. Thus, you will be able to do a good chunk of work which data scientists do. We will accomplish this goal through active snippets of executable code. These are examples of R code (around 100 executable snippets of code) embedded in the textbook itself. More importantly, you will be able to modify the code and execute the modified code without need to install any application on your machine. This will allow you to understand the code in the book through the \u201cwhat if\u201d exploratory process. Thus, every code snippet is just an invitation to endless modifications. This is why we call this textbook - active.\n\nAnother unique aspect of this textbook is its reliance on data puzzles. These are synthetic data sets with embedded patterns and rules generated by our tool called DataMaker. We will present our data puzzles (dynamic list, may vary from year to year) in section 8 following introduction to plots, then we will proceed to freestyle data exploration. This will allow us to learn more about our data, form the leads, and finally state our hypotheses. We will follow up by an elementary introduction to hypothesis testing through a permutation test. We will learn how to calculate p-values and how to use them to defend our findings against the randomness trap. This will be particularly important in case of multiple hypotheses when one has to be particularly careful to avoid \u201cfalse positives\u201d . We will introduce Bayesian reasoning and learn how to compute posterior odds of a belief given an observation. All these important concepts will be introduced via executable snippets of code and \u201cwhat if\u201d practicing. We will then enter the key section of the book - the data puzzles section. In the data puzzle section, for each data set we will go through the process of getting to know the data and using the concepts learned so far by executing code tailored to each of the data puzzles.\n\nIn the second part of the book we discuss prediction models. We focus on decision trees (rpart() - recursive partitioning) and linear regression. But we also show how to use other machine learning methods from the rich R-library. We go over cross-validation and show how to build prediction models which combine multiple machine learning models. We stress the importance of knowing your data first, instead of just blind application of machine learning packages. Humans in the loop is very important and prior data exploration and visualization leads to improved quality prediction models. Students can practice prediction model building on especially prediction snippets to make themselves prepared for Kaggle based prediction challenge competition which takes the last months of the data 101 class. The last leaderboard of 2022 challenge is presented here LeaderBoard .\n\nWe will use as few R functions as possible to achieve our goals. In fact we will demonstrate how using less than ten R functions is sufficient for us. In the appendix, we show many more useful commands of R which eventually you would have to use. However, our goal in this short textbook, is to present the shortest path to data analysis which will let you import the data, plot it, make some analysis yourself and use R-libraries to build machine learning models. In this textbook and in this class we do not teach how to clean the data (data wrangling) and how to deal with a wide variety of data types. We also do not address complex data transformations such as multi-frame operations like merge function. We also do not explain how different machine learning methods work, we only show you how to use them. It is similar to teaching one how to drive a car without knowing how a car engine works.\n\nSections 2.5 and 2.6 provide the lists of all concepts which we cover in our active textbook and all R functions which are needed. Notice how small the set of R functions is. It is important for programming novices to start small and also see how far this small set of functions can get you.\n\nOur question roulette allows self-testing on nearly 100 questions relevant to the material. Each question is answered, but students are encouraged first to answer questions themselves and only then follow it with checking the correct answer. The code roulette, on the other hand, consists of around 100 of simple common data science coding tasks.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"1.1": {"Name": "\ud83c\udfa6 Data 101 for Instructors \ud83c\udfa6", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1k4m-dts_Y5hlyxTMPbPAVdXa0Drf_OtI/preview", "child": {}, "parent": "1", "snippets": {}}, "1.2": {"Name": "\ud83c\udfa6 Examples of data 101 assignments \ud83c\udfa6", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1PuyGKbroUALo-i0A9ngvycJ7En38mOqn/preview", "child": {}, "parent": "1", "snippets": {}}, "1.3": {"Name": "Data 101 Intro", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "1", "snippets": [{"Details": "", "PreExCode": "", "Query": "#Assume that we end up with Q quizzes and H Homeworks and E exams. Assume that Homeowrks and exams are heavy weighted (2x). Lets compute the initial weights. The equation Qx + 2Hx + 2Ex = 100, leads to x =100/Q+2H+2E\nQ=10\nH=5\nE=2\n#what are the weights?\nx<-round(100/(Q+2*H+2*E),2)\ncat('Weight of each quiz is' , x, \"\\n\" )\ncat('Weight of each homework and exam is', 2*x)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "1.4": {"Name": "GradeFlex", "Details": "GradeFlex introduces a novel grading approach, currently in the prototyping phase. It empowers students to customize the weights of specific grading components, such as quizzes, homework, and exams. By increasing the weights where they excel and decreasing them where they perform less favorably, students personalize their grading weights, potentially resulting in distinct weight distributions for each individual to optimize their scores. Students start with initial default weights and adjust them to adjusted weights - to their advantage. \n\n GradeFlex is also dynamic; initial default weights adapt based on the quantity of grading items. For instance, if there are 10 quizzes, 2 exams, and 6 homework assignments, the initial weights differ from a scenario with 5 quizzes, 1 exam, and 6 homework assignments. This flexibility liberates instructors from specifying the number and weights of grading items at the semester's onset, providing a departure from static and often challenging-to-anticipate grading structures, particularly for unpredictable elements like quiz quantities.\n\nGradeFlex features two types of weights: regular weight for grading items and heavy weight (typically double the regular weight) for select items, as illustrated in the following snippet. It's important to note that the Canvas score computation will not be applicable in this context. \n\n\nAfter the semester is completed, and you get the initial score, you can start modifying weights. You can increase the weight of the grading item that you did well on and decrease the weight of the one you did poorly. These increases and decreases have to be within the limits - probably between 25-50%. And the TOTAL sum of weights always has to be equalt to 100%. You will use our UI to submit your selected weights to the TA. TA will verify it and your final score will be the basis of your final grade. ", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "1", "snippets": {}}, "1.5": {"Name": "Important References", "Details": "- Statistical Learning with Applications in R
\n- Statistical Learning with Applications in Python (just published in July, 2023)
\nHere is the site where you can download pdfs of the two textbooks mentioned above:\n\n An Introduction to Statistical Learning (https://www.statlearning.com/) \n\n\nAdditional resources:\nhttps://www.crumplab.com/statistics/\n\n What Is Data Literacy", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "1", "snippets": {}}}, "parent": "1", "snippets": {}}, "2": {"Name": "Setting Up R", "Details": "

\n
Important Instructions
\n
- Installation of R is required before installing RStudio
  - \u201cR\u201d is a programming language, and,
  - \u201cRStudio\u201d is an Integrated Development Environment (IDE) which provides you a platform to code in R.
  \n
\n
\n
How to download and install R & RStudio?
\n
- \n
  Downloading and installing R.
  \n
  - For Windows Users.
    - Click on the link provided below or copy paste it on your favourite browser and go to the website.
      - https://cran.r-project.org/bin/windows/base/
      \n
    - Click on the link at top left where it says \u201cDownload R 4.0.3 for windows\u201d or the latest at the time of your installation.
    - Open the downloaded file and follow the instructions as it is.
    \n
  - For MAC Users.
    - Click on the link provided below or copy paste it on your favourite browser and go to the website.
      - https://cloud.r-project.org/bin/macosx/
      \n
    - Under \u201cLatest release\u201d, click on \u201cR-4.0.3.pkg\u201d or the latest at the time of your installation.
    - Open the downloaded file and follow the instructions as it is.
    \n
  \n
- \n
  Downloading and installing RStudio.
  \n
  - For Windows Users.
    - Click on the link below or copy paste it in your favourite browser.
      - https://rstudio.com/products/rstudio/download/
      \n
    - Scroll down almost till the end of the web page until you find a section named \u201cAll Installers\u201d.
    - Click on the download link beside \u201cWindows 10/8/7\u201d to download the windows version of RStudio.
    - Install RStudio by clicking on the downloaded file and following the instructions as it is.
    \n
  - For MAC Users.
    - Click on the link below or copy paste it in your favourite browser.
      - https://rstudio.com/products/rstudio/download/
      \n
    - Scroll down almost till the end of the web page until you find a section named \u201cAll Installers\u201d.
    - Click on the link beside \u201cmacOS 10.13+\u201d to start your download the MAC version of RStudio.
    - Install RStudio by clicking on the downloaded file and following the instructions as it is.
    \n
  \n
\n

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"2.1": {"Name": "How to upload a data set?", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "2", "snippets": {}}, "2.2": {"Name": "Grades dataset", "Details": "

Download: moody2022.csv

Grades in Professor Moody\u2019s class data set.

Our working data set will be the Moody data set which stores data about students\u2019 grades in a large signature class taught by Professor Moody. The data set stores individual scores of students in class, their major, seniority and GPA. Data scientists may ask many questions such as, given the student\u2019s score in class, does the final grade depend on the major and/or student\u2019s seniority? For example, is it more difficult for computer science majors to earn an A, pass the class, than, say for students majoring in psychology? Does GPA play any role in grading? It should not - but maybe it does? We are still far away from being able to ask such questions, for now we will use Moody data set in code snippets which illustrates the core R functions which we will use in the active textbook.

\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n

Table 2.1: Snippet of Grades Dataset
	Major	Score	Seniority	GPA	Grade
48	Economics	50	Freshman	3.0	D
166	Statistics	76	Junior	4.0	C
516	Statistics	14	Freshman	4.0	F
122	Psychology	97	Senior	2.8	A
883	CS	97	Senior	4.0	B

\n
To upload the dataset/file present in csv format the read.csv() and read.csv2() functions are frequently used The read.csv() and read.csv2() have different separator symbol: for the former this is a comma, whereas the latter uses a semicolon.
\n
\n
There are two options while accessing the dataset from your local machine:
\n
1. To avoid giving long directory paths for accessing the dataset, one should use the command getwd() to get the current working directory and store the dataset in the same directory.
\n

$\"\"$ \n

Getwd

To access the dataset stored in the same directory one can use the following: read.csv(\u201cmoody2022.csv\u201d).

$\"\"$ \n

Store the moody dataset in the same directory

One can also store the dataset at a different location and can access it using the following command: (Suppose the dataset is stored inside the folder Data101_Tutorials on the desktop)

- For Windows Users.\n  - Example: read.csv(\"C:/Users/Desktop/Data101_Tutorials/moody2022.csv\")\n\n- For MAC Users.\n  - Example: read.csv(\"/Users/Desktop/Data101_Tutorials/moody2022.csv\")\n

Note: The directory path given here is the current working directory hosted on Github where the dataset has been stored.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "2", "snippets": [{"Details": "", "PreExCode": "", "Query": "# Read in the data\ndf <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n# Print out `df`\nhead(df)", "Result": "", "Title": "Reading csv data", "Type": "R", "Video": "", "Width": ""}]}, "2.3": {"Name": "Saving your work", "Details": "

To save your work go to File -> Save. It will ask you to give a name for your .R file and then click on Save.

$\"\"$ \n

Save

After making modifications to your saved file, you will need to save the file again. If the name of the file on the top is in Red Color indicates that the file have unsaved changes.

$\"\"$ \n

Unsaved File

Go to File -> Save to save your .R file again. After saving the file the color of the file name i.e. HW1.R will again change back to black.

$\"\"$ \n

Saved File

Note: You can create multiple files inside the same project such as for your each homework assignments

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "2", "snippets": {}}, "2.4": {"Name": "General R References", "Details": "

https://www.w3schools.com/r/
https://cran.r-project.org/doc/contrib/Short-refcard.pdf
https://www.amazon.com/Statistics-Engineers-Scientists-William-Navidi/dp/0073376337/ref=pd_lpo_3?pd_rd_i=0073376337&psc=1
https://data101.cs.rutgers.edu/laboratory/
Google for R
Data Science Central
Rutgers Data Science Club

\r\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "2", "snippets": {}}, "2.5": {"Name": "Textbook Concepts", "Details": "

\n
Hypothesis testing
\n
\n
Difference of means hypothesis testing
\n
\n
Null Hypothesis
\n
\n
Alternative Hypothesis
\n
\n
z-value
\n
\n
critical value
\n
\n
significance level
\n
\n
p-value
\n
\n
Bonferroni correction
\n
\n
Chi square test
\n
\n
Independence
\n
\n
Multiple Hypothesis testing
\n
\n
False Discovery Proportion
\n
\n
Contingency Matrix
\n
\n
Bayesian Reasoning
\n
\n
Prior odds
\n
\n
Posterior odds
\n
\n
Likelihood ratio
\n
\n
False positive
\n
\n
True positive
\n
\n
Crossvalidation
\n
\n
Decision trees
\n
\n
Linear regression
\n
\n
Recursive partitioning
\n
\n
MSE
\n
\n
Prediction accuracy
\n
\n
Training
\n
\n
Testing
\n

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "2", "snippets": {}}, "2.6": {"Name": "R101-R functions used in this class", "Details": "

\n
Elementary instructions: c() , mean() , nrow() , rep(), sd() , cut()
\n
\n
Plots: plot() , barplot() , boxplot() , mosaicplot()
\n
\n
Data Transformations: subset() , tapply() , table() , aggregate()
\n
\n
Library functions: chisq.test() , pnorm() , Permutation() , rpart() , predict() , lm() , crossvalidation()
\n
\n
Parameters of rpart: minsplit , minbucket , cp
\n

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "2", "snippets": {}}, "2.7": {"Name": "\ud83c\udfa6 Video - how to create data frame variable in R Studio \ud83c\udfa6", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1_R9NYSNDqg1WeyT8tUzag4JPxYzHyFbQ/preview", "child": {}, "parent": "2", "snippets": {}}}, "parent": "2", "snippets": {}}, "3": {"Name": "\ud83c\udfa6 Basic R Code - R101 Video \ud83c\udfa6", "Details": "In this section we introduce the absolutely basic R instructions, call it R101, which will be sufficient for the entire data 101 class. This is a very small subset of the entire R. The good news is that using this very small subset of R we can accomplish all coding objectives for data 101!\n\nWhy do we start with this specific set of functions? Because we would like to enable you to evaluate simple queries against data. Queries are \"coded questions\" about a data set (data frame). For example data set Moody[Major, Seniority, Score, GPA, Grade] stores student data, their scores in data 101 and their final grade. Here are some examples of quesitons which we will learn how to code and execute aganist the Moody data frame:\n\n1. What is the max score of computer sciece students who got a B in data 101?\n\n2. What is the mean score of psychology majors?\n\n3. What is the most frequent grade that freshmen get in data 101?\n\n4. Compare mean score of all majors who received A in data 101\n\n5. What is the distribution of grades among economics majors who scored at least 60 percent?\n\n6. What is the probability that computer science senior who scored more than 90% gets an A in data 101?\n\n7. What is the most likely grade that a junior who is stat major gets in data 101?\n\n8. What is the smallest score that A-student senior gets in data 101?\n\n9. Are there any students with GPA larger than 3.5 who failed data 101?\n\n10. Is mean score of students with high GPA (higher than 3.5) higher than the for the rest of the students.\n\n\nThe set we present below is a mix of simple arithmetic aggregate functions such as mean() 3.4.1, max() 3.4.3, sum() , basic data structures such as vectors and data frames and finally, two core functions defined for data frames: subset() 3.5, tapply() 3.6 and table() 3.3 function defined on vectors.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1yBlbfz7243JvMbAxSJJR8D6mECn6T6_n/preview", "child": {"3.0": {"Name": "Practice overview Video", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1n0el6zIruPmhrYCCF4znHU1flaFiSzsD/preview", "child": {}, "parent": "3", "snippets": {}}, "3.1": {"Name": "Vectors", "Details": "See the https://sparkbyexamples.com/r-programming/how-to-create-a-vector-in-r/ for full description ", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"3.1.1": {"Name": "Categorical Vectors", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "#Lets create 3 vectors with title, author and year.\ncolor <- c('Red','Blue','Yellow','Green')\n\n#Lets look at how the created vectors look.\ncolor", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.1.2": {"Name": "Numerical Vectors", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "#Lets create a vectors with numerical sequence.\nyear <- 2018:2022\n\n#Lets look at how the created vectors look.\nyear", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.1.3": {"Name": "Let's Chat about vectors!", "Details": "\n Can we review vectors in R?\n $\"ChatGPT$ \n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": {}}}, "parent": "3", "snippets": {}}, "3.2": {"Name": "Data Frames", "Details": "

See the documentation in https://sparkbyexamples.com/r-programming/different-ways-to-create-a-dataframe-in-r/

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"3.2.1": {"Name": "Data Frame Creation ", "Details": "

See the documentation in https://sparkbyexamples.com/r-programming/different-ways-to-create-a-dataframe-in-r/

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "# Load the dataset into the moody variable\nmoody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n# Now lets view the dataframe moody with just 5-6 tuples\nhead(moody)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.2.2": {"Name": "Simple data Frame Subsetting", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "# Load the dataset into the moody variable\nmoody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n# Return row 1\nmoody[1, ]\n\n# Return column 5\nmoody[, 5]\n\n# Rows 1:5 and column 2\nmoody[1:5, 2]\n\n# Give me rows 1-3 and columns 2 and 4 of moody\nmoody[1:3, c(2:4)]", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "3", "snippets": {}}, "3.3": {"Name": "Table", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"3.3.1": {"Name": "Table()", "Details": "For more detailed description - prompt GPT 4 with the prompt \"Describe table() instruction in R\"", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "# moody<-read.csv(\"../files/dataset/moody2020b.csv\") #static Load\nmoody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\") #web load\n\n#lets make a table for the grades of students and counts of students for each Grade. \ngrades <- table(moody$Grade)\n\n#Joint distribution of grade and major\ntable(moody$Grade, moody$Major)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.3.2": {"Name": "Chat about tables ", "Details": "\n Chat about tables\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": {}}}, "parent": "3", "snippets": {}}, "3.4": {"Name": "Basic Functions ", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"3.4.1": {"Name": "mean()", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n#Lets look at the mean of score column.\nmean(moody$Score)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.4.2": {"Name": "length()", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n#Lets look at the length of the grade column \nlength(moody$Grade)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.4.3": {"Name": "max()", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n#lets look at the maximum value of the score in the score column\nmax(moody$Score)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.4.4": {"Name": "min()", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n#Lets look at the minimum value of score in the score column.\nmin(moody$Score)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.4.5": {"Name": "sd()", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "Now we are ready to introduce basic data transformation techniques such as slicing and dicing. Slicing, otherwise known as subsetting, allows the selection of data frame subsets. These subsets are defined by boolean conditions built from Attribute op value pairs where op is one of the arithmetic operators such as =, !=, < etc. For example (Score >70)& (Grade ==\u2019A\u2019) refers to a subset of a data frame describing students who scored more than 70 points and got an A.\r\n\r\nDicing refers to eliminating some of the attributes from a data frame - it is vertical slicing - which results in a more \u201cnarrow\u201d frame. Finally we can also expand our data frame with new, so called derived, attributes. This is a very useful operation in data analysis since it allows so-called \u201cfeature engineering\u201d. These new user-defined features can lead to totally new insights into the data.", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n#Lets look at the standard deviation of score column\nsd(moody$Score)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "3", "snippets": {}}, "3.5": {"Name": "Subset", "Details": "

See the documentation of subset() at https://sparkbyexamples.com/r-programming/r-subset-function-usage/?expand_article=1

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"3.5.1": {"Name": "subset function ", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n#Subset of rows\nmoody_psychology<-subset(moody, Major== 'Psychology')\nnrow(moody)\nnrow(moody_psychology)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.5.2": {"Name": "subsetting columns", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n#Alternate way to subset.\nmoody[moody$Major==\"Psychology\", ]\nmoody[moody$Major!=\"Psychology\", ]\nmoody[moody$Score >80, ]\nmoody[moody$Score >80 & moody$Grade == 'B', ]", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.5.3": {"Name": "subsetting rows and columns", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\ncolnames(moody)\n#subset of columns\nmoody3<-subset(moody, select = -c(1))\nncol(moody3)\n# You can see the number of columns has been reduced by 1, due to sub-setting without column 1\nncol(moody3)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.5.4": {"Name": "Chat about data frame subsetting ", "Details": "\n Lets talk about subsetting\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n#Subset of Rows and Columns\nmoody1<-subset(moody, select = c(2:4), Major==\"Psychology\")\ncolnames(moody1)\n#Notice that only 3 columns are remaining\ndim(moody1)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.5.5": {"Name": "Most important role of subseting - data exploration", "Details": "Subsetting data frames plays a crucial role in data exploration for data scientists. It allows them to identify interesting and meaningful subsets of data that exhibit patterns or behaviors distinct from the overall dataset. Here are the key points regarding subsetting and its importance in data exploration:\n\nUncovering Unusual Patterns: Subsetting data frames helps data scientists identify subsets where patterns or characteristics deviate from the general population. By focusing on specific subsets, they can uncover unique insights that may not be apparent when analyzing the entire dataset. For example, exploring a subset of psychology majors who score below 80 may reveal a surprisingly high frequency of A grades, indicating potential anomalies or factors specific to that group.\n\nIdentifying Disproportionate Outcomes: Subsetting allows data scientists to investigate subsets with disproportionately high or low outcomes compared to the overall population. By isolating specific groups, such as computer science majors who scored below 50, they can determine if there are significantly higher chances of failing the class within that subset. This analysis helps identify factors or variables that may have a significant impact on outcomes within specific subgroups.\n\nComparative Analysis: Subsetting enables data scientists to compare different subsets within the dataset. For instance, comparing the median scores for A between computer science majors and economics majors can provide insights into potential disparities in grading criteria or performance expectations. By examining subsets, they can uncover variations and make meaningful comparisons across different groups or categories.\n\nVisualization and Summary Statistics: Subsets derived from data frames can be visualized through various plots, graphs, or charts to gain a visual understanding of the patterns or differences. Additionally, core functions such as mean(), max(), and min() can be applied to the subsets to calculate summary statistics and quantify the observed patterns or variations.\n\nBy leveraging the power of subsetting, data scientists can delve deeper into the data, discover unique subsets with distinct characteristics, and gain valuable insights that may inform further analysis, modeling, or decision-making processes. Subsetting allows for a targeted exploration that goes beyond the general trends and uncovers hidden patterns or relationships within the dataset.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": {}}}, "parent": "3", "snippets": {}}, "3.6": {"Name": "tapply", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"3.6.1": {"Name": "tapply()", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n#Distribution of grades for seniors who major in Economics\ntapply(moody[moody$Seniority == 'Junior',]$Score, moody[moody$Seniority == 'Junior',]$Grade,mean)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.6.2": {"Name": "Chat about tapply", "Details": "\n Chat about tapply function\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n#Distribution of grades for juniors\ntable(moody[moody$Seniority == 'Junior',]$Grade)\n#Distribution of grades for seniors who major in Economics\ntable(moody[moody$Seniority == 'Senior' & moody$Major == 'Economics', ,]$Grade)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "3", "snippets": {}}, "3.7": {"Name": "String operations", "Details": "This the family of simple functions which help to get inside the string \ngrepl(\"[A-Z]\", input_string)\ngrepl(\"[0-9]\", input_string)\ngrepl(\"[[:punct:]]\", input_string)\n\nTo test if the string contains capital letter(s), numbers or special characters. \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "istring <-'Abqkd123&hhsD'\ncat('This checks for capital letters in the string')\ncontains_capital <- grepl(\"[A-Z]\", istring)\ncontains_capital\ncontains_number <- grepl(\"[0-9]\", istring)\ncat('This checks for special characters in the string')\ncontains_special_char <- grepl(\"[[:punct:]]\", istring)\ncontains_special_char\ncat('This calculates the number of characters in the string')\nnchar(istring)\n\n\nuppercase_only <- gsub(\"[^A-Z]\", \"\", istring)\n\n# Count the number of uppercase letters\nnum_uppercase_letters <- nchar(uppercase_only)\n\n# Print the result\ncat(\"Number of capital letters:\", num_uppercase_letters, \"\\n\")", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.8": {"Name": "Examples of basic queries in R", "Details": "In this subsection, we showcase a series of straightforward queries written in R and executed on various data sets. These queries highlight the practical application of coding, even with the simplest \"one liners,\" allowing us to gain further insights into the data. Such queries serve as an initial step in exploring the nature of the data set, working hand in hand with visual plots. They play a crucial role in conducting preliminary data exploration and serve as an important first step in becoming acquainted with the data using basic programming techniques. we begin with simple checking of column names and values which occur in these columns. ", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\ncolnames(moody)\nsummary(moody)\nunique(moody$Seniority)", "Result": "", "Title": "Getting to know the data set, what columns does the data frame have? Distributions of values", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\nmin(moody[moody$Grade=='A',]$Score)", "Result": "", "Title": "What is the minimum score for A students?", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\nmax(moody[moody$Grade=='B',]$Score)", "Result": "", "Title": "What is the maximum score for B students?", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\ntapply(moody[moody$Grade=='A',]$Score, moody[moody$Grade=='A',]$Major, min)", "Result": "", "Title": "What is lowest minimal score for A grade among all majors?", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\nmoody$ScoreIntervals<-cut(moody$Score,breaks=c(0,60,80,90,100),labels=c(\"Low\",\"Medium\",'Good', \"Excellent\"))\ntable(moody$ScoreIntervals)", "Result": "", "Title": "Using cut() to divide Score into 4 intervals and create new attribute ScoreIntervals", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\nmoody$ScoreIntervals<-cut(moody$Score,breaks=c(0,60,80,90,100),labels=c('Low','Medium','Good', 'Excellent'))\ntable(moody$ScoreIntervals, moody$Grade)", "Result": "", "Title": "Using ScoreIntervals to see distribution of Grades over 4 new Score intervals", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\ntable(moody$Major, moody$Grade)", "Result": "", "Title": "Which major gets the most A's? Notice that table() provides you more information than that - but you can easilly spot the answer to most A's, most F's, least A's, in general most/least of any grades..Such a simple function and so much info!", "Type": "R", "Video": "", "Width": ""}]}, "3.9": {"Name": "Looking for hidden patterns in data through simple queries in R", "Details": "\nPatterns in data refer to the identifiable relationships and trends that exist among different variables within a dataset. These patterns can be categorized into three main types:\n\nNumerical vs. Numerical Relationship (Correlations):\n\nThis type of pattern involves numerical variables and is often referred to as correlations. For example, in a dataset of students, there may be a correlation between a student's score in a class and their GPA. Students with higher GPA may acheive higher score in class. As we will see it is NOT the case in Moody data set. \n\nCategorical vs. Categorical Relationship:\n\nThis pattern involves categorical variables. For instance, you can examine the relationship between a student's major (e.g., psychology vs computer science) and their grades. One may find that psychology majors have a higher likelihood of passing a class with at least a C grade compared to computer science majors.\n\nNumerical vs. Categorical Relationship:\n\nThis pattern explores the interaction between numerical and categorical variables. For example, one can analyze the relationship between a student's score (numerical) and their final grade (categorical). Generally, a higher score is associated with a better grade, but the specific cutoff points for grades may vary based on factors like the student's major or seniority level. The cutoff points in Moody data set clearly do not seem to be pre-determined. We do not know how Professor Moody decides grades in his class.\n\nData without any discernible patterns is essentially random. However, even random can be highly interesting or even shocking. It would be highly surprising if Professor Moody's data exhibited no patterns and, for instance, students could achieve any grade for any score just randomly!\n\nData scientists act as data investigators, similar to private detectives. Their role involves uncovering patterns and correlations within the data, which can provide valuable insights. While many patterns in data are interesting, only a subset of them is actionable. For example, a clear correlation between a student's class score and their final grade is actionable because it suggests that improving class performance can lead to a better grade. On the other hand, differences in the relationship between major and grade may not be actionable, as students typically won't change their major based on these patterns. However, the relationship between seniority and grade is actionable and can help students decide when to take a class, such as as a freshman or senior, depending on their goals and seniority level.\n\nThe following snippet shows differences between students of different majors. \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\ntapply(moody[moody$Major=='Economics',]$Score, moody[moody$Major=='Economics',]$Seniority, mean)", "Result": "", "Title": "What queries does this code allows you to answer?", "Type": "R", "Video": "", "Width": ""}, {"Details": "Now, lets discover what are fraction of A's in CS and Psychology respectively?", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n#Does major affect the grade averages and cutoff points\ncat('Grade Averages in Psychology ')\nmoodyP<-moody[moody$Major=='Psychology',]\ntapply(moodyP$Score, moodyP$Grade, mean)\ncat('Grade Averages in Computer Science ')\nmoodyCS<-moody[moody$Major=='CS',]\ntapply(moodyCS$Score, moodyCS$Grade, mean)\ncat('Grade Averages in Statistics')\nmoodyStat<-moody[moody$Major=='Statistics',]\ntapply(moodyStat$Score, moodyStat$Grade, mean)\ncat('Grade Averages in Economics')\nmoodyEcon<-moody[moody$Major=='Economics',]\ntapply(moodyEcon$Score, moodyEcon$Grade, mean)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "Does higher GPA gives us an edge in data 101 for a better grade? Depends on Major? How?", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\ncat(\"Grade distribution per major\")\ntable(moody$Grade, moody$Major)\ncat(\"Grade distribution per seniority\")\ntable(moody$Grade, moody$Seniority)\ntable(moody$Grade, moody$Major)\na<-table(moody$Grade, moody$Major)\ncat(\"Fraction of A's for CS majors\")\na[1,1]/sum(a[,1])\ncat(\"Fraction of A's for Psychology majors\")\na[1,3]/sum(a[,3])", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\ncat(\"Does higher GPA get you better grade?\")\ntapply(moody$GPA, moody$Grade, mean)\ncat(\"Does higher GPA get you better grade if you are psychology major?\")\nmoodyP<-moody[moody$Major=='Psychology',]\ntapply(moodyP$GPA, moodyP$Grade, mean)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.10": {"Name": "Eyeballer queries ", "Details": "In class you guessed predictive patterns for different data sets. Here is the next step, how would verify that this patterns hold over much larger data sets? You would have to write code. ", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "quiz<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Quizzes.csv')\nHireShort<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/HireShort.csv')\n#Lets look at Quizzes Eyballer mini dataset\ncat('How likely is a quiz when weather is Sunny and today is different day of the week than day of last quiz?')\ntable(quiz[quiz$Day!=quiz$LastQuiz & quiz$Weather =='Sunny',]$Quiz)\ncat('How likely is a quiz when weather is Cloudy and today is different day of the week than day of last quiz?')\ntable(quiz[quiz$Day!=quiz$LastQuiz & quiz$Weather =='Cloudy',]$Quiz)\ncat('How likely is a quiz when weather is Sunny and today is same day of the week than day of last quiz?')\ntable(quiz[quiz$Day==quiz$LastQuiz & quiz$Weather =='Sunny',]$Quiz)\ncat('How likely is a quiz when weather is Cloudy and today is same day of the week than day of last quiz?')\ntable(quiz[quiz$Day==quiz$LastQuiz & quiz$Weather =='Cloudy',]$Quiz)\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.11": {"Name": "Merging data frames", "Details": "We can merge multiple data frames on matching values of two columns. It is important when our data is located in multiple data frames", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "3", "snippets": [{"Details": "", "PreExCode": "", "Query": "movies<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Movies2022F-4-new.csv')\nContinents<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Continents.csv')\nmerg<- merge(movies, Continents, by.x= \"country\", by.y='Country')\nmerg", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "3.12": {"Name": "Questions (Video)", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/13VwWkvsKOZC0mo6Uitqb9oyb0dfv8od3/preview", "child": {}, "parent": "3", "snippets": {}}}, "parent": "3", "snippets": {}}, "4": {"Name": "\ud83c\udfa6 Plots (Video) \ud83c\udfa6", "Details": "

When you import your data to R studio one of the first things you do is plot. Data visualization is a key components of data analysis. Before we talk about plots, we introduce some very basis data structures in R: vectors, data frames and tables. These are introduced below in the form of code snippets that you can run and modify.

Then we are ready to plot!

We will introduce several basic plots such as scatter plot, bar plot, boxplot and mosaic plot. How do we know which plot to apply? It depends on whether the variables to be plotted are categorical or numerical. Below we show a simple table which can serve as a guide which plot to use depending on types of variables to be plotted.

\n\n \n \n \n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n


NUM x NUM	scatter plot
CAT x CAT	mosaic plot
CAT x NUM	box plot
NUM	box plot, histogram
CAT	bargraph

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"4.0": {"Name": "\ud83c\udfa6 Practice (Video) \ud83c\udfa6", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1HWY5-N4hln32itL7ZN6x2WzkHLjJezhT/preview", "child": {}, "parent": "4", "snippets": {}}, "4.1": {"Name": "Scatter Plot", "Details": "

Scatter Plot are used to plot two numerical variables.
Hence it is used when both the labels are numerical values.

Lets look at example of scatter plot using Moody.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "4", "snippets": [{"Details": "", "PreExCode": "", "Query": "# Let's look at a 2 attribute scatter plot.\n# moody<-read.csv(\"../files/dataset/moody2020b.csv\") #static Load\nmoody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2020b.csv\") #web load\nplot(moody$participation,moody$score,ylab=\"score\",xlab=\"participation\",main=\" Participation vs Score\",col=\"red\")", "Result": "", "Title": "Scatter plot", "Type": "R", "Video": "", "Width": ""}]}, "4.2": {"Name": "Bar Plot", "Details": "

A bar plot are used to plot a categorical variable.
This rectangle height is proportional to the value of the variable in the vector.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "4", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\ncolors<- c('red','blue','cyan','yellow','green') # Assigning different colors to bars\n\n#lets make a table for the grades of students and counts of students for each Grade. \n\nt<-table(moody$Grade)\n\n#once we have the table lets create a barplot for it.\n\nbarplot(t,xlab=\"Grade\",ylab=\"Number of Students\",col=colors, \n main=\"Barplot for student grade distribution\",border=\"black\")", "Result": "", "Title": "Bar plot", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "# moody<-read.csv(\"../files/dataset/moody2020b.csv\") #static Load\nmoody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\nmoody<-moody[moody$Score >80,]\ncolors<- c('red','blue','cyan','yellow','green') # Assigning different colors to bars\n\n\n\nt<-table(moody$Grade)\n\n\n\nbarplot(t,xlab=\"Grade\",ylab=\"Number of Students\",col=colors, \n main=\"Barplot for student grade distribution for high scores\",border=\"black\")", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "4.3": {"Name": "Box Plot", "Details": "

A boxplot is used to display a numerical variable.
A boxplot shows the distribution of data in a dataset.

$\"Boxplot\"$

A boxplot shows the following things:
- Minimum
- Maximum
- Median
- First quartile
- Third quartile
- Outliers
\n

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "4", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\ncolors<- c('red','blue','cyan','yellow','green') # Assigning different colors to bars\n#Suppose you want to find the distribution of students score per Grade. We use box plot for getting that. \nboxplot(Score~Grade,data=moody,xlab=\"Grade\",ylab=\"Score\", main=\"Boxplot of grade vs score\",col=colors,border=\"black\")\n\n# the circles represent outliers.", "Result": "", "Title": "Box plot", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\ncolors<- c('red','blue','cyan','yellow','green') # Assigning different colors to bars\nmoody<-moody[moody$Major=='Psychology',]\n#Suppose you want to find the distribution of students score per Grade. We use box plot for getting that. \nboxplot(Score~Grade,data=moody,xlab=\"Grade\",ylab=\"Score\", main=\"Boxplot of grade vs score for Psychology majors\",col=colors,border=\"black\")\n\n\n# the circles represent outliers.", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "4.4": {"Name": "Mosaic Plot", "Details": "

Mosaic plot is used to visualize two categorical variables.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "4", "snippets": [{"Details": "", "PreExCode": "", "Query": "# moody<-read.csv(\"../files/dataset/moody2020b.csv\") #static Load\nmoody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2020b.csv\") #web load\ncolors<- c('red','blue','cyan','yellow','green') # Assigning different colors to bars\n\n#suppose you want to find numbers of students with a particular grade based on their texting habits. Use Mosiac-plot.\n\nmosaicplot(moody$grade~moody$texting,xlab = 'Grade',ylab = 'Texting habit', main = \"Mosiac of grade vs texing habit in class\",col=colors,border=\"black\")", "Result": "", "Title": "Mosaic plot", "Type": "R", "Video": "", "Width": ""}]}, "4.5": {"Name": "Misleading Graphs", "Details": "

Beware of misleading graphs.

The following graphs artificially exaggerate their claims by manipulating either the Y-axis or X-axis. Typically this effect can be achieved by moving the beginning of the scale (Y or X) from zero to much larger value. Such axes manipulations result in exaggerating otherwise minor trends and differences.

$\"\"$ \n

Figure 4.6.1

\n
\n

Differences between calories are not as large as it appears on the graph above. The range is just between 590 and 720, but because of moving the origin of the X-axis to 590, the bottom three bars seem to be multiple times larger than the top 3. KFC and MacDonald look much better than they should.

$\"\"$ \n

Figure 4.6.2

\n
\n

Number of people on welfare appears to be growing rapidly. In fact, it is growing by total of around 12% in 3 years. This is a far cry from 4 times \u2013 when judging from the height of the last bar as compared with the first, leftmost bar.

$\"\"$ \n

Figure 4.6.3

\n
\n

It seems like the fraction of Democrats who agree with the court is much higher than the fraction of Republicans or Independents (seems like 3x of democrats agree with the court as compared to Republicans). By moving the origin of the Y-axis to 50, the difference of 8% points between Democrats and republicans is grossly exaggerated.

Similar effects are achieved in the graphs below \u2013 moving the origins of axes exaggerates the difference in bar sizes.

$\"\"$ \n

Figure 4.6.5

\n
\n

$\"\"$ \n

Figure 4.6.6

\n
\n

The home price increase is not massive! It only appears to be by manipulation of the graph! Neither is average class score for Ms Smith vs Mr. Jones.

Optical Illusions

Depending how pie-chart is presented some slices may appear much larger than they really are. For example item C is the same size as slice A, but it appears much larger.

$\"\"$ \n

Figure 4.6.7

\n
\n

Notice the graph below is reversed. What looks like a drop is in fact increasing!

$\"\"$ \n

Figure 4.6.8

\n
\n

Misleading Message

In these two side-by-side graphs we have Y-axes labeled by different variables (number of countries and cases per 10000 respectively). This is why the comparison is a bit like apples and oranges.

$\"\"$ \n

Figure 4.6.9

\n
\n

Correlation vs Causation

This is one of many examples of graphs showing spurious correlation which looks like causation. The two unrelated quantities change in time very similarly but have nothing to do with one another!

$\"\"$ \n

Figure 4.6.10

\n
\n

One sided arguments

The graph below shows only half of what should be the full argument. What about people who did not go to college at all? We are missing the \u201cbase\u201d here.

$\"\"$ \n

Figure 4.6.11

\n
\n

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "4", "snippets": {}}, "4.6": {"Name": "Scavenger Hunt: looking for secret pattern embedded in Moody data set", "Details": "At the beginning of the semester, Professor Moody established grade cutoffs.\n\n'A' - score at least 90\n B' - score at least 80 and less than 90\n'C' - scores above 60 and less than 80\n'D' - scores above 50 and below 60 and finally\n'F' - scores below 50 \nHowever, it is apparent that Professor Moody did not adhere to these cutoffs. He is indeed moody and a bit mercurial! \n\nAt times, he awarded 'A's to scores below 90 and 'D's to scores above 60, effectively lowering the 'C' grade to a 'D' for some students (they were NOT happy). What led to this situation? Did any of the three behavioral attributes\u2014Texting, Dozing, and Asking\u2014play a role? Let's investigate using straightforward graphs and simple queries.\"\r\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "4", "snippets": [{"Details": "Please observe that there is an overlap in the boxplot between grades 'A' and 'B,' as well as between other adjacent grades. In essence, this means that a student can receive different grades for the same score. Is this unfair? Undoubtedly, the student with the lower grade would argue that it is. Perhaps, on occasion, even a higher score might result in a lower grade. So, what serves as the foundation for Professor Moody's grading system? Clearly, it is not solely based on the numerical score. But what else could it be?\r\n\r\nAdditionally, it's worth noting that we also collect information regarding whether students doze off during Professor Moody's classes, whether they engage in texting, and if they ask questions. Could this supplementary information influence the assignment of the final grade? If so, in what manner?\"", "PreExCode": "", "Query": "moody<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Moody2023cFall.csv')\ncolors<-c('red','blue','black','green')\nhead(moody)\nboxplot(Score~Grade,data=moody,xlab=\"Grade\",ylab=\"Score\", main=\"Boxplot of grade vs score\",col=colors,border=\"black\")-c('red','blue','black','green')\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "Now let us look at overlaps between D and C. There are cases when students score exceeds 60 and they still get a D (should be C). Why?", "PreExCode": "", "Query": "moody<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Moody2023cFall.csv')\nmoodyA<-moody[moody$Grade=='A' & moody$Score <90,]\ncat('Which of the three behaviours - dozing, texting or asking can play a role in Moody grading. Lets figure it out, Is it texting?')\ntable(moodyA$Texting)\ncat(' Dozing?')\ntable(moodyA$Dozing)\ncat(' Asking?')\ntable(moodyA$Asking)\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "No let us see the border cases for A. How can a student with the score between 85 and 90 get a B. It turns out if is when the \"Always\" ask questions. Professor Moody likes good quesitons as a measure of interest in his lecture!", "PreExCode": "", "Query": "moody<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Moody2023cFall.csv')\nmoodyD<-moody[moody$Grade=='D' & moody$Score >60,]\ncat('Which of the three behaviours - dozing, texting or asking can play a role in Moody grading. Lets figure it out, Is it texting?')\ntable(moodyD$Texting)\ncat(' Dozing?')\ntable(moodyD$Dozing)\ncat(' Asking?')\ntable(moodyD$Asking)\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "It seems that asking a lot of questions has the opposite effect on students who are struggling academically. Students who scored between 60 and 65, which should typically earn them a C grade, ended up with only a D. Professor Moody might have been bothered by the abundance of questions that appeared too trivial to the professor. This situation is certainly not a pleasant one!", "PreExCode": "", "Query": "moody<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Moody2023cFall.csv')\ncat('Checking out the low scoring As for what they have in common')\nmoody[moody$Grade =='A' & moody$Score>80 & moody$Score <90,]\ncat('Checking out the low scoring As for what they have in common')\nmoody[moody$Grade =='A' & moody$Score>85 & moody$Score <90,]\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "Solution to HW2a for the Fall Semester 2023 edition of data 101.\n\"Sitting at even row is good for you - you pass even when you get less than score of 50", "PreExCode": "", "Query": "moody<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Moody2023cFall.csv')\ncat('Checking out the high scoring Ds for what they have in common')\nmoody[moody$Grade =='D' & moody$Score>60 & moody$Score <65,]\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "Solution to HW2b for the Fall Semester 2023 edition of data 101.\n\"Sitting at row >5\" is bad for you - you fail even when you get more than 50 score", "PreExCode": "", "Query": "moody<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/moody2023F1.csv')\nboxplot(moody$Score~moody$Grade)\nmoody[moody$Score >40 & moody$Score<50 & moody$Grade=='P',]\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "Now we can test the discovered patterns by building simple prediction models from our data and testing it on the same data (of course later, we will insist on distingishing training and testing data). This will give us an idea how well our models fit the training data (that is the moody sets we used - F1 and F2). The pattern which I embedded into data should lead to the prediction model with higherst accuracy. But one may be able to \"overfit\" my model to the data set and do even better. ", "PreExCode": "", "Query": "moody<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/moody2023F2.csv')\n\nboxplot(moody$Score~moody$Grade)\ncat('Eyeballing - what students who failed the class despite scoring between 50 and 60 have in common')\nmoody[moody$Score >50 & moody$Score<60 & moody$Grade=='F',]\ncat('Eyeballing - what students who passed the class when scoring between 50 and 60 have in common')\nmoody[moody$Score >50 & moody$Score<60 & moody$Grade=='P',]\ncat('How many students failed the class despite scoring between 50 and 60')\nnrow(moody[moody$Score >50 & moody$Score<60 & moody$Grade=='F',])\ncat('How many students passed the class when scoring between 50 and 60')\nnrow(moody[moody$Score >50 & moody$Score<60 & moody$Grade=='P',])\ncat('How many students who sat above row 5 passed the class when scoring between 50 and 60')\nnrow(moody[moody$Score >50 & moody$Score<60 & moody$Row >5 & moody$Grade=='P',])\ncat('How many students who sat above row 5 failed the class when scoring between 50 and 60')\nnrow(moody[moody$Score >50 & moody$Score<60 & moody$Row >5 & moody$Grade=='F',])\ncat('Two important subsets')\n#moody[moody$Score >50 & moody$Score<60 & moody$Row >5 & moody$Grade=='F',]\n#moody[moody$Row>5 & moody$Grade=='P',]", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "moody<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/moody2023F1.csv')\n\n#Let's now build prediction model based on our pattern for moody2023F1 data set\ndecision <- rep('F',nrow(moody))\neven<-c(2,4,6,8)\ndecision[(moody$Row %in% even) & moody$Score>40 & moody$Score<50] <- 'P'\ndecision[moody$Score>50] <- 'P'\ncat('This is acuracy of this model - it is almost 0.99!')\nmean(decision==moody$Grade)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "moody<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/moody2023F2.csv')\n\n#Let's now build prediction model based on our pattern for moody2023F2 data set\ndecision <- rep('P',nrow(moody))\ndecision[moody$Score <50]<-'F'\ndecision[(moody$Row>5) & moody$Score>50 & moody$Score<60] <- 'F'\ncat('This is acuracy of this model -try yours!')\nmean(decision==moody$Grade)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "moody<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/moody2023F2.csv')\n\n#Let's now build prediction model based on our pattern for moody2023F2 data set\ndecision <- rep('P',nrow(moody))\ndecision[moody$Score <50]<-'F'\ndecision[moody$Row>5 & (moody$Section =='A1'| moody$Section =='A3' | moody$Section =='B1' | moody$Section =='B3')& moody$Score>50 & moody$Score<60] <- 'F'\ncat('This is acuracy of more preciese model with side sections and high seating is penalized-try yours!')\nmean(decision==moody$Grade)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "4.7": {"Name": "Additional References", "Details": "

https://www.datamentor.io/r-programming/plot-function/\r\nhttps://www.data-to-viz.com/graph/sankey.html\r\n

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "4", "snippets": {}}, "4.8": {"Name": "Chat about plots with GPT", "Details": "\n Lets talk about plots and coloring\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "4", "snippets": {}}}, "parent": "4", "snippets": {}}, "5": {"Name": "Feature engineering", "Details": "In R, it is possible to create new attributes (columns) for a data frame \"on the fly.\" These new attributes are essentially new vectors that are frequently defined as functions of existing attributes. As a result, they are referred to as derived attributes.\r\n\r\nDerived attributes are significant in both data exploration and the development of prediction models. They often enable the identification of important patterns within the data. Additionally, derived attributes may possess greater predictive power than the original attributes in the imported data sets.\r\n\r\nIn the field of machine learning, the term \"feature engineering\" is frequently employed to describe the process of creating derived attributes.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1czO-UJafMmMBfBuxZ1tNFXITKZ9Coahj/preview", "child": {"5.1": {"Name": "Making new categorical attributes", "Details": "Here we define a new attribute PF (Pass/Fail) to \u201cPass\u201d. Students who got A, B or C, passed. Students who received F, failed. We are grouping values of Grade into two categories of a new categorical attribute PF.\r\n\r\nThe line 5 replaces \u201cPass\u201d by \u201cFail\u201d for students who received F.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "5", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022_new.csv\") #web load\n\nmoody$PF<-'Pass'\nmoody[moody$GRADE=='F',]$PF<-'Fail'\nmoody", "Result": "", "Title": "Categorical grouping - to create new categorical attribute", "Type": "R", "Video": "", "Width": ""}]}, "5.2": {"Name": "Making categorical attribute from numerical attribute using function Cut()", "Details": "cut() function divides the range of x into intervals. Provides ability to label intervals as well. It plays important role in defining derived attributes from attributes which are numerical.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "5", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022_new.csv\") #web load\n\n# Cut Example using breaks - Cutting data using defined vector. \nscore1 <- cut(moody$SCORE,breaks=c(0,50,100),labels=c(\"F\",\"P\"))\ntable(score1)", "Result": "", "Title": "Using Cut: from Numerical to Categorical attribute", "Type": "R", "Video": "", "Width": ""}]}, "5.3": {"Name": "Making new numerical attribute from numerical attributes", "Details": "Suppose we would like to combine score and participation into one combined score. We can define a new numerical attribute from SCORE and PARTICIPATION . We can see that the moody data frame will be expanded by the additional attribute.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "5", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022_new.csv\") #web load\n\nmoody$Combined <- (moody$SCORE + 10* moody$PARTICIPATION)\nmoody[1:10,]", "Result": "", "Title": "Combining attributes to make a new attribute", "Type": "R", "Video": "", "Width": ""}]}, "5.4": {"Name": "More complex example of defining derived attributes", "Details": "The way we define combined score attributes rewards students even for poor participation. Their combined score is always higher than their score in class even if their participation was quite low. It would make more sense to define combined score by either penalizing for poor performance or rewarding good performance.\r\n\r\nThe next snippet illustrates defining such a new numerical attribute, $adjustedScore of a student in the Moody data frame. adjustedScore penalizes low participation or rewards for good participation.\r\n\r\nScore is adjusted by the value of participation attribute in the following way:\r\n\r\nIf participation is larger than 0.5 - a bonus proportional to participation * 10 is added to the score.\r\nIf participation is smaller than 0.5, a penalty of 1-participation) * 10 is subtracted from the score.\r\nIn this way, for someone with very small participation, the 10 point penalty will be imposed (10 points subtracted from the score). Conversely, someone with perfect participation (1.0) will receive a 10 point bonus.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "5", "snippets": [{"Details": "", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022_new.csv\") #web load\n\nmoody$AdjustedScore <-0\n\nmoody[moody$PARTICIPATION<0.50, ]$AdjustedScore <- moody[moody$ PARTICIPATION <0.50, ]$SCORE -10*(1-moody[moody$ PARTICIPATION <0.50, ]$ PARTICIPATION)\n\nmoody[moody$ PARTICIPATION >=0.50, ]$AdjustedScore <- moody[moody$ PARTICIPATION >=0.50, ]$SCORE +10*moody[moody$ PARTICIPATION >=0.50, ]$ PARTICIPATION\n\n#Lets see a sample of new rows of moody with the new derived attribute. We are skipping a few attributes for readability - showing only Score, Grade, Particpation and Adjusted Score\n\nmoody[1:10,c(1,2,5,6)]", "Result": "", "Title": "More complex example - using conditionals. ", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "# CODING IN CLASS\nmoody <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n# Problem 1 - Low_Score\n\nmeanA<-mean(moody[moody$Grade=='A',]$Score)\nminA<- min(moody[moody$Grade=='A',]$Score)\nmeanB<-mean(moody[moody$Grade=='B',]$Score)\nminB<- min(moody[moody$Grade=='B',]$Score)\nmeanC<-mean(moody[moody$Grade=='C',]$Score)\nminC<- min(moody[moody$Grade=='C',]$Score)\nminD<- min(moody[moody$Grade=='D',]$Score)\nmoody$Low_Score<-'False'\nmoody[moody$Grade=='A' & moody$Score > minA & moody$Score < meanA,]$Low_Score<-'True'\nmoody[moody$Grade=='B' & moody$Score > minB & moody$Score < meanB,]$Low_Score<-'True'\nmoody[moody$Grade=='C' & moody$Score > minC & moody$Score < meanC,]$Low_Score<-'True'\nhead(moody)\n\n\n# Problem2 - Bump_Up\n\nminA\nmoody$BUMP_UP<-'False'\nmoody[moody$Grade=='B' & moody$Score > minA,]$BUMP_UP<-'True'\nmoody[moody$Grade=='C' & moody$Score > minB,]$BUMP_UP<-'True'\nmoody[moody$Grade=='D' & moody$Score > minC,]$BUMP_UP<-'True'\nmoody[moody$Grade=='F' & moody$Score > minD,]$BUMP_UP<-'True'\nhead(moody)\n\n# Problem3 - Different cutpoints for CS\n\nmoody$newGrade<-moody$Grade\nmoody[moody$Score > 60 & moody$Major=='CS',]$newGrade<-'A'\nmoody[moody$Score > 80 & moody$Major=='CS',]$newGrade<-'B'\nmoody[moody$Score > 95 & moody$Major=='CS',]$newGrade<-'C'\nhead(moody)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "5.5": {"Name": "Feature engineering", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "5", "snippets": {}}}, "parent": "5", "snippets": {}}, "6": {"Name": "Data Science vs Databases", "Details": "

At the beginning there is always data. In R, it is a data frame which is a set of rows and columns. Our Moody data set is an example of a data set. It contains thousands of rows and a handful of columns.
But is it a complete data set or is it just \u201ctip of an iceberg\u201d? Is our dataset \u201call there is\u201d? Or is it just a sample from a possibly much larger, unknown, universe?

Data science and statistics deal with situations when data is partial. In other words we do not have the whole data set. We only have a sample, even if this sample is very large. No matter how large the data is, it is still just a sample from a larger hypothetical data set that may involve future, unknown yet, data values.

On the other hand ,in the field of databases we believe that our data set is complete. Data is \u201cas presented\u201d. We query such data and treat the answers as exact values, since we fundamentally trust that the data we have is correct and complete. This is quite different from the data science approach which always treats data as an incomplete sample of the unknown, real, data set. Thus, data science deals with approximations.

We formulate the main principle behind data science

Data is always a sample

Example: Airbnb data set.

Let airbnb be the data set describing Airbnb rentals in New York City. It has 5287 tuples and is a set of listings with room type, neighborhood, neighborhood group (more specific names of hundreds of NYC neighborhoods such as Tribeca, East Village, Park Slope etc), floor and price.

\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n

Table 5.1: Snippet of small airbnb Dataset
id	name	host_name	neighbourhood_group	neighbourhood	room_type	floor	price
5756241	1 bedroom in a 2 bedroom apartment	Teri	Manhattan	Midtown	Entire home/apt	16	531.3010
5692807	Charming Two Bedroom in Chelsea 30 days+	Eric	Manhattan	Chelsea	Entire home/apt	1	287.3102
6286253	SunLit Room in Hip Neighborhood	Rebekah	Brooklyn	Bushwick	Private room	1	267.7120
594732	Greenwich Village Skylit 1BR +deck!	Chris	Manhattan	Greenwich Village	Entire home/apt	16	537.4393
35199464	Penthouse on the beach 3 floors of art, fun,surf,	Sata	Queens	Far Rockaway	Entire home/apt	5	152.7281

There are two approaches of treating this data set:

\n
Approach 1: airbnb data set is complete. We just treat it as \u201cas is\u201d. We can run queries against Airbnb data sets and return definite query answers. For example we can ask about a sample of Airbnb rental offerings which are located in Brooklyn (say, it is 38.1%). Or the average price of Brooklyn rental? Say, it is $176.67. Answers to our queries are treated as exact.
\n
\n
Approach 2: airbnb data set is incomplete. It is just a sample of much larger, unknown set of all prices of airbnb rentals ever (even though we do not have timestamps associated with rentals stored in the Airbnb table) If this is the case, answers to queries are no longer definite and deterministic but rather estimators of real values which are unknown to us.
\n

Let us examine these same queries: query Q1 about frequency of Brooklyn Airbnb rentals and query Q2 about mean price of Brooklyn Airbnb rentals.

\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"6.1": {"Name": "Query 1: Fraction of Brooklyn Airbnb Rentals", "Details": "We begin by selecting a sample of 1000 tuples out of the total of 35,000 tuples from the entire Airbnb dataset, which we'll refer to as 'data.' For the sake of illustration, let's envision this 'data' as the entirety of our Airbnb database. It's important to recognize that the proportion of Brooklyn apartments within this sample may often deviate significantly from the proportion of Brooklyn apartments in the entire Airbnb dataset.\n\nIn cases where we lack knowledge about the complete Airbnb dataset, and our only reference point is this 'data' sample, it becomes evident that we can only regard the observed fraction of Brooklyn apartments as an approximation, not an exact value. This concept lies at the core of the data science approach, where each dataset is inherently a 'sample,' contrasting with the database approach that treats each dataset as comprehensive", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "6", "snippets": [{"Details": "", "PreExCode": "", "Query": "\nKonradRandom <- function(df, N, M, anchor, value, auctionName, NumberTopConditions = 10, Y, W) {\n # Calculate base ratio\n total_rows <- nrow(df)\n auctionHouse_rows <- nrow(subset(df, auctionHouse == auctionName))\n base_ratio <- auctionHouse_rows / total_rows\n base_ratio <- round(base_ratio, 2) # Round to 2 digits\n \n # Print the base ratio\n cat(\"Calculated Base Ratio:\", base_ratio, \"\\n\")\n \n minrate <- base_ratio\n conditions_with_ratios <- list()\n \n # First, include the condition 'anchor == value' with its ratio\n initial_condition <- paste0(anchor, \" == '\", value, \"'\")\n initial_condition_subset <- subset(df, eval(parse(text = initial_condition)))\n initial_condition_count <- nrow(initial_condition_subset)\n \n initial_and_auction_subset <- subset(initial_condition_subset, auctionHouse == auctionName)\n initial_and_auction_count <- nrow(initial_and_auction_subset)\n \n initial_ratio <- if (initial_condition_count > 0) {\n initial_and_auction_count / initial_condition_count\n } else {\n 0\n }\n initial_ratio <- round(initial_ratio, 2) # Round to 2 digits\n \n # Include this condition in the results\n conditions_with_ratios <- append(conditions_with_ratios, list(list(condition = initial_condition, ratio = initial_ratio, count = initial_condition_count)))\n \n # Adjust minrate if necessary\n if (initial_ratio > minrate) {\n minrate <- initial_ratio\n }\n \n # Then proceed with the usual logic\n for (i in 1:N) {\n X <- sample(Y, size = sample(1:3, 1)) # Sample from Y\n \n # Ensure X only contains columns present in df\n X <- X[X %in% colnames(df)]\n \n # Ensure auctionHouse and anchor are not in X\n X <- setdiff(X, c(\"auctionHouse\", anchor))\n \n if (length(X) > 0) {\n df_subset <- subset(df, get(anchor) == value)\n \n # Ensure df_subset only contains valid columns before applying Match\n df_subset <- df_subset[, X, drop = FALSE]\n \n # Select one random tuple from df_subset for Match\n if (nrow(df_subset) > 0) {\n random_tuple <- df_subset[sample(nrow(df_subset), 1), , drop = FALSE]\n conditions <- Match(random_tuple, X)\n \n if (length(conditions) > 0) {\n condition <- conditions[[1]]\n \n # Sort the condition attributes to eliminate differences due to order\n sorted_condition <- paste(sort(unlist(strsplit(condition, \" & \"))), collapse = \" & \")\n \n # Check if this sorted condition already exists\n if (!any(sapply(conditions_with_ratios, function(x) x$condition == sorted_condition))) {\n # Evaluate the number of rows that satisfy the condition\n condition_subset <- subset(df, eval(parse(text = condition)))\n condition_count <- nrow(condition_subset)\n \n # Evaluate the number of rows that satisfy the condition and have auctionHouse == auctionName\n condition_and_auction_subset <- subset(condition_subset, auctionHouse == auctionName)\n condition_and_auction_count <- nrow(condition_and_auction_subset)\n \n # Calculate the ratio\n ratio <- if (condition_count > 0) {\n condition_and_auction_count / condition_count\n } else {\n 0\n }\n ratio <- round(ratio, 2) # Round to 2 digits\n \n # Filter based on ratio, minrate, and the minimum number of tuples (W)\n if (ratio > M * minrate && condition_count >= W) {\n conditions_with_ratios <- append(conditions_with_ratios, list(list(condition = sorted_condition, ratio = ratio, count = condition_count)))\n if (ratio > minrate) {\n minrate <- ratio\n }\n }\n }\n }\n }\n }\n }\n \n # Print the number of valid conditions\n cat(\"Number of valid conditions:\", length(conditions_with_ratios), \"\\n\")\n \n # Print the base ratio\n cat(\"Base Rate:\", base_ratio, \"\\n\")\n \n # Print the ratio for the initial condition\n cat(\"Ratio for\", initial_condition, \":\", initial_ratio, \"\\n\")\n \n # Determine the top conditions based on the specified number\n if (length(conditions_with_ratios) > 0) {\n sorted_conditions <- conditions_with_ratios[order(sapply(conditions_with_ratios, function(x) -x$ratio))]\n top_conditions <- head(sorted_conditions, NumberTopConditions)\n \n # Print the top conditions\n cat(\"\\nTOP\", NumberTopConditions, \"CONDITIONS for\", auctionName, \":\\n\")\n for (i in seq_along(top_conditions)) {\n cat(paste0(\"Condition \", i, \": \", top_conditions[[i]]$condition, \"\\n\"))\n cat(paste0(\"Ratio: \", top_conditions[[i]]$ratio, \"\\n\"))\n cat(paste0(\"Number of Tuples: \", top_conditions[[i]]$count, \"\\n\"))\n }\n }\n}\n", "Result": "", "Title": "This is the last one", "Type": "R", "Video": "", "Width": ""}, {"Details": "

The answer to our query - the real fraction of Brooklyn rentals differs now from 42.4%. It is close, but not the same. This means that the portion of Brooklyn apartments we derived from the Airbnb data sample is merely an approximation of the actual proportion of Brooklyn apartments. The accuracy of this approximation hinges on the size of our sample dataset, which, in our case, consisted of 1000 data tuples. Generally, the smaller the sample size, the less precise our estimate becomes. It's important to note that data is essentially a subset or sample of a larger population. This approach is fundamental in the field of data science.

The following three histograms show distribution of frequency of Brooklyn apartments across 1000 samples of different sizes. We show three sample sizes here: samples of 1000 apartments each, 100 apartments each and finally 5000 apartments each taken from the airbnbReal data set. Notice how the spread of the frequency values depend on the sample size. The larger the data sample is, the \u201ctighter\u201d is the histogram. In the case of a sample size of 1000, we may obtain frequencies of Brooklyn offerings ranging from 38 to 46%.

$\"\"$ \n

Figure 6.1: Distribution of the fraction of Brooklyn apartments - 1000 samples each of the size = 100

The Histogram above shows distribution of frequencies of Brooklyn Airbnb rentals if our database has only 100 offerings. Notice that now if we run 1000 such samples of size 100, the spread is much larger (it is determined by standard deviation) . Now we can get frequencies ranging from 30% to 55%!

$\"\"$ \n

Figure 6.2: Distribution of the fraction of Brooklyn apartments - 1000 samples each of the size = 1000

Finally, the last of the three histograms shows spread of frequencies over 1000 samples each of size 5000. This time the spread is much smaller \u2013 and we range from 41 to 44% with some outliers of course. All these histograms illustrate a simple fact that if we believe our data set to be incomplete and just a sample of a larger data set, we can only answer queries with estimates.

$\"\"$ \n

Figure 6.3: Distribution of the fraction of Brooklyn apartments - 1000 samples each of the size = 5000

", "PreExCode": "", "Query": "# testing", "Result": "", "Title": "Fraction of Brooklyn Airbnb rentals", "Type": "R", "Video": "", "Width": ""}]}, "6.2": {"Name": "Query 2: Mean price of Brooklyn Airbnb rentals", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "6", "snippets": [{"Details": "

In a similar manner to Query 1, we display histograms depicting the average price of Airbnb offerings in Brooklyn using 1000 samples of sizes 1000, 100, and 5000 respectively from the airbnbReal dataset, which consists of 35,573 tuples. Once again, we observe that as the sample size increases, the histograms become more tightly distributed. For instance, in samples of size 100, the average price can range from the 140s to 220s.\n\n These histograms showcasing the average price exemplify a fundamental concept in statistics known as the Central Limit Theorem. This remarkable theorem states that irrespective of the original data distribution, the distribution of sample means follows a normal distribution, commonly known as the Bell curve, as demonstrated in the histograms below.\n\n Furthermore, if we consider our Airbnb dataset solely as a sample, we must provide measures of the accuracy of the estimates it provides for any given query. Only when we accept the data 'as is' can we confidently discuss definitive answers to queries. Otherwise, regardless of the dataset's size, it remains merely a sample. This succinctly summarizes the essence of data science methodology\u2014nothing is exact, and everything is merely a sample. Consequently, it is crucial to go beyond distrusting the data and instead provide measures that assess the quality of estimates derived from data as samples.

$\"\"$ \n

Figure 6.4: Distribution of mean Brooklyn airbnb rental price for 1000 samples each of size = 100

$\"\"$ \n

Figure 6.5: Distribution of mean Brooklyn airbnb rental price for 1000 samples each of size = 1000

$\"\"$ \n

Figure 6.6: Distribution of mean Brooklyn airbnb rental price for 1000 samples each of size = 5000

\n", "PreExCode": "", "Query": "airbnb<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/airbnb.csv') \ncolnames(airbnb)\nset.seed(13245) \ndata<-airbnb[sample(nrow(airbnb), 1000), ]\ncat('Mean price of Brooklyn Airbnb Rentals in sample of Airibnb')\nround(mean(data[data$neighbourhood_group=='Brooklyn',]$price),2)\ncat('Mean price of Brooklyn Airbnb Rentals in entire Airibnb')\nround(mean(airbnb[airbnb$neighbourhood_group=='Brooklyn',]$price),2)", "Result": "", "Title": "Mean price of Brooklyn airbnb rentals ", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "6", "snippets": {}}, "7": {"Name": " Fooled By Randomness ", "Details": "

In this section we show how random data can fool us. Often, random findings may look real, in fact so real that we mistake them for real discoveries. It is one of the main objectives of statistics and data science to help identify such false discoveries. This chapter provides a motivation and justification for hypothesis testing methods, multiple hypothesis corrections, p-values and permutation tests which we discuss in subsequent sections of the active textbook.

Here we show how randomly generated data set can lead to seemingly astonishing discoveries \u2013 which are simply \u2026random, although they look anything but random!

In order to illustrate false discoveries we have used our DataMaker data generation tool to generate a data set Cars describing hypothetical sales data for car dealerships in 15 major cities in the US. We have considered 10 major car makers, different seasons of the year (winter, spring, summer, fall). In addition Cars stores information about buyer gender as well as buyer\u2019s age. Below we show a sample of several tuples of Cars

\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n

Table 7.1: Snippet of Cars Dataset
	Dealership	Season	Car	Buyer	Buyer_Age
11400	LA	Summer	Hyundai	Male	72
47718	San Diego	Spring	Subaru	Female	63
20297	LA	Fall	Ram	Female	74
23292	Houston	Fall	Chevrolet	Female	57
22900	Philladephia	Summer	GMC	Male	42

We have generated the Cars data set as completely random data set, with uniform distributions of dealerships, car makes sold there, four seasons, buyer\u2019s gender\u2019s and their age. Thus any buyer with any age is equally likely to buy any car in any of the dealerships.

We have run several queries on Cars dataset and demonstrated through the following snippets, how random data can lead to query findings which look \u201creal\u201d.

Let us start with the first snippet (snippet 7.1) which shows that indeed our data set follows uniform distribution and it has been randomly generated.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"7.0": {"Name": "\ud83c\udfa6 Practice (Video) \ud83c\udfa6", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1_Sw9XLdekHj2SeEOO5wo9a5GqnPXv6AU/preview", "child": {}, "parent": "7", "snippets": {}}, "7.1": {"Name": "False Discoveries", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"7.1.1": {"Name": "Random or Not? Monte Carlo test", "Details": "Here we show to test if a given query brings random results or is likely to be real? We do it by scrambling the source data frame N times randomly and seeing distribution of query results over random scrambles (permutations). Then we see where the observed result can be placed. What fraction of all random results is even more extreme than the observed result. Typically 5% is considered to be the \"red line\" for randomness. If more than 5% then RANDOM. If less than 5% than likely NOT RANDOM ", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "7", "snippets": [{"Details": "", "PreExCode": "", "Query": "df0 <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Cars2022.csv\")\n\nscramble <- function(df) {\n df_scrambled <- as.data.frame(lapply(df, sample))\n return(df_scrambled)\n}\n\nmultiscramble_query <- function(df0, n, Q) {\n # Compute result0 using the initial data frame\n result0 <- eval(parse(text = Q))\n \n # Initialize storage for results\n results <- numeric(n)\n \n # Perform n scrambles\n for (i in 1:n) {\n df_scrambled <- scramble(df0) # Scramble the dataset differently each time\n results[i] <- eval(parse(text = Q), envir = list(df0 = df_scrambled)) # Evaluate query in the scrambled df0 context\n }\n \n # Compute the fraction where result > result0\n fraction <- sum(results > result0) / n\n \n # Return result0, fraction, and all results\n return(list(result0 = result0, fraction_larger = fraction, all_results = results))\n}\n\n# Define the query string (must be in quotes)\nQ <- \"{\n Narrow <- df0[df0$Car == 'GMC' & df0$Dealership == 'LA' & df0$Buyer_Age > 25 & df0$Buyer_Age < 30, ]\n Q_table <- round(table(Narrow$Buyer) / nrow(Narrow), 4)\n Q_table['Female'] - Q_table['Male']\n}\"\n\n\n# Run the function with 1000 scrambles\nresult <- multiscramble_query(df0, 100, Q)\n\n# Print results\nprint(result)\n", "Result": "", "Title": "\n", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "set.seed(123) # For reproducibility\n\n# Define dataset size\nn <- 10000 # Total number of rows\n\n# Generate random car brands\ncar_brands <- c(\"GMC\", \"Ford\", \"Toyota\", \"Honda\", \"BMW\")\n\n# Generate random dealerships\ndealerships <- c(\"LA\", \"NY\", \"SF\", \"Chicago\", \"Houston\")\n\n# Generate random ages\nages <- sample(18:65, n, replace = TRUE)\n\n# Generate random buyers (Female or Male)\nbuyers <- sample(c(\"Female\", \"Male\"), n, replace = TRUE, prob = c(0.5, 0.5))\n\n# Create the data frame\ndf0 <- data.frame(\n Car = sample(car_brands, n, replace = TRUE),\n Dealership = sample(dealerships, n, replace = TRUE),\n Buyer_Age = ages,\n Buyer = buyers\n)\n\n# Ensure 80% of buyers aged 25-30 buying GMC in LA are Female\nmask <- df0$Car == \"GMC\" & df0$Dealership == \"LA\" & df0$Buyer_Age > 25 & df0$Buyer_Age < 30\nnum_target <- sum(mask)\nnum_females_needed <- round(0.8 * num_target)\n\n# Adjust to meet 80% Female condition\nif (num_target > 0) {\n target_indices <- which(mask)\n female_indices <- sample(target_indices, num_females_needed)\n df0$Buyer[female_indices] <- \"Female\"\n male_indices <- setdiff(target_indices, female_indices)\n df0$Buyer[male_indices] <- \"Male\"\n}\n\n# Display final dataset\nprint(table(df0$Buyer[df0$Car == \"GMC\" & df0$Dealership == \"LA\" & df0$Buyer_Age > 25 & df0$Buyer_Age < 30]) / num_target)\n\n# View sample\nhead(df0)\nnrow(df0)\n\n\nscramble <- function(df) {\n df_scrambled <- as.data.frame(lapply(df, sample))\n return(df_scrambled)\n}\n\nmultiscramble_query <- function(df0, n, Q) {\n # Compute result0 using the initial data frame\n result0 <- eval(parse(text = Q))\n \n # Initialize storage for results\n results <- numeric(n)\n \n # Perform n scrambles\n for (i in 1:n) {\n df_scrambled <- scramble(df0) # Scramble the dataset differently each time\n results[i] <- eval(parse(text = Q), envir = list(df0 = df_scrambled)) # Evaluate query in the scrambled df0 context\n }\n \n # Compute the fraction where result > result0\n fraction <- sum(results > result0) / n\n \n # Return result0, fraction, and all results\n return(list(result0 = result0, fraction_larger = fraction, all_results = results))\n}\n\n# Define the query string (must be in quotes)\nQ <- \"{\n Narrow <- df0[df0$Car == 'GMC' & df0$Dealership == 'LA' & df0$Buyer_Age > 25 & df0$Buyer_Age < 30, ]\n Q_table <- round(table(Narrow$Buyer) / nrow(Narrow), 4)\n Q_table['Female'] - Q_table['Male']\n}\"\n\n# Run the function with 1000 scrambles\nresult <- multiscramble_query(df0, 100, Q)\n\n# Print results\nprint(result)\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "7", "snippets": [{"Details": "Snippet 7.2: Among senior customers (Age > 70) of LA dealership and when the season is Winter, 13.1% buy Toyota and only 6.25% buy GMC. Interesting findings! Twice as many seniors in LA buy Toyota than GMC!", "PreExCode": "", "Query": "Cars <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Cars2022.csv\")\nNarrow<-Cars[Cars$Dealership=='LA'& Cars$Buyer_Age>70&Cars$Season=='Winter',]\nresult<-round(table(Narrow$Car)/nrow(Narrow),4)\nv<-as.data.frame(result)\ncolnames(v)[1]<-'Car'\nv", "Result": "", "Title": ".", "Type": "R", "Video": "", "Width": ""}, {"Details": "We discover seemingly a significant disparity between genders among Honda Buyers in Austin. 8% difference between female and male customer share seems significant.\n\nSnippet 7.5: Mean age of Jeep buyers in Chicago in spring is 4 years higher than in summer. It is 54 in spring vs 50 in summer.", "PreExCode": "", "Query": "# Read in the data\nCars <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Cars2022.csv\")\nNarrow<-Cars[Cars$Car=='Honda' &Cars$Dealership=='Austin',]\nround(table(Narrow$Buyer)/nrow(Narrow),4)", "Result": "", "Title": "Snippet 7.3: Gender distribution of Honda purchases in Austin dealership favors female customers \u2013 54% Female to 46% Male", "Type": "R", "Video": "", "Width": ""}, {"Details": "Younger Jeep Buyers in Chicago wait for summer!\n\nSnippet 7.6: The gender gap among GMC buyers in LA who are in late twenties is 22%!, 61% are female and only 39% are Male", "PreExCode": "", "Query": "# Read in the data\nCars <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Cars2022.csv\")\nNarrow<-Cars[Cars$Car=='Jeep'&Cars$Dealership=='Chicago',]\ntapply(Narrow$Buyer_Age, Narrow$Season, mean)", "Result": "", "Title": "Snippet 7.4 Mean age of Jeep buyers in Chicago in spring is 4 years higher than in summer. It is 54 in spring vs 50 in summer.", "Type": "R", "Video": "", "Width": ""}, {"Details": "GMCs are drastically more popular among women than among men in LA. Certainly all these discoveries are false! Tricks that random data plays on us.\n\nThere is a number of factors leading to these false discoveries:\n\nSubsets of data are pretty narrow \u2013 GMC buyers in winter in one dealership who are in a certain age bracket constitute a small subpopulation of the whole data set. So called Law of small numbers which we discuss later states that such small data sets are prone to extreme results.\n\nThere is an exponential number of queries which look like the queries in snippets 7.1-7.5 and differ only by choice of parameters (age, dealership name, and car make name). Running a sufficient number of such queries every time changing one or more parameters, we are simply bound to find some false positives. This is called multiple hypothesis testing. Before we presented queries 7.1-7.5 above we have run a number of queries which did not generate false discoveries. This is sometimes called p-value hunt and multiple hypothesis testing and Bonferroni coefficient (discussed in section 9) takes this trial and error process under consideration.\n\nDifference of means may be due to a simple random process. We need a permutation test, z-test and similar test to see if a conclusion like in snippet 7.3 is statistically legitimate. It is not. But we cannot just eyeball it and decide.", "PreExCode": "", "Query": "# Read in the data\nCars <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Cars2022.csv\")\nNarrow<-Cars[Cars$Car=='GMC' &Cars$Dealership=='LA' & Cars$Buyer_Age >25 &Cars$Buyer_Age <30 ,]\nround(table(Narrow$Buyer)/nrow(Narrow),4)\n", "Result": "", "Title": "Snippet 7.5: The gender gap among GMC buyers in LA who are in late twenties is 22%!, 61% are female and only 39% are Male", "Type": "R", "Video": "", "Width": ""}]}, "7.2": {"Name": "More about sampling", "Details": "We have already shown the distribution of prices of Brooklyn apartments across samples of different sizes. Here we show more examples of sampling and its impact on different queries.\n\nWe will begin by showing how we select a sample of a given size from the data set. Here for Cars, we select 5000 random tuples from the total of 50,000 tuples. \n\nSnippet 7.7: Sampling", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "7", "snippets": [{"Details": "The function sample() permutes the rows of Cars data frame. Then we select the first 5000 tuples from the shuffled Cars data frame (you may change this number and observe that smaller the sample, the more discrepancy between the sample and the entire data set)\n\nSnippet 7.8: Effect of random samples of different sizes on the average age of a buyer", "PreExCode": "", "Query": "# Read in the data\nCars <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Cars2022.csv\")\n \nv<-sample(1:nrow(Cars))[1:5000]\nSampleCars<-Cars[v, ]\nSampleCars[1:10,] ", "Result": "", "Title": "Snippet 7.6: Sampling", "Type": "R", "Video": "", "Width": ""}, {"Details": "Snippet 7.9: Effect of random samples of different sizes on the average age of a buyer of Ford in Chicago", "PreExCode": "", "Query": "# Read in the data\nCars <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Cars2022.csv\")\nv<-sample(1:nrow(Cars))[1:5000]\nSampleCars<-Cars[v, ]\nmean(SampleCars$Buyer_Age)\nmean(Cars$Buyer_Age)", "Result": "", "Title": "Snippet 7.7: Effect of random samples of different sizes on the average age of a buyer.", "Type": "R", "Video": "", "Width": ""}, {"Details": "The more narrow is the query, the more discrepancy there is between query results over samples.\n", "PreExCode": "", "Query": "# Read in the data\nCars <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Cars2022.csv\")\nv<-sample(1:nrow(Cars))[1:5000]\nSampleCars<-Cars[v, ]\nm<-mean(SampleCars[SampleCars$Dealership=='Chicago' & SampleCars$Car=='Ford',]$Buyer_Age)\ncat(\"The mean age of Ford buyer at Chicago dealership based on the sample size of 5000 is:\",m)\nn<-mean(Cars[Cars$Dealership=='Chicago'&Cars$Car=='Ford',]$Buyer_Age)\ncat(\"The mean age of Ford buyer at Chicago dealership based on entire data set is:\",n)", "Result": "", "Title": "Snippet 7.8: Effect of random samples of different sizes on the average age of a buyer of Ford in Chicago", "Type": "R", "Video": "", "Width": ""}, {"Details": "We can observe quite substantial variation of GMC market share among buyers younger than 30. The market share varies from 7 to 10% among different samples. Just run the code several times to observe variation. Narrow queries (with small number of results) run on small samples lead to extreme results (law of small numbers) If we only considered GMC market share (dropping age constraints), we would observe much smaller market share variation.\n", "PreExCode": "", "Query": "# Read in the data\nCars <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Cars2022.csv\")\n\nv<-sample(1:nrow(Cars))[1:5000]\nSampleCars<-Cars[v, ]\nSampleCars<-SampleCars[SampleCars$Buyer_Age < 30, ]\n \np<-nrow(SampleCars[SampleCars$Car=='GMC',])/nrow(SampleCars)\ncat(\"Fraction of GMC trucks in the random sample of 5000 sold vehicles, for buyers younger than 30, is:\", p)", "Result": "", "Title": "Snippet 7.10: Analysis of market for GMC buyers younger than 30", "Type": "R", "Video": "", "Width": ""}, {"Details": "Notice how for a smaller sample of 500 the distribution of average age varies for different car makes \u2013again run the code several times to get varying results and varying skewdness depending on the sample.\n\n", "PreExCode": "", "Query": "# Read in the data\nCars <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Cars2022.csv\")\n\nv<-sample(1:nrow(Cars))[1:500]\nSampleCars<-Cars[v, ]\nresult<-tapply(SampleCars[SampleCars$Dealership=='Chicago',]$Buyer_Age, SampleCars[SampleCars$Dealership=='Chicago',]$Car, mean)\nas.data.frame(result)\n", "Result": "", "Title": "Snippet 7..11 Average age distribution grouped by all car makes in in Chicago", "Type": "R", "Video": "", "Width": ""}]}, "7.3": {"Name": "Confidence Intervals", "Details": "

\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n

Table 6.2: Snippet of Election Dataset
	Party	VoterID
10760	Republican	19875
10604	Democratic	17739
2630	Democratic	10671
3976	Republican	14173
12724	Democratic	10092

This data contains synthetic data on 200000 voters in some small county who voted either Democrat or Republican. The majority of voters (55%) voted republican, while the minority voted for Democrats. Typically votes are not counted simultaneously and we are all familiar with \u201ccalling elections\u201d on the basis of a smaller subset of counted votes.

In the following snippet we would like students to experience the effects of samples of different sizes. Samples may provide different results from an even more decisive win of Republicans (like 60%+) to a slight win by Democrats.

\n\nSnippet 7.13: Distribution of votes for Democrats vs Republicans in a random sample of 100 voters may deviate from 40:60 split in the general population.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "7", "snippets": [{"Details": "

Just run the code several times and observe how the distribution of votes between Democrats and Republicans changes. Also, change the size of the sample from 100 to 1000 and perhaps even to 10000. Observe how more \u201cstable\u201d the results are and how much closer the vote distribution is now to the true vote distribution in the general population of voters.

The next snippet is the only one when we use a simple for loop in this entire textbook.

Below we display the histogram of 1000 samples of size 500 and we observe how the support for Democrats varies. Please observe that the histogram\u2019s shape approached Bell\u2019s curve with the mean of 45% (true voter support for Democrats). Notice however that there is a fraction of samples which would incorrectly have Democrats as winners, exceeding 50% of vote.

$\"\"$ \n

Figure 7.1 Democratic votes

", "PreExCode": "", "Query": "# Read in the data\nElections <-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/ElectionsID.csv\")\ncolnames(Elections)\n\nv <- sample(1:nrow(Elections))\nElections<-Elections[v,]\nSampleE<-Elections[1:100,]\ntable(SampleE$Party)", "Result": "", "Title": "Snippet 7.12: Distribution of votes for Democrats vs Republicans in a random sample of 100 voters may deviate from 40:60 split in the general population.", "Type": "R", "Video": "", "Width": ""}]}, "7.4": {"Name": "Law of Small Numbers", "Details": "The term \"Law of small numbers\" was coined by Kahneman in his book \"Thinking, Fast and Slow.\" This law states that small samples are more susceptible to producing extreme results compared to large samples. Kahneman provides an example related to the incidence of kidney cancers in small rural counties in the Midwest. Interestingly, kidney cancers appear to be less frequent in these counties, possibly due to healthier lifestyles and organic food consumption. However, kidney cancers are also found to be more prevalent in the same small rural counties, which can be attributed to poverty and limited access to healthcare. In reality, neither of these narratives is accurate. The occurrence of extreme results is simply more likely in small samples, similar to randomly selecting a small number of balls from an urn containing both red and black balls. With a small number of balls, extreme outcomes (such as all black or all red) occur more frequently compared to a larger selection.\n\nTo illustrate the law of small numbers, let's consider a dataset of car dealership sales. When analyzing small samples (e.g., size 10), we may observe that winter is the season when car sales dramatically exceed the expected seasonal sales (constituting approximately 25% of sales for each of the four seasons in our synthetic dataset). However, by examining different samples of size 10, we might conclude that winter is the season when sales significantly decline compared to expectations. Thus, we observe the opposite effect!\n\nTo understand the effects of the law of small numbers, run the provided code snippet multiple times, and also modify the sample size from 10 to 20, 30, and even down to 5. This will allow you to observe how the law of small numbers influences the results.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "7", "snippets": [{"Details": "", "PreExCode": "", "Query": "# Read in the data\nCars <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Cars2022.csv\")\nv<-sample(1:nrow(Cars))\nsampleCars<-Cars[v,]\ntable(sampleCars[1:10,]$Season)", "Result": "", "Title": "Snippet 7.13 Law of small numbers snippet", "Type": "R", "Video": "", "Width": ""}]}, "7.5": {"Name": "Randomness and repeatability", "Details": "In the realm of random data, observed trends do not persist or repeat themselves. They emerge and fade away with each new random sample.\n\nNow, let's consider the likelihood of GMC vehicles being purchased more frequently by females than males in Columbus, season after season, and sample after sample. If the data consistently follows a random pattern, meaning that each new set of data is generated randomly, it is highly probable that we will witness a reversal of this trend. In fact, with each new dataset, the trend may reverse completely. In the field of data science, we strive to identify enduring trends that can be regarded as representative of a broader pattern extending beyond the current sample. The entirety of the dataset is, after all, a subset of an unknown \"universal dataset.\" Here, we present an illustration of what occurs when the universal dataset is, in fact, randomly uniform.\n\nIt is crucial to note that if we generate new samples, we may witness a complete reversal of trends. In the same dealership, Toyota may emerge as the most popular brand while Honda becomes the least popular, and in another sample, Toyota and Honda may switch places.\n\nRandomness serves as the primary factor that deceives us and should be treated with utmost seriousness. Eliminating randomness as the \"cause\" of our observations stands as a fundamental objective in data science. Hypothesis testing enables us to account for randomness, and the concept of p-value provides a quantitative measure of the potential impact of random data.\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "7", "snippets": {}}, "7.6": {"Name": "Data sets with embedded patterns", "Details": "Using the DataMaker tool we have embedded the following two patterns into Cars data:\n\nSubaru in Austin is bought by ~ 96% females\n\nThe following snippet shows the skewed distribution of females due to the pattern injected by DataMaker into the cars data set.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "7", "snippets": [{"Details": "This pattern is intentionally designed to be valid, rather than misleading finding. We have purposely exaggerated it to significantly skew the gender distribution, with proportions exceeding 90%. However, what if we slightly deviate from a perfectly uniform or random distribution? For instance, what if we introduce a 60:40 distribution in the context of cars? In such cases, the challenge for statistical evaluation methods lies in not dismissing these true discoveries. As the actual pattern approaches the realm of possible random patterns, the likelihood of encountering false negatives increases. Thus, while we strive to reject false positives, we may inadvertently fall into the trap of false negatives and reject genuine discoveries. Exercising excessive caution might prevent us from uncovering real business value within our data. On the other hand, being overly adventurous could lead to false findings. Each of these two errors carries its own costs. So, is it preferable to err on the side of optimism rather than pessimism?", "PreExCode": "", "Query": "# Read in the data\nCarsB <- read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Cars2022b.csv')\nNarrow<-CarsB[CarsB$Car=='Subaru'& CarsB$Dealership =='Austin',]\ntable(Narrow$Buyer)\n\nfemalebuyers <- nrow(Narrow[Narrow$Buyer == 'Female',]) / nrow(Narrow)\ncat('frequency of female buyers within the narrow query')\nfemalebuyers ", "Result": "", "Title": "Snippet 7.14 Car data set with embedded pattern", "Type": "R", "Video": "", "Width": ""}]}, "7.7": {"Name": "\ud83c\udfa6 Hidden Variables (Video) \ud83c\udfa6", "Details": "Another common scenario where data can be misleading is the presence of hidden variables, also known as the dilemma between correlation and causation.\n\nTo illustrate the well-known issue of hidden variables, let's consider a synthetic dataset that describes the incomes of law school graduates from the top 100 universities, five years after graduation.\n\nEach data entry includes the income of a graduate, the ranking of the school they attended (ranging from 1 to 100), the tuition they paid, their SAT score, and their GPA. The prevailing belief is that the ranking of a school (and the tuition paid) is the primary factor influencing higher salaries. The notion is that the higher the school's rank, the higher the salary after five years of graduation. This is often emphasized by schools to attract prospective students, suggesting that higher tuition leads to greater future earnings\u2014a worthwhile investment.\n\nHowever, this belief is not entirely accurate.\n\nThe true variable that drives salaries is the SAT score. Students with higher SAT scores are more likely to be accepted into higher-ranked schools, while those with lower SAT scores tend to be admitted to lower-ranked schools. Consequently, as highlighted in a notable New York Times article, higher-ranked schools primarily accept academically stronger students, and the school's \"success\" is less related to their teaching methods and more associated with the academic caliber of the students they admit.\n\nThis relationship is illustrated by the linear correlation between income and the SAT score. Similarly, there exists a linear relationship between SAT scores and the school ranking (higher-ranked schools admit students with higher SAT scores), as well as a linear relationship between tuition and the school's rank (higher-ranked schools tend to charge higher tuitions).\n\nThis dilemma raises a fundamental question regarding causality versus correlation. The SAT score serves as the hidden variable in this example, which can potentially mislead us into believing that income is driven by school rank or tuition. In reality, both variables are influenced by the SAT score. Therefore, in this dataset, it is the SAT score that ultimately drives the incomes.\n\nIn this example, the SAT score functions as the hidden variable.\n\n\n\n\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1q4Y0WdppuN70oFTCHZklVABPbXpeUnlo/preview", "child": {}, "parent": "7", "snippets": [{"Details": "Notice also that GPA in this data set has really no correlation with Income. Of course in reality GPA may be correlated with income, here in our synthetic data set it is not. One can observe it by plotting Income as a function of GPA.", "PreExCode": "", "Query": "IS<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/IncomeSchool.csv\")\ncat(\"Correlation is not causation. Even though relationships are linear - the causality seem to obvious between SAT scores and incomes. So called best schools simply select best students - now wonder they claim their high tuition is worth paying\")\nplot(IS$SAT, IS$Income)\nplot(IS$Tuition, IS$Income)\nplot(IS$SchoolRank, IS$Income)\n\nplot(IS$SchoolRank, IS$Income)\nplot(IS$SAT, IS$Income)\nplot(IS$SAT, IS$SchoolRank)", "Result": "", "Title": "Snippet 7.15 Incomes of school graduates \u2013 the case of Hidden Variable", "Type": "R", "Video": "", "Width": ""}]}, "7.8": {"Name": "Additional References", "Details": "RPubs article on Sampling Distribution and the Central Limit Theorem - https://rpubs.com/ranjeetapegu/Central-Limit-Theorem\n\nAnswering Questions with Data (Central Limit Theorem) - https://www.crumplab.com/statistics/04-SamplesPopulations.html#the-central-limit-theorem\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "7", "snippets": {}}}, "parent": "7", "snippets": [{"Details": "We observe that each car make is consistently sold approximately 9% of the time, and this 9% transaction frequency holds across all dealerships spanning from LA to NYC. Furthermore, it is evident that roughly 25% of transactions occur in each season. Additionally, the distribution of buyers' gender remains evenly split, with a 50:50 ratio in all transactions, regardless of the car or dealership. In essence, the data exhibits an unrealistically uniform pattern, lacking any noteworthy variations.\r\n\r\nHowever, even in this entirely random dataset, some seemingly intriguing findings surface. In the following sections, we will showcase a few examples of these false discoveries and subsequently delve into explanations for how such findings can arise solely from randomness. We will explore the potential deception that can result from these random discoveries and discuss strategies to avoid placing undue trust in them. In the subsequent subsection, we will address the issues related to sampling and the risks associated with drawing conclusions based on data samples.\"\r\n", "PreExCode": "", "Query": "# Read in the data\nCars <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Cars2022.csv\")\n\nnrow(Cars)\nsummary(Cars)\nunique(Cars$Dealership)\nunique(Cars$Car)\nunique(Cars$Season)\nunique(Cars$Buyer)\ntable(Cars$Dealership)\ntable(Cars$Car)\ntable(Cars$Season)\ntable(Cars$Buyer)\ntapply(Cars$Buyer_Age, Cars$Car, mean)\ntapply(Cars$Buyer_Age, Cars$Dealership, mean)\ntapply(Cars$Buyer_Age, Cars$Season, mean)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "8": {"Name": "\ud83c\udfa6 Central Limit Theorem - The special role of Normal distribution \ud83c\udfa6", "Details": "The central limit theorem states that when you have a population with a mean value of \u03bc and a standard deviation of \u03c3, and you take sufficiently large random samples from the population with replacement, the distribution of the sample means will be approximately normally distributed. It is worth noting how surprising this is. Regardless of the original distribution of the data, the distribution of means from the samples will be normal. In section 6 of the active textbook, which covers data science and databases, there is an illustration of the normal distribution of means obtained from samples of Airbnb rental prices. Despite the fact that the price distribution of Airbnb rentals is far from normal, in fact, it is multimodal, the distribution of means derived from the samples follows a normal distribution.\n\n

$\"\"$ \n

Figure 8.1: Distribution of prices of apartments in Airbnb rentals

\n\n

$\"\"$ \n

Figure 8.2: Distribution of sample mean apartment rental price in airbnb dataset for 10000 samples each of size = 100

\n\n\n

$\"\"$ \n

Figure 8.3: Distribution of sample mean apartment rental price in airbnb dataset for 10000 samples each of size = 1000

\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/15VuwTD_XDfYfz5tvqNAshb7JtfcMV6RZ/preview", "child": {"8.1": {"Name": "\ud83c\udfa6 Normal Distribution - video \ud83c\udfa6 ", "Details": "

Let us now introduce the normal distribution and its characteristic Bell curve.

$\"\"$ \n

Figure 8.4 Histogram of normal distribution generated in snippet 8.1

There are two important parameters of normal distribution: mean and standard deviation. The mean of histogram from 8.4 is equal to 10 and the standard deviation is equal to 1.

Standard deviation is a very important measure of data spread. It is as important as the mean value, since it tells us how widely or narrowly data is spread around the mean. Mathematically, standard deviation is defined by the following formula:

$\"\"$ \n

The squaring of differences between data points and the mean value is important since it treats negative and positive differences identically. Dividing by the number of data points is important for normalization.

Given a numerical variable x, sd(x) is the R function which computes standard deviation. It is demonstrated in the snippet 8.1

These will be useful later in hypothesis testing.

In the following snippet we use two functions rnorm() and pnorm(). Function rnorm() generates a number of data points according to normal distribution assuming the mean=10 and standard deviation (sd) =1. Normal distribution is the core distribution in data science (although power law distribution is also gaining ground as we explain in a separate section). Normal distribution plays a special role due to the Central Limit Theorem ??.

Feel free to change parameters of rnorm() in snippet 8.1 and see the effect of mean and standard deviation in the shapes of resulting bell curves.

The statement length(normal[normal >x]) shows the number of data points which are larger than x. In the snippet x=12. Notice how quickly length(normal[normal >x]) goes down to zero when one increases x. The statement z<-(12-10)/10 calculate the z-value, which is defined as difference between x and the mean of the distribution divided by standard deviation. This helps normalization and allows talking about vastly different data sets in the same terms.

Z-value is the distance measured in sigmas, where sigma stands for one standard deviation.

The function pnorm(z) calculates the fraction of data points in normal distribution whose z-value is less than z. Regardless of what standard deviation and mean of a given data set is, this fraction is always the same for the same values of z. Z-value is the common denominator normalizing all data distributions. With z-values we may disregard the range of data as well as its mean. It allows us to define distance from the mean in units which are standard deviations (sigmas).

The reader is encouraged to change all the parameters of this snippet and see the effect of z-value on the number of values exceeding z. This fraction becomes negligible roughly around z approaching 3. For z-values larger than 5, one cannot observe a single value with z>5, unless the data set is very large. Thus, the decline of the Gaussian Bell curve is very steep!

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1EQuDbCtzXx_srrdwwAEWK8_YZvspN5Gj/preview", "child": {}, "parent": "8", "snippets": [{"Details": "Next snippet shows the concept of z-value and shows the function pnorm()", "PreExCode": "", "Query": "y<-rnorm(10000, mean = 10, sd = 0.1)\nhist(y, main = \"Normal DIstribution\")\nlength(y)\nlength(y[y>10.3])\ncat('Lets now increase standard deviation 10 fold and see how what is the number of values above 10.3')\ny<-rnorm(10000, mean = 10, sd = 1)\nhist(y, main = \"Normal Distribution with larger spread\")\nlength(y[y>10.3])\nlength(y[y>12])\ncat('Lets use pnorm function to see the fraction of the bell curve which is smaller than 3z where z is the z-value')\npnorm(3)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "A popular measures of centrality of a sample or population include the mean and the median. To effectively describe a sample or population, a measure of the variability is also needed. The variance and the standard deviation (equal to the square root of the variance) are useful measures of variability. There are different formulas for the variance and standard deviation, depending on whether your data represents an entire population of scores or a sample (subset) of scores from the population.\n\nIf data represent a Population: Divide by n.\nIf data represent a Sample: Divide by n-1.\n\nLets understanding variance and standard deviation better using the snippet below. For this example, assume the data represent a sample:", "PreExCode": "", "Query": "y<-rnorm(10000, mean = 10, sd = 1)\ncat('what fraction of values are smaller than 12.5?')\ncat('z-value of 12.5 is 2.5')\ncat('One way to compute this is to use pnorm function')\npnorm(2.5)\ncat('Another is to use length functon')\nlength(y[y<12.5])\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "x <- c(8,2,6,4,6,2,4,4) \n\n(mean(x)-x) #differences from the mean\nsum(mean(x)-x) #sum the differences from the mean\n\n(mean(x)-x)^2 #differences from the mean squared\nSS=sum((mean(x)-x)^2) #sum of differences from the mean squared\n\nn <- length(x) #number of observations\n\nvariance <- SS/(n-1)\nstandard_deviation <- sqrt(variance)\nstandard_deviation\n\nsd(x) #confirm the computation with the sd() function", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "# Define the z0 parameter\nz0 <- 1.5 # Replace with the desired z-score threshold\n\n# Create a sequence of z-scores\nz_scores <- seq(-4, 4, length.out = 1000)\n\n# Calculate the density of the normal distribution\ndensity <- dnorm(z_scores)\n\n# Plot the normal distribution\nplot(z_scores, density, type = \"l\", col = \"blue\", lwd = 2,\n main = \"Normal Distribution with Highlighted Area for Z-scores > z0\",\n xlab = \"Z-score\", ylab = \"Density\")\n\n# Highlight the area where z-scores are greater than z0\nx_highlight <- seq(z0, 4, length.out = 1000)\ny_highlight <- dnorm(x_highlight)\npolygon(c(z0, x_highlight, 4), c(0, y_highlight, 0), col = \"red\", border = NA)\n\n# Add a vertical line at z0\nabline(v = z0, col = \"black\", lty = 2)\n\n# Add legend\nlegend(\"topright\", legend = c(\"Normal Distribution\", \"Area > z0\"),\n col = c(\"blue\", \"red\"), lwd = 2, lty = c(1, NA), fill = c(NA, \"red\"))\n\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "8", "snippets": {}}, "9": {"Name": "Ask GPT: Confidence intervals for\nestimation of the mean", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"9.1": {"Name": "Confidence Interval for means", "Details": "

Prompt: Introduce notions of confidence interval for means and confidence level using sample of 500 tuples from airbnb database and mean of rental price

To introduce the notions of confidence interval for means and confidence level using a sample of 500 tuples from an Airbnb database and the mean rental price, follow these steps:

Confidence Interval for Means

A confidence interval gives a range of values for the mean rental price, within which we are fairly certain the true population mean lies. This interval is calculated using the sample mean, standard deviation, and sample size.

\n\n

Confidence Level

The confidence level represents the probability that the confidence interval contains the true population mean. Common confidence levels are 90%, 95%, and 99%.

\n\n

Example Calculation

\n\n

Let's say we have a sample of 500 tuples from the Airbnb database, and we want to calculate the 95% confidence interval for the mean rental price.

\n\n

Step 1: Calculate the Sample Mean and Standard Deviation

\n\n

Assume the sample mean ( $\n \n \n \overset{\ud835\udc65}{\n} \n \n \n$ ) is $150, and the sample standard deviation ( $\n \n \n \ud835\udc60 \n \n \n$ ) is $30.

\n\n

Step 2: Determine the Confidence Level and Find the Critical Value

\n\n

For a 95% confidence level, the critical value ( $\n \n \n \n^{\ud835\udc67 \n \u2217 \n} \n \n \n$ ) from the standard normal distribution is approximately 1.96.

\n\n

Step 3: Calculate the Standard Error of the Mean

\n\n

The standard error of the mean (SEM) is calculated using the formula:\n $\n \n \n SEM \n = \n \frac{\n}{\ud835\udc60} \n \n \n$ \n

\n\n

Where:

\n\n

$\n \n \n \ud835\udc60 \n \n \n$ is the sample standard deviation.
$\n \n \n \ud835\udc5b \n \n \n$ is the sample size.

\n\n

For our example:\n $\n \n \n SEM \n = \n \frac{\n}{30} \n \u2248 \n 1.34 \n \n \n$

$\n \n \n \n \n \n$

\n\n

Step 4: Calculate the Margin of Error

\n\n

The margin of error (MOE) is calculated using the formula:\n $\n \n \n MOE \n = \n \n^{\ud835\udc67 \n \u2217 \n} \n \u00d7 \n SEM \n \n \n$ \n

\n\n

For our example:\n $\n \n \n MOE \n = \n 1.96 \n \u00d7 \n 1.34 \n \u2248 \n 2.63 \n \n \n$

Step 5: Calculate the Confidence Interval

\n\n\n\n

The confidence interval is calculated using the formula:\n $\n \n \n CI \n = \n \overset{\ud835\udc65}{\n} \n \u00b1 \n MOE \n \n \n$ \n

\n\n

For our example:\n $\n \n \n CI \n = \n 150 \n \u00b1 \n 2.63 \n \n \n$ \n

\n\n

This gives us a confidence interval of:\n $\n \n \n 147.37 \n \u2264 \n \ud835\udf07 \n \u2264 \n 152.63 \n \n \n$

Interpretation

\n\n

With a 95% confidence level, we can say that we are 95% confident that the true mean rental price of the population lies between $\n 147.37 \n \ud835\udc4e \n \ud835\udc5b \n \ud835\udc51 \n$ 152.63.

Summary

\n\n\n

Confidence Interval: Provides a range of values within which the true mean rental price is likely to fall.
Confidence Level: Indicates the probability that the confidence interval contains the true population mean.
Example Calculation: Using a sample of 500 tuples with a mean rental price of $150 and a standard deviation of $30, the 95% confidence interval is $147.37 to $152.63.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "9", "snippets": {}}, "9.2": {"Name": "Computing Confidence interval and confidence level for Airbnb data frame", "Details": "

Prompt : Compute in R the confidence interval and confidence level based on the following sample df from Airbnb data frame

\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "9", "snippets": [{"Details": "", "PreExCode": "", "Query": "# Load necessary libraries\nlibrary(stats)\n\n# Load the Airbnb data frame from the given URL\nairbnb_df <- read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/airbnb.csv')\n\n# Sample 500 tuples from the Airbnb data frame\nset.seed(123) # For reproducibility\nsample_df <- airbnb_df[sample(nrow(airbnb_df), 500), ]\n\n# Compute the sample mean and standard deviation\nsample_mean <- mean(sample_df$price, na.rm = TRUE)\nsample_sd <- sd(sample_df$price, na.rm = TRUE)\n\n# Determine the sample size\nn <- nrow(sample_df)\n\n# Compute the standard error of the mean\nsem <- sample_sd / sqrt(n)\n\n# Determine the critical value for a 95% confidence level\nconfidence_level <- 0.95\nalpha <- 1 - confidence_level\ncritical_value <- qnorm(1 - alpha / 2)\n\n# Compute the margin of error\nmargin_of_error <- critical_value * sem\n\n# Compute the confidence interval\nlower_bound <- sample_mean - margin_of_error\nupper_bound <- sample_mean + margin_of_error\n\n# Print the results\ncat(\"Sample Mean:\", sample_mean, \"\\n\")\ncat(\"Sample Standard Deviation:\", sample_sd, \"\\n\")\ncat(\"Sample Size:\", n, \"\\n\")\ncat(\"Standard Error of the Mean:\", sem, \"\\n\")\ncat(\"Critical Value:\", critical_value, \"\\n\")\ncat(\"Margin of Error:\", margin_of_error, \"\\n\")\ncat(\"95% Confidence Interval: [\", lower_bound, \",\", upper_bound, \"]\\n\")\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "9.3": {"Name": "Given margin of error compute confidence level and confidence interval", "Details": "

Prompt: Let's assume we want the margin of error for the price to be just $1. What would the confidence interval be? What would the confidence level be? Can you calculate it in R using a sample data frame from the Airbnb data

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "9", "snippets": [{"Details": "", "PreExCode": "", "Query": "# Load necessary libraries\nlibrary(stats)\n\n# Load the Airbnb data frame from the given URL\nairbnb_df <- read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/airbnb.csv')\n\n# Sample 500 tuples from the Airbnb data frame\nset.seed(123) # For reproducibility\nsample_indices <- sample(nrow(airbnb_df), 500)\nsample_df <- airbnb_df[sample_indices, ]\n\n# Compute the sample mean and standard deviation\nsample_mean <- mean(sample_df$price, na.rm = TRUE)\nsample_sd <- sd(sample_df$price, na.rm = TRUE)\n\n# Determine the sample size\nn <- nrow(sample_df)\n\n# Compute the standard error of the mean\nsem <- sample_sd / sqrt(n)\n\n# Desired margin of error\ndesired_margin_of_error <- 1\n\n# Calculate the critical value required to achieve the desired margin of error\ncritical_value <- desired_margin_of_error / sem\n\n# Determine the corresponding confidence level\nconfidence_level <- 2 * pnorm(critical_value) - 1\n\n# Compute the confidence interval\nlower_bound <- sample_mean - desired_margin_of_error\nupper_bound <- sample_mean + desired_margin_of_error\n\n# Print the results\ncat(\"Sample Mean:\", sample_mean, \"\\n\")\ncat(\"Sample Standard Deviation:\", sample_sd, \"\\n\")\ncat(\"Sample Size:\", n, \"\\n\")\ncat(\"Standard Error of the Mean:\", sem, \"\\n\")\ncat(\"Critical Value:\", critical_value, \"\\n\")\ncat(\"Desired Margin of Error:\", desired_margin_of_error, \"\\n\")\ncat(\"Corresponding Confidence Level:\", confidence_level * 100, \"%\\n\")\ncat(\"Confidence Interval with Margin of Error $1: [\", lower_bound, \",\", upper_bound, \"]\\n\")\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "9.4": {"Name": "Compute sample size necessary to achieve confidence level for given margin of error\n", "Details": "Prompt: How many more tuples would have to be in the sample of airbnb to achieve 90% confidence interval for $1 margin ?
\n

To determine the number of tuples required in the sample to achieve a confidence interval with a $1 margin of error at a 90% confidence level, we can follow these steps:

\n
Identify the critical value (z-score) for a 90% confidence level: This can be found using the standard normal distribution.
\n
\n
Use the formula for margin of error:
$\n \n \n Margin of Error \n = \n \ud835\udc67 \n \u00d7 \n \n (\n \frac{\n}{\ud835\udf0e} \n) \n \n \n \n$ \n
Where $\n \n \n \ud835\udf0e \n \n \n$ is the standard deviation of the sample, $\n \n \n \ud835\udc5b \n \n \n$ is the sample size, and $\n \n \n \ud835\udc67 \n \n \n$ is the critical value.
\n
\n
Rearrange the formula to solve for the required sample size $\n \n \n \ud835\udc5b \n \n \n$ :
$\n \n \n \ud835\udc5b \n = \n \n^{\n (\n \frac{\n}{\n \ud835\udc67 \n \u00d7 \n \ud835\udf0e \n} \n) \n \n 2 \n} \n \n \n$ \n

Let's assume we have a sample standard deviation ( $\n \n \n \ud835\udf0e \n \n \n$ ) based on the sample data.

Here's how you can calculate it in R:

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "9", "snippets": [{"Details": "", "PreExCode": "", "Query": "# Load necessary libraries\nlibrary(stats)\n\n# Load the Airbnb data frame from the given URL\nairbnb_df <- read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/airbnb.csv')\n\n# Sample 500 tuples from the Airbnb data frame\nset.seed(123) # For reproducibility\nsample_indices <- sample(nrow(airbnb_df), 500)\nsample_df <- airbnb_df[sample_indices, ]\n\n# Compute the sample standard deviation\nsample_sd <- sd(sample_df$price, na.rm = TRUE)\n\n# Desired margin of error\ndesired_margin_of_error <- 1\n\n# Critical value for 90% confidence level\nconfidence_level <- 0.90\nalpha <- 1 - confidence_level\nz_score <- qnorm(1 - alpha / 2)\n\n# Compute the required sample size\nrequired_sample_size <- (z_score * sample_sd / desired_margin_of_error)^2\n\n# Print the results\ncat(\"Sample Standard Deviation:\", sample_sd, \"\\n\")\ncat(\"Critical Value for 90% Confidence Level:\", z_score, \"\\n\")\ncat(\"Desired Margin of Error:\", desired_margin_of_error, \"\\n\")\ncat(\"Required Sample Size:\", ceiling(required_sample_size), \"\\n\")\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "9", "snippets": {}}, "10": {"Name": "Ask GPT: Confidence Intervals for estimation of proportions", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"10.1": {"Name": "Confidence Interval for Proportions", "Details": "

Prompt : Introduce calculation of confidence intervals for proportions. Use Voting as example, say we have Biden vs Trump

To introduce the calculation of confidence intervals for proportions, let's use the example of a voting scenario between two candidates, Biden and Trump. Suppose we have survey data on the proportion of voters who support each candidate.

\n\n

Example Scenario

Let's say we surveyed 1000 voters, and 540 of them indicated they would vote for Biden. We want to calculate a 95% confidence interval for the proportion of voters who support Biden.

Step-by-Step Calculation

\n
Calculate the sample proportion: This is the proportion of the sample that supports Biden.
$\\hat{p} = \\frac{x}{n}$ \n
Where:
\n
- $\\hat{p}$ is the sample proportion.
- $\n \n \n x \n \n \n \n$ is the number of voters who support Biden.
- $n$ is the total number of voters surveyed.
\n
For our example:
$\\hat{p} = \\frac{540}{1000} = 0.54$ \n
$Determine the standard error (SE) : This measures the variability of the sample proportion.$ \n $SE = \\sqrt{\\frac{\\hat{p} \\cdot (1 - \\hat{p})}{n}}$ \n
For our example:
$SE = \\sqrt{\\frac{0.54 \\cdot (1 - 0.54)}{1000}} = \\sqrt{\\frac{0.54 \\cdot 0.46}{1000}} = \\sqrt{0.0002484} \\approx 0.01576$ \n
\n
Find the critical value (z) for a 95% confidence level*: For a 95% confidence interval, the critical value from the standard normal distribution is approximately 1.96.
\n
\n
Calculate the margin of error (MOE):
$MOE = z^* \\cdot SE$ $MOE = 1.96 \\cdot 0.01576 \\approx 0.03088$ \n
\n
Determine the confidence interval (CI):
$\\text{CI} = \\hat{p} \\pm MOE$ $\\text{CI} = 0.54 \\pm 0.03088$ $0.50912 \\leq p \\leq 0.57088$ \n

R Implementation

Here is how you can calculate this in R:

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "10", "snippets": [{"Details": "", "PreExCode": "", "Query": "# Given data\nn <- 1000 # Total number of voters surveyed\nvoters_data <- read.csv(\"https://raw.githubusercontent.com/VishMenon23/Data101_Dataset/main/Election/voters_survey.csv\")\n\n# Biden support\np_hat <- sum(voters_data$Vote == \"Biden\") / n\n\n# Standard error\nSE <- sqrt((p_hat * (1 - p_hat)) / n)\n\n# Critical value for 95% confidence level\nz <- 1.96\n\n# Margin of error\nMOE <- z * SE\n\n# Confidence interval\nlower_bound <- p_hat - MOE\nupper_bound <- p_hat + MOE\n\n# Print the results\ncat(\"Biden support:\", p_hat, \"\\n\")\ncat(\"Standard Error:\", SE, \"\\n\")\ncat(\"Margin of Error:\", MOE, \"\\n\")\ncat(\"95% Confidence Interval: [\", lower_bound, \",\", upper_bound, \"]\\n\")", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "10.2": {"Name": "Computing confidence interval for proportions with given confidence level\n", "Details": "

Prompt : Let df represent spending amounts by people in shopping mall varying between $5 and $200 and distributed normally. Determine with 99% confidence estimate of proportion of people who spent more than $100 in R\n

\n\n

To determine the 99% confidence estimate of the proportion of people who spent more than $25 in a shopping mall, we need to follow these steps:

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "10", "snippets": [{"Details": "", "PreExCode": "", "Query": "n <- 1000 # Total number of voters surveyed\ndf <- read.csv(\"https://raw.githubusercontent.com/VishMenon23/Data101_Dataset/main/Election/spending_data.csv\")\n\n# Calculate the proportion of people who spent more than $100\np_hat <- sum(df$Spending > 100) / n\n\n# Standard error of the proportion\nSE <- sqrt((p_hat * (1 - p_hat)) / n)\n# Critical value for 99% confidence level\nz <- qnorm(1 - 0.005)\n\n# Margin of error\nMOE <- z * SE\n\n# Confidence interval\nlower_bound <- p_hat - MOE\nupper_bound <- p_hat + MOE\n\n# Print the results\ncat(\"Sample Proportion:\", p_hat, \"\\n\")\ncat(\"Standard Error:\", SE, \"\\n\")\ncat(\"Margin of Error:\", MOE, \"\\n\")\ncat(\"99% Confidence Interval: [\", lower_bound, \",\", upper_bound, \"]\\n\")", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "10.3": {"Name": "Computing confidence level from sample data", "Details": "

Prompt : Suppose that we want to determine the fraction of spendings above $150 within margin of error of 1%. What confidence level could we get from this sample?\n

\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "10", "snippets": [{"Details": "", "PreExCode": "", "Query": "n <- 1000\ndf <- read.csv(\"https://raw.githubusercontent.com/VishMenon23/Data101_Dataset/main/Election/spending_data.csv\")\n\n# Calculate the proportion of people who spent more than $150\np_hat <- sum(df$Spending > 150) / n\n# Calculate the sample proportion of people who spent more than $300\np_hat <- sum(df$Spending > 150) / n\n\n# Desired margin of error\ndesired_moe <- 0.01\n\n# Standard error of the proportion\nSE <- sqrt((p_hat * (1 - p_hat)) / n)\n\n# Determine the z-score needed for the desired margin of error\nz <- desired_moe / SE\n\n# Calculate the confidence level corresponding to the z-score\nconfidence_level <- 2 * pnorm(z) - 1\n\n# Print the results\ncat(\"Sample Proportion of spendings > $150:\", p_hat, \"\\n\")\ncat(\"Standard Error:\", SE, \"\\n\")\ncat(\"Desired Margin of Error:\", desired_moe, \"\\n\")\ncat(\"Z-score required:\", z, \"\\n\")\ncat(\"Confidence Level:\", confidence_level * 100, \"%\\n\")", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "10.4": {"Name": "Computing Confidence level and sample size", "Details": "

Prompt : Get confidence level for proportion of spenders who spend at least  $50 within margin of error of 1%. How much bigger sample df we need to get this confidence level to 90%?\n

\n\n

To determine the required sample size to achieve a 90% confidence level with a margin of error of 1% for the proportion of spenders above $50, we can use the formula for the sample size of a proportion:

$n = \\left( \\frac{Z^2 \\cdot p \\cdot (1-p)}{E^2} \\right)$

Where:

$n$ = required sample size
$Z$ = Z-value (critical value) corresponding to the desired confidence level
$p$ = estimated proportion of spenders above $50
$E$ = margin of error

For a 90% confidence level, the Z-value (Z-score) is approximately 1.645.

Assuming we don't know the proportion ( $p$ ), we use 0.5 as it provides the maximum variability and thus the largest sample size, which is a conservative estimate. The margin of error ( $E$ ) is 0.01 (1%).

Let's calculate the required sample size:

$n = \\left( \\frac{(1.645)^2 \\cdot 0.5 \\cdot (1-0.5)}{(0.01)^2} \\right)$

$n = \\left( \\frac{2.706025 \\cdot 0.25}{0.0001} \\right)$ $n = \\left( \\frac{0.67650625}{0.0001} \\right)$

$\n \n \n n \n = \n 6765.0625 \n \n \n$

So, the required sample size is approximately 6766.

To achieve a 90% confidence level with a margin of error of 1% for the proportion of spenders above $50, you would need a sample size of about 6766.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "10", "snippets": {}}, "10.5": {"Name": "Compute confidence interval for supermarket transaction data", "Details": "

Prompt : let df be a data frame of 500 transactions in supermarket.  Attributes  of df are Bread, Butter, Milk Eggs and Water with binary values.  Values are Yes/No (bought this item or not). Establish 98% confidence interval for proportion of all transactions which buy Bread when they buy Milk\n

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "10", "snippets": [{"Details": "", "PreExCode": "", "Query": "# Load necessary library\nlibrary(stats)\n\n# Read the CSV file\ndf <- read.csv(\"https://raw.githubusercontent.com/VishMenon23/Data101_Dataset/main/Election/grocery.csv\")\n\n# Filter transactions where Milk is bought\nmilk_transactions <- df[df$Milk == \"Yes\", ]\n\n# Calculate the proportion of these transactions where Bread is also bought\np_hat <- mean(milk_transactions$Bread == \"Yes\")\n\n# Number of transactions where Milk is bought\nn <- nrow(milk_transactions)\n\n# Calculate the standard error\nse <- sqrt(p_hat * (1 - p_hat) / n)\n\n# Calculate the 98% confidence interval\nz <- qnorm(0.99) # for 98% confidence interval, we use 0.99 because it's two-tailed\nci_lower <- p_hat - z * se\nci_upper <- p_hat + z * se\n\n# Print the results\ncat(\"The 98% confidence interval for the proportion of transactions that buy Bread when they buy Milk is:\", ci_lower, \"to\", ci_upper, \"\\n\")\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "10.6": {"Name": "Estimating survey size for elections\n", "Details": "

Prompt : How many people need to be surveyed in order to estimate Biden support with 99% confidence to be within the margin of 1% error? \n

\n\n

To determine the required sample size to estimate Biden's support with 99% confidence and a margin of error of 1%, we can use the formula for sample size estimation in proportions. Here's the derivation and the R code:

Derivation of the Sample Size Formula

1. Margin of Error (E): The margin of error is the maximum allowable difference between the sample proportion and the true population proportion. For this problem, the margin of error is 0.01 (1%).

2. Confidence Level (Z): The confidence level is 99%, which corresponds to a Z-value (critical value) from the standard normal distribution. For 99% confidence, the Z-value is approximately 2.576.

3. Estimated Proportion (p): Since we do not have an initial estimate, we use p = 0.5, which maximizes the required sample size and is a conservative approach.

The formula for the sample size $n$ is:

$n = \\frac{Z^2 \\cdot p \\cdot (1 - p)}{E^2}$

Where:

$Z$ is the Z-value (2.576 for 99% confidence)
$p$ is the estimated proportion (0.5)
$E$ is the margin of error (0.01)

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "10", "snippets": {}}, "10.7": {"Name": "Estimate who wins according to survey", "Details": "\nPrompt : Assuming margin of error of 1% show confidence intervals for biden and trump vote among voters under 30\n
\n\n

This code calculates the proportion of voters supporting each candidate, and computes the confidence intervals for both the overall population and young voters.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "10", "snippets": [{"Details": "", "PreExCode": "", "Query": "# Read the CSV file\nsurvey_data <- read.csv(\"https://raw.githubusercontent.com/VishMenon23/Data101_Dataset/main/Election/survey_data.csv\")\n\n# Define age group for young voters\nyoung_voters <- survey_data[survey_data$age <= 29, ]\n\n# Calculate proportion of voters for each candidate\ntotal_biden_support <- mean(survey_data$candidate == \"Biden\")\ntotal_trump_support <- mean(survey_data$candidate == \"Trump\")\n\nyoung_biden_support <- mean(young_voters$candidate == \"Biden\")\nyoung_trump_support <- mean(young_voters$candidate == \"Trump\")\n\n# Calculate confidence intervals\ntotal_margin_error <- 0.01\nyoung_margin_error <- 0.02\n\ntotal_biden_ci <- c(total_biden_support - total_margin_error, total_biden_support + total_margin_error)\ntotal_trump_ci <- c(total_trump_support - total_margin_error, total_trump_support + total_margin_error)\n\nyoung_biden_ci <- c(young_biden_support - young_margin_error, young_biden_support + young_margin_error)\nyoung_trump_ci <- c(young_trump_support - young_margin_error, young_trump_support + young_margin_error)\n\n# Print results\ncat(\"Overall:\\n\")\ncat(\"Biden support:\", total_biden_support * 100, \"%\\n\")\ncat(\"Trump support:\", total_trump_support * 100, \"%\\n\")\ncat(\"Biden confidence interval:\", total_biden_ci * 100, \"%\\n\")\ncat(\"Trump confidence interval:\", total_trump_ci * 100, \"%\\n\\n\")\n\ncat(\"Young Voters (age <= 29):\\n\")\ncat(\"Biden support:\", young_biden_support * 100, \"%\\n\")\ncat(\"Trump support:\", young_trump_support * 100, \"%\\n\")\ncat(\"Biden confidence interval:\", young_biden_ci * 100, \"%\\n\")\ncat(\"Trump confidence interval:\", young_trump_ci * 100, \"%\\n\")\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "10.8": {"Name": "Confidence interval, given confidence level for vote of 65+ age group", "Details": "Prompt : Who wins 65+ voter group and what is the confidence interval for confidence level of 95%?
\n\n

To determine who wins the 65+ voter group and calculate the 95% confidence interval, we can use the same methodology as before. We'll filter the dataset to include only respondents who are 65 years old or older, calculate the proportion of voters for each candidate, and then determine the confidence interval.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "10", "snippets": [{"Details": "", "PreExCode": "Assume that survey of 1000 people says Harris - 49%, Trump - 46%. Are these results within margin of error?", "Query": "# Read the CSV file\nsurvey_data <- read.csv(\"https://raw.githubusercontent.com/VishMenon23/Data101_Dataset/main/Election/survey_data.csv\")\n\n# Calculate proportion of Biden voters\ntotal_biden_support <- mean(survey_data$candidate == \"Biden\")\n\n# Define sample size\nn <- 5000\n\n# Calculate 99% confidence margin of error\nz_score <- 2.576\nmargin_of_error_99 <- z_score * sqrt((total_biden_support * (1 - total_biden_support)) / n)\n\n# Calculate 99% confidence interval\nbiden_ci_99 <- c(total_biden_support - margin_of_error_99, total_biden_support + margin_of_error_99)\n\n# Print results\ncat(\"Biden support:\", total_biden_support * 100, \"%\\n\")\ncat(\"99% Confidence Margin of Error:\", margin_of_error_99 * 100, \"%\\n\")\ncat(\"99% Confidence Interval for Biden's vote:\", biden_ci_99 * 100, \"%\\n\")\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "10.9": {"Name": "Survey of 1000 says Harris - 49%. Trump 46%..are these results within margin of error?", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "10", "snippets": [{"Details": "", "PreExCode": "Assume that survey of 1000 people says Harris - 49%, Trump - 46%. Are these results within margin of error?", "Query": "# Define the variables\nn <- 1000 # sample size\np_harris <- 0.49 # proportion for Harris\np_trump <- 0.46 # proportion for Trump\nz <- 1.96 # Z-score for 95% confidence level\n\n# Calculate the margin of error for Harris\nmoe_harris <- z * sqrt((p_harris * (1 - p_harris)) / n)\n\n# Calculate the margin of error for Trump\nmoe_trump <- z * sqrt((p_trump * (1 - p_trump)) / n)\n\n# Calculate the confidence intervals for Harris\nci_harris_lower <- p_harris - moe_harris\nci_harris_upper <- p_harris + moe_harris\n\n# Calculate the confidence intervals for Trump\nci_trump_lower <- p_trump - moe_trump\nci_trump_upper <- p_trump + moe_trump\n\n# Output the results\ncat(\"Confidence Interval for Harris: [\", ci_harris_lower, \", \", ci_harris_upper, \"]\\n\")\ncat(\"Confidence Interval for Trump: [\", ci_trump_lower, \", \", ci_trump_upper, \"]\\n\")\ncat(\"Margins of Error: Harris =\", moe_harris, \", Trump =\", moe_trump, \"\\n\")", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "Now, lets run 100,000 simulations of polls with the distribution 49:46:5 between Harris, Trump and unknowns and see how often can Trump win?", "PreExCode": "", "Query": "#How much do the polls lie?\n\n# Step 1: Generate population data of size 100,000\npopulation_size <- 100000\nharris_support <- 0.49\ntrump_support <- 0.46\nuncommitted_support <- 1 - harris_support - trump_support\n\n# Generate the population with 49% for Harris, 46% for Trump, and the rest uncommitted\npopulation <- sample(c(\"Harris\", \"Trump\", \"Uncommitted\"), size = population_size, \n prob = c(harris_support, trump_support, uncommitted_support), replace = TRUE)\n\n# Step 2: Run 100 surveys, each with a sample size of 1000\nsurvey_size <- 1000\nnum_surveys <- 100\ntrump_wins <- 0\nresults <- data.frame(Survey = integer(), Harris_Support = numeric(), Trump_Support = numeric())\n\nfor (i in 1:num_surveys) {\n # Take a random sample of 1000 from the population\n survey_sample <- sample(population, size = survey_size, replace = TRUE)\n \n # Calculate the proportion of support for each candidate in this survey\n harris_support_survey <- mean(survey_sample == \"Harris\")\n trump_support_survey <- mean(survey_sample == \"Trump\")\n \n # Store the survey results\n results <- rbind(results, data.frame(Survey = i, Harris_Support = harris_support_survey, \n Trump_Support = trump_support_survey))\n \n # Count how many surveys Trump won (i.e., Trump support > Harris support)\n if (trump_support_survey > harris_support_survey) {\n trump_wins <- trump_wins + 1\n }\n}\n\n# Step 3: Display how many times Trump won\ncat(\"Trump won\", trump_wins, \"out of\", num_surveys, \"surveys.\\n\")\n\n# Step 4: Show the most extreme polls won by Trump\nextreme_trump_wins <- results[results$Trump_Support > results$Harris_Support, ]\nextreme_trump_wins <- extreme_trump_wins[order(-extreme_trump_wins$Trump_Support), ]\ncat(\"Most extreme polls won by Trump:\\n\")\nprint(head(extreme_trump_wins, 5))", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "This is example of Monte Carlo simulation of elections assuming correct number of electors for each state and probabilities of winning the state by one of the candidates. ", "PreExCode": "", "Query": "# Parameters\r\nharris_support <- 0.49\r\ntrump_support <- 0.46\r\nundecided <- 0.05\r\nsample_size <- 1000\r\nsimulations <- 100000\r\n\r\n# Simulate elections\r\nset.seed(123) # For reproducibility\r\nharris_wins <- 0\r\ntrump_wins <- 0\r\n\r\nfor (i in 1:simulations) {\r\n votes <- sample(c('Harris', 'Trump', 'Undecided'), size = sample_size, replace = TRUE, \r\n prob = c(harris_support, trump_support, undecided))\r\n \r\n # Count the votes for each candidate\r\n harris_votes <- sum(votes == 'Harris')\r\n trump_votes <- sum(votes == 'Trump')\r\n \r\n # Determine winner\r\n if (trump_votes > harris_votes) {\r\n trump_wins <- trump_wins + 1\r\n } else {\r\n harris_wins <- harris_wins + 1\r\n }\r\n}\r\n\r\n# Calculate probabilities\r\ntrump_win_probability <- trump_wins / simulations\r\nharris_win_probability <- harris_wins / simulations\r\n\r\n# Print results\r\ncat(\"Trump win probability:\", trump_win_probability, \"\\n\")\r\ncat(\"Harris win probability:\", harris_win_probability, \"\\n\")\r\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "Finally, here we use binomnial distribution to compute how likely the survey result it assuming that we know the real final result of a candidate. ", "PreExCode": "", "Query": "Elections <- function(v, N) {\n # Define the states and their corresponding electoral votes\n states <- c(\n \"Alabama\", \"Alaska\", \"Arizona\", \"Arkansas\", \"California\", \"Colorado\", \"Connecticut\",\n \"Delaware\", \"Florida\", \"Georgia\", \"Hawaii\", \"Idaho\", \"Illinois\", \"Indiana\", \"Iowa\",\n \"Kansas\", \"Kentucky\", \"Louisiana\", \"Maine\", \"Maryland\", \"Massachusetts\", \"Michigan\",\n \"Minnesota\", \"Mississippi\", \"Missouri\", \"Montana\", \"Nebraska\", \"Nevada\", \"New Hampshire\",\n \"New Jersey\", \"New Mexico\", \"New York\", \"North Carolina\", \"North Dakota\", \"Ohio\",\n \"Oklahoma\", \"Oregon\", \"Pennsylvania\", \"Rhode Island\", \"South Carolina\", \"South Dakota\",\n \"Tennessee\", \"Texas\", \"Utah\", \"Vermont\", \"Virginia\", \"Washington\", \"West Virginia\",\n \"Wisconsin\", \"Wyoming\"\n )\n \n electoral_votes <- c(\n 9, 3, 11, 6, 55, 9, 7, 3, 29, 16, 4, 4, 20, 11, 6, 6, 8, 8, 4, 10, 11, 15, 10, 6, 10, 3,\n 5, 6, 4, 14, 5, 29, 15, 3, 18, 7, 7, 20, 4, 9, 3, 11, 40, 6, 3, 13, 12, 4, 10, 3\n )\n \n # Initialize counters\n trump_wins <- 0\n \n for (i in 1:N) {\n trump_votes <- 0\n kamala_votes <- 0\n \n # Simulate the election for each state\n for (j in 1:length(states)) {\n if (runif(1) < v[j]) {\n trump_votes <- trump_votes + electoral_votes[j]\n } else {\n kamala_votes <- kamala_votes + electoral_votes[j]\n }\n }\n \n # Determine the winner for this simulation\n if (trump_votes > kamala_votes) {\n trump_wins <- trump_wins + 1\n }\n }\n \n # Calculate the percentage of runs that Trump wins\n percentage_trump_wins <- (trump_wins / N) * 100\n \n return(percentage_trump_wins)\n}\n\n# Example usage:\nv <- rep(1, 50)\nN <- 10000 \nv[3]<-0.53 #Arizona\nv[5]<-0 #California\nv[6]<-0 #Colorado\nv[7]<-0 #Connecticut\nv[8]<-0 #Delware\nv[9]<-1 #Florida\nv[10]<-0.53 #Georgia\nv[11]<-0 #Hawai\nv[13]<-0 #Illionois\nv[19]<-0 #Maine\nv[20]<-0 #Maryland\nv[21]<-0 #Mass\nv[22]<-0.53 #Michigan\nv[23]<-0 #Minnesota\nv[28]<-0.5 #Nevada\nv[29]<-0 #NH\nv[30]<-0 #New jersey\nv[31]<-0 #New Mexico\nv[32]<-0 #New York\nv[33]<-0.52 #North Carolina\nv[37]<-0 #Oregon\nv[38]<-0.54 #Penna\nv[39]<-0 #RI\nv[40]<-1 # South Caroline\nv[43]<-1 #Texas\nv[45]<-0 #Vermont\nv[46]<-0 #Virginia\nv[47]<-0 #Washington\nv[49]<-0.52 #Wisconsin\n\nresult <- Elections(v, N)\nprint(result)\nstates <- c(\n \"Alabama\", \"Alaska\", \"Arizona\", \"Arkansas\", \"California\", \"Colorado\", \"Connecticut\",\n \"Delaware\", \"Florida\", \"Georgia\", \"Hawaii\", \"Idaho\", \"Illinois\", \"Indiana\", \"Iowa\",\n \"Kansas\", \"Kentucky\", \"Louisiana\", \"Maine\", \"Maryland\", \"Massachusetts\", \"Michigan\",\n \"Minnesota\", \"Mississippi\", \"Missouri\", \"Montana\", \"Nebraska\", \"Nevada\", \"New Hampshire\",\n \"New Jersey\", \"New Mexico\", \"New York\", \"North Carolina\", \"North Dakota\", \"Ohio\",\n \"Oklahoma\", \"Oregon\", \"Pennsylvania\", \"Rhode Island\", \"South Carolina\", \"South Dakota\",\n \"Tennessee\", \"Texas\", \"Utah\", \"Vermont\", \"Virginia\", \"Washington\", \"West Virginia\",\n \"Wisconsin\", \"Wyoming\"\n)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "# Parameters\ntrue_trump_support <- 0.48 # Hypothetical Trump support in the population\nsample_size <- 1000 # Sample size for the survey\nthreshold <- 460 # Trump gets fewer than 460 votes in the sample (46% support)\n\n# Calculate the probability that Trump gets fewer than 460 votes in the sample of 1000\nprob_trump_loses_sample_3pt <- pbinom(threshold, size = sample_size, prob = true_trump_support)\n\n# Print the result\nprob_trump_loses_sample_3pt\n\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "10", "snippets": {}}, "11": {"Name": "Hypothesis Testing", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"11.01": {"Name": "Preidciton Challenge 4", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "11", "snippets": {}}, "11.1": {"Name": "Introduction ", "Details": "

Randomness is the biggest enemy of data scientists. How to distinguish what\u2019s real from what\u2019s random? This is the goal of hypothesis testing. We will introduce hypothesis testing through the permutation test. To illustrate the permutation test, let us start with a simple example of a dataset storing information about traffic in Lincoln and Holland tunnels.

\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n

Table 9.1: Snippet of Traffic Dataset
	TUNNEL	DAY	VOLUME_PER_MINUTE
1844	Lincoln	weekday	81.0
1050	Holland	weekend	57.5
1630	Lincoln	weekday	75.0
718	Holland	weekday	61.5
1455	Lincoln	weekday	62.0
2635	Lincoln	weekend	72.0
720	Holland	weekday	53.5
948	Holland	weekday	91.5
2065	Lincoln	weekday	69.0
288	Holland	weekday	65.5

\nWe observe that Lincoln traffic is higher than Holland tunnel traffic by calculating average traffic volume per minute for each of the tunnels using the provided data.\n\nWe conclude with 68.54 mean volume per minute for Lincoln and 67.71 mean volume per minute for the Holland tunnel. This seems to indicate that Lincoln traffic is higher than Holland traffic. But is it? Or is it just random deviation? Perhaps if we took more measurements, the trend would be reversed? This is where the permutation test comes handy. First, let us talk about the null hypothesis and the alternative hypothesis.\n\nNull hypothesis for the Lincoln-Holland tunnel observation is that, not surprisingly, there is no difference in traffic between the two tunnels.\n\nThe alternative hypothesis states that Lincoln tunnel is more busy than Holland tunnel.\n\nDoes observed data (observed traffic difference) provide us with enough evidence to reject null hypothesis and in fact support the alternative hypothesis? To answer this question we need to decide whether the observed result is reasonably likely to come up randomly under the condition that NULL hypothesis holds.\n\nHow likely it is that observed difference (D=0.83) comes randomly? Permutation test helps us to estimate the chance that D=0.83 will come up randomly under the condition that traffic in Holland and Lincoln tunnels is equal.\n\nIn each permutation of the permutation test we randomly scramble the traffic table once. Permutation test is run many times, typically 10,000, even 100,000 times, and each permutation simulates a random process by simply reassigning the traffic volume values randomly between tunnels. The numbers of traffic measurements in Holland and Lincoln tunnels respectively remain the same. Existing values are scrambled though - breaking any relationship between volume numbers and tunnel names. Each permutation is like rolling a dice. How often will this random process produce the result which is at least as extreme as D=0.83 that we have observed? The less often it happens the more likely it is that what we have observed is NOT random. For example, if we can get our observed result only 3 times in 1000 rolls of a dice (permutation test) it means that with probability of 99.7% our observed result cannot be random.\n\nPermutation test provides a palpable experience of randomness. Just roll the dice many times and see how often you can get the observed result or more. If you can get D>0.83 relatively often (above what is called significance level usually it is at least 5% of the time), then you cannot reject the null hypothesis. In other words the conclusion that your observation appeared RANDOMLY. Otherwise, we can conclude that observation was not random - and we reject the null hypothesis.\n\nNow let's introduce the critical concept of p-value - the crucial measure in hypothesis testing which helps us to get fooled by randomness!\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"11.1.1": {"Name": "p-value", "Details": "A p-value, short for \"probability value\" or \"p-probability,\" is a statistical measure used in hypothesis testing to assess the strength of evidence against a null hypothesis. It helps researchers determine whether the results of an experiment or study are statistically significant or merely due to random chance.\n\nHere's how it works:\n\nNull Hypothesis (H0): This is the default or initial assumption that there is no significant effect or relationship between the variables being studied. It represents the status quo or a lack of effect.\n\nAlternative Hypothesis (Ha): This is the hypothesis that researchers want to test, suggesting that there is a significant effect or relationship between the variables.\n\nCollect Data: Researchers collect data through experiments, surveys, observations, or other research methods.\n\n\nP-Value Calculation: The p-value represents the probability of obtaining results as extreme as, or more extreme than, the observed results if the null hypothesis were true. In other words, it measures the likelihood of observing the data purely by chance.\n\nInterpretation:\n\nIf the p-value is small (typically less than a predetermined significance level, often denoted as \u03b1, e.g., 0.05), researchers reject the null hypothesis. This suggests that there is strong evidence that the effect or relationship being studied is not due to random chance.\nIf the p-value is large, researchers fail to reject the null hypothesis. This means that the data do not provide sufficient evidence to conclude that there is a significant effect or relationship.\nIt's important to note that a small p-value does not prove that the null hypothesis is false, nor does a large p-value prove that it is true. Instead, it provides a measure of the strength of evidence against the null hypothesis. Researchers should consider other factors, such as the practical significance of the effect and the study's design, when interpreting p-values.\n\nIt's also crucial to use p-values responsibly, as they can be influenced by factors like sample size and multiple testing, and they do not provide information about the magnitude or practical significance of an effect. Additionally, p-values are just one tool in statistical analysis, and they should be used in conjunction with other methods and considered within the broader context of scientific research.\n\nNotice, that every time we run the permutation test function we may get slightly different p-values. This is because permutations are random. The more times we run a permutation test, the closer it will approximate the \u201creal\u201d p-value. Snippet 9.2 shows permutation test results for the Traffic data set.\n\nAnother test which is often used for difference of means hypothesis testing is the z-test. It is described very well in the attached link to the Khan Academy lecture. Here we run z-test function in one of the following snippets.\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "11", "snippets": {}}}, "parent": "11", "snippets": {}}, "11.2": {"Name": "Permutation test ", "Details": "The following snippet 9.2 shows the code for hypothesis test of difference of means.\n\nIs the mean traffic (VOLUME_PER_MINUTE) in the Lincoln tunnel bigger than mean traffic (VOLUME_PER_MINUTE) in the Holland?\n\nWe will use here the permutation function from active textbook library with the following syntax. We start with our example\n\nPermutation(traffic, 'TUNNEL', 'VOLUME_PER_MINUTE',10000, 'Holland', 'Lincoln')\n\nHere traffic is the data frame, TUNNEL is a categorical variable, VOLUME_PER_MINUTE is a numerical variable (we will be taking the mean() of). 10000 is the number of times \nwe evaluate the permutaiton test and 'Holland 'and 'Lincoln' are two values of TUNNEL. This function will be testing hypothesis that mean volume_per_minute in Lincoln tunnel is higher than mean volume_per_minute for Holland tunnel. \n\nGeneral syntax of permutation function is as follows:\n\nPermutation(df, 'CAT', 'NUM',N, 'v1, 'v2'), where df is the name of data frame, CAT is a categocial variable, NUM is the numerical variable (we are interested in mean(NUM)), N is the number of permutations and v1 and v2 are two values of CAT variable. \n\n\n\n\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"11.2.1": {"Name": "\ud83c\udfa6 Permutation test on step at a time -Video \ud83c\udfa6 ", "Details": "In this snippet you can run permutation test one permutation at a time. This helps us to observe how data can vary with each permutation. ", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1EQuDbCtzXx_srrdwwAEWK8_YZvspN5Gj/preview", "child": {}, "parent": "11", "snippets": [{"Details": "", "PreExCode": "", "Query": "traffic<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Traffic2022.csv')\n\nranNum <- sample(1:nrow(traffic),nrow(traffic))\nranNum[1:5]\n\nVOLUME_PER_MINUTE<-traffic$VOLUME_PER_MINUTE[ranNum]\nTUNNEL<-traffic$TUNNEL\n\nPermuted_traffic<-data.frame(TUNNEL, VOLUME_PER_MINUTE)\n\nmean(traffic[traffic$TUNNEL=='Lincoln', ]$VOLUME_PER_MINUTE) -mean(traffic[traffic$TUNNEL=='Holland', ]$VOLUME_PER_MINUTE)\n\nmean(Permuted_traffic[Permuted_traffic$TUNNEL=='Lincoln', ]$VOLUME_PER_MINUTE)-mean(Permuted_traffic[Permuted_traffic$TUNNEL=='Holland', ]$VOLUME_PER_MINUTE)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "11", "snippets": [{"Details": "", "PreExCode": "library(dplyr)\noptions(warn=-1)\nPermutation<- function(df1,c1,c2,n,w1,w2){\n df <- as.data.frame(df1)\n D_null<-c()\n V1<-df[,c1]\n V2<-df[,c2]\n sub.value1 <- df[df[, c1] == w1, c2]\n sub.value2 <- df[df[, c1] == w2, c2]\n D <- mean(sub.value2, na.rm=TRUE) - mean(sub.value1, na.rm=TRUE)\n m=length(V1)\n l=length(V1[V1==w2])\n for(jj in 1:n){\n null <- rep(w1,length(V1))\n null[sample(m,l)] <- w2\n nf <- data.frame(Key=null, Value=V2)\n names(nf) <- c(\"Key\",\"Value\")\n w1_null <- nf[nf$Key == w1,2]\n w2_null <- nf[nf$Key == w2,2]\n D_null <- c(D_null,mean(w2_null, na.rm=TRUE) - mean(w1_null, na.rm=TRUE))\n }\n myhist<-hist(D_null, prob=TRUE)\n multiplier <- myhist$counts / myhist$density\n mydensity <- density(D_null, adjust=2)\n mydensity$y <- mydensity$y * multiplier[1]\n plot(myhist)\n lines(mydensity, col='blue')\n abline(v=D, col='red')\n M<-mean(D_null>D)\n return(M)\n}", "Query": "traffic<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Traffic2022.csv\")\np<-Permutation(traffic, 'TUNNEL', 'VOLUME_PER_MINUTE',10000, 'Holland', 'Lincoln')\ncat(\"The p-value of the hypothesis is equal to:\",p)\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "11.3": {"Name": " z-test ", "Details": "A Z-test is a statistical hypothesis test used to determine whether there is a significant difference between the means of two groups when the population standard deviation is known. It is commonly used when you want to compare a sample mean to a known population mean or compare the means of two independent samples.\n\nThe Z-test is often used in situations where you have a large enough sample size, and you know the population standard deviation. If the population standard deviation is unknown or the sample size is small, a t-test may be more appropriate. The Z-test assumes that the data follows a normal distribution. \nIt's worth noting that in practice, many statistical analyses are performed using software or calculators that handle these calculations automatically, making it easier to conduct hypothesis tests like the Z-test.\n\nThis is what we do here - we use a function to compute the p-value using z test - without having you to calculate z-statistic. The function does it for you. It is important to point out that the p-value returned by z test function is always the same no matter how many times you execute z test function. As we pointed out it is NOT the case for permutation test where p-values are different every time. Which one is correct? The true answer is - none of them is exact, both return estimates, but estimates which are good enough to decide in the underlying hypothesis test. \n\n\n\nNull Hypothesis - Traffic in Holland tunnel is the same as traffic in Lincoln tunnel.\n\nAlternative Hypothesis - Traffic in the LincolnTunnel is larger than traffic in the Holland tunnel.\n\nIn the snippet 9.3 we end up calculating the p-value which leads to rejection of Null hypothesis (good news for data scientist, bad for the sceptic). Indeed, p-value is less than the significance level of 5%. This means, that under null hypothesis it is extremely unlikely (less than 5% chance) to see the result which is at least as big as the observed difference of means. \n\nWe are using Active Textbook library function \n\nz_test(TRAFFIC,\"TUNNEL\", \"VOLUME_PER_MINUTE\",\"Holland\", \"Lincoln\")\n\nIn the syntax similar to permutation function, except that we do not need to specify N - the number of permutations. The body of z_test function calculates standard deviations and means as well as uses approriate tables to translate z-value to p-value. \n\nGeneral form of z_test function is as follows:\nz_test(df,\"CAT, \"NUM\",\"v1\", \"v2\")\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"11.3.1": {"Name": "z-test from aggregates", "Details": "Now we will show the \"word problem\" for hypothesis testing when input data is not raw data frame, but 4 parameters, two means and two standard deviations. Here is the word problem:\n\nLets assume we have 500 data points for parties with 'HipHop' music played and 300 data points with 'Techno' music played. Lets assume that\n\nmean_attendace(HipHop) = 70\nmean_attendace(Techno) = 68\nsd(HipHop) = 20\nsd(Techno) = 20\n \nz_test_from_agg(m1,m2, sd1,sd2. N1,N2) where N1 and N2 are numbers of samples from the two groups. \n\nWe assume that m1", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "11", "snippets": [{"Details": "", "PreExCode": "library(dplyr)\noptions(warn=-1)\nPermutation<- function(df1,c1,c2,n,w1,w2){\n df <- as.data.frame(df1)\n D_null<-c()\n V1<-df[,c1]\n V2<-df[,c2]\n sub.value1 <- df[df[, c1] == w1, c2]\n sub.value2 <- df[df[, c1] == w2, c2]\n D <- mean(sub.value2, na.rm=TRUE) - mean(sub.value1, na.rm=TRUE)\n m=length(V1)\n l=length(V1[V1==w2])\n for(jj in 1:n){\n null <- rep(w1,length(V1))\n null[sample(m,l)] <- w2\n nf <- data.frame(Key=null, Value=V2)\n names(nf) <- c(\"Key\",\"Value\")\n w1_null <- nf[nf$Key == w1,2]\n w2_null <- nf[nf$Key == w2,2]\n D_null <- c(D_null,mean(w2_null, na.rm=TRUE) - mean(w1_null, na.rm=TRUE))\n }\n myhist<-hist(D_null, prob=TRUE)\n multiplier <- myhist$counts / myhist$density\n mydensity <- density(D_null, adjust=2)\n mydensity$y <- mydensity$y * multiplier[1]\n plot(myhist)\n lines(mydensity, col='blue')\n abline(v=D, col='red')\n M<-mean(D_null>D)\n return(M)\n}", "Query": "Val1<-rnorm(10,mean=25, sd=10)\nVal2<-rnorm(10,mean=30, sd=10)\n \nCat1<-rep(\"GroupA\",10) # for example GroupA can be Holland Tunnel\nCat2<-rep(\"GroupB\",10) # for example Group B will be Lincoln Tunnel\n\n\n\n#The rep command will repeat the labels \"Group A\" or \"Group B\". These labels will be then permuted by permutation test\n\nCat<-c(Cat1,Cat2) # A variable with first 10 values GroupA and next 10 values GroupB\n\nVal<-c(Val1,Val2)\n\nd<-data.frame(Cat,Val)\nd\n\np<-Permutation(d, \"Cat\", \"Val\",1000,\"GroupA\", \"GroupB\")\ncat(\"The p-value of the hypothesis is equal to:\",p)\nObserved_Difference<-mean(d[d$Cat=='GroupB',2])-mean(d[d$Cat=='GroupA',2])\ncat(\"The observed difference is equal to:\",Observed_Difference)\n\n#This will calculate the mean of the second column (having 10 random values for each group), and the mean of groupB values is subtracted from the mean of groupA values, which will give you the value of the difference of the mean.\n \n #Try changing mean and sd values. When you run this you will see that the difference is sometimes negative #or sometimes positive.", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "11.4.2": {"Name": "z-test", "Details": "We will now apply z-test for our synthetic example of two distributions. ", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "11", "snippets": [{"Details": "", "PreExCode": "library(dplyr)\r\noptions(warn=-1)\r\nz_test<- function(data,col1,col2,sub1,sub2) {\r\n data <- as.data.frame(data)\r\n V1<-data[,col1]\r\n V2<-data[,col2]\r\n #data clean and subset, either\r\n X <- subset(data, V1 == sub2)\r\n Y <- subset(data, V1 == sub1)\r\n x <- X[,col2]\r\n y <- Y[,col2]\r\n #z score\r\n zeta<-(mean(x)-mean(y))/(sqrt(sd(x)^2/length(x) +sd(y)^2/length(y)))\r\n print(paste(zeta,\" is the z-value\"))\r\n #plot red line\r\n r <- max(zeta+0.5, 5)\r\n plot(x=seq(from = -r, to= r, by=0.1),y=dnorm(seq(from = -r, to= r, by=0.1),mean=0),type='l',xlab = 'mean difference', ylab='possibility')\r\n abline(v=zeta, col='red')\r\n #get p\r\n p = 1-pnorm(zeta)\r\n print(paste(p, \" is the p-value\"))\r\n return(p)\r\n}\r", "Query": "Val1<-rnorm(10,mean=25, sd=10)\nVal2<-rnorm(10,mean=35, sd=10)\nCat1<-rep(\"GroupA\",10) \nCat2<-rep(\"GroupB\",10) \nCat<-c(Cat1,Cat2) \nVal<-c(Val1,Val2)\n\nd<-data.frame(Cat,Val)\nObserved_Difference<-mean(d[d$Cat=='GroupB',2])-mean(d[d$Cat=='GroupA',2])\nObserved_Difference\n\nz_test(d,\"Cat\", \"Val\",\"GroupA\", \"GroupB\")", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "11", "snippets": {}}, "11.5": {"Name": "\ud83c\udfa6 Hypothesis testing - questions-Video \ud83c\udfa6", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1-NmRn-mHRIbbXXeO3ILEkhrnFPkQCt7K/preview", "child": {}, "parent": "11", "snippets": {}}}, "parent": "11", "snippets": {}}, "12": {"Name": "Multiple Hypotheses Correction ", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"12.0": {"Name": "\ud83c\udfa6 Multiple Hypothesis - associations \ud83c\udfa6", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1zPOpJQRIsd6HXWTMPKUeZ-9KCAxqh13D/preview", "child": {}, "parent": "12", "snippets": {}}, "12.1": {"Name": "Introduction ", "Details": "We often consider multiple possible hypotheses in our search for discovery to find one with lowest possible p-value. Consciously or subconsciously we are engaging, what is often called, p-value hunting. We have to be very careful! We may \u201cdiscover\u201d what is simply random even if we correctly calculate p-value and compare it with the significance level. It is very important to learn about multiple hypothesis traps very early in the process of learning data science.\n\nFor example assume that we are looking for associations between sales of individual items in a supermarket. Does bread sell with butter? Does coffee sell with spring water? There is an exponential number of possible combinations (N choose 2 to be exact, where N is the number of items). For each such pair we perform hypothesis testing. If one test is performed at the 5% significance level and the corresponding null hypothesis is true, there is only a 5% chance of incorrectly rejecting the null hypothesis. However, if 100 tests are each conducted at the 5% significance level and all corresponding null hypotheses are true, the expected number of incorrect rejections (also known as false positives or Type I errors) is 5. If the tests are statistically independent from each other, the probability of at least one incorrect rejection is approximately 99.4%. Thus, we will almost surely find one false positive! In other words we will be fooled by data.\n\nBonferroni correction is a method to counteract the multiple hypothesis (often called multiple comparison problem. Make it harder to reject null hypotheses by dividing the significance level by number of hypotheses. The Bonferroni correction compensates for that increase by testing each individual hypothesis at a significance level of \u03b1 / m where m is the number of hypotheses. For example, if a trial is testing me = 20 hypotheses with a desired \u03b1 = 0.05, then the Bonferroni correction would test each individual hypothesis at\n\nThus, there is a very simple remedy for multiple hypothesis traps. Just divide the significance level by the number of (potential) hypotheses tested. This will make it harder, often much much harder to reject the null hypothesis and yell Eureka! Critics say that in fact Bonferroni correction is too conservative and too \u201cpro-null\u201d and tough on alternative hypotheses to be acceptable.\n\nThe unwanted side effect of Bonferroni correction is that we may fail to reject the null hypothesis too often. Bonferroni correction makes discovery sometimes too hard, making data scientists too conservative and accepting null hypothesis when they should be rejected. It may also be the case that even Bonferroni correction will not protect us, as we will show in our example below. But at least we will be much less likely to make fools of ourselves coming with false discoveries leading potentially to very wrong business decisions.\n\nThere are other less conservative methods of correcting for multiple hypotheses - such as the Benjamini-Hochberg method described in the attached slides.\n\nWe illustrate the p-value hunt in snippet 10.1 below. It is based on synthetic data set showing summer temperatures in New Jersey townships. Table below describes the data set based on hypothetical temperature readings in various municipalities of New Jersey over summer. Is one city experiencing higher average temperatures than another? Can we find such a pair of cities? This is the ultimate p-value hunt. Let\u2019s compare townshiships pair by pair, until we find a pair with sufficiently large differences of mean temperatures and sufficiently low p-value. Careful! You may come up with false discovery if you do not correct for multiple hypotheses!\n\n

\n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n\n\n\n \n \n \n \n \n\n\n\n \n \n \n \n \n\n\n\n \n \n \n \n \n\n\n \n \n

Table 10.1: Snippet of Summer Temperature Dataset
	Township	Temperature
887	Holmdel	71
374	North Brunswick	86
1513	Lawrencville	94
791	Asbury Park	96
1265	Highland Park	82

\nThe snippet below shows several permutation tests for different pairs of townships and difference of means of temperatures hypothesis test. Three of five pairs show p\u2013values less than customary significance level of 5%. Should we then reject the null hypothesis and conclude that indeed Ocean Grove is warmer than New Brunswick or that New Brunswick is warmer than Holmdel or that Red Bank is warmer than Holmdel? Indeed, the three pairs result in p-values significantly lower than 5%. If we incorrectly disregard the number of hypotheses considered, we may come to wrong conclusions supporting these alternative hypotheses. But there are around 20 townships in the Temp data set. Thus there are around 200 possible hypotheses (200 pairs of townships) which we may consider in our p-value hunt. If we apply Bonferroni correction for N=200, the significance level will be 200 times lower, instead of 5%, it will be 0.025%. Two of the three hypotheses (Ocean Grove vs New Brunswick and New Brunswick vs Holmdel) meets the new significance level. Indeed in both cases p-values are significantly larger than 0.025%. Thus, for four pairs out of five pairs we can reject null hypotheses.\n\nNow we can disclose that we have created our Temp data set completely randomly - assigning random temperatures between 50 and 100 degrees to each township. Thus, without Bonferroni coefficient we would be fooled by data, not once, but twice in our four tests. We would find a trend when it does not exist - it is simply random deviation. The Temp data set assigned random temperatures between 50 and 100 degrees to around 20 townships in New Jersey.\n\nIt turns out however that even Bonferroni correction is not sufficient to protect us against incorrectly rejecting null hypothesis. Indeed, for Red Bank and Holmdel, we conclude that Red Band is warmer than Holmdel with p-value ~ 0! (see the last permutation test in the snippet). This p-value falls even below significance level adjusted with Bonferroni correction (0.025%). It only shows that dealing with multiple hypotheses is a risky adventure. We may end up being fooled by data even when we apply Bonferroni correction. But at least we are less likely to fall into the trap of multiple hypotheses when we apply Bonferroni Correction.\n\nWithout Bonferroni correction we would have to incorrectly reject the null hypothesis in the last three permutation tests (3rd, 4th and 5th) tests. We will still reject null hypothesis (and have false positive discovery) in the last, 5th case. Indeed, the 5th p-value is less than 0.00025 which is the significance level after Bonferroni correction (0.025%).\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "12", "snippets": {}}, "12.2": {"Name": "Hypothesis Hunting", "Details": "Lets assume that the task is to find \"strong\" hypotheses in a data set. How do we \"hunt\" for hypotheses?\nWe can use plots but we can also use tapply() as indicated here. After running tapply 4 times here we can find anything we looking for in the HW4 - just the data set is different.\n\nFor strong alternative hypotheses look at tapply() where mean values differ a lot. We can even try subsets - take a look at the snippets for the movie set below. However, always remeber that you have to correct for the number of hypotheses tested! Hunting is not for free. The more hypotheses we test, the stricter is the siginifcance level. Remember about Bonferoni coeifficent!", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "12", "snippets": [{"Details": "In the upcoming section, we will explore the concept of p-value hunting using two different datasets: the Airbnb dataset and the movie-IMDB dataset. In both cases, our goal is to discover the most compelling evidence against the null hypotheses regarding the mean prices (for the Airbnb data) and the mean IMDb scores (for the movie dataset). By utilizing the tapply() function on various subsets of the datasets, we can identify the most robust hypotheses with the lowest p-values. However, it is crucial to be cautious of the risks involved in this pursuit and account for multiple hypotheses by assessing the number of tests conducted.", "PreExCode": "library(dplyr)\r\noptions(warn=-1)\r\nPermutation<- function(df1,c1,c2,n,w1,w2){\r\n df <- as.data.frame(df1)\r\n D_null<-c()\r\n V1<-df[,c1]\r\n V2<-df[,c2]\r\n sub.value1 <- df[df[, c1] == w1, c2]\r\n sub.value2 <- df[df[, c1] == w2, c2]\r\n D <- mean(sub.value2, na.rm=TRUE) - mean(sub.value1, na.rm=TRUE)\r\n m=length(V1)\r\n l=length(V1[V1==w2])\r\n for(jj in 1:n){\r\n null <- rep(w1,length(V1))\r\n null[sample(m,l)] <- w2\r\n nf <- data.frame(Key=null, Value=V2)\r\n names(nf) <- c(\"Key\",\"Value\")\r\n w1_null <- nf[nf$Key == w1,2]\r\n w2_null <- nf[nf$Key == w2,2]\r\n D_null <- c(D_null,mean(w2_null, na.rm=TRUE) - mean(w1_null, na.rm=TRUE))\r\n }\r\n myhist<-hist(D_null, prob=TRUE)\r\n multiplier <- myhist$counts / myhist$density\r\n mydensity <- density(D_null, adjust=2)\r\n mydensity$y <- mydensity$y * multiplier[1]\r\n plot(myhist)\r\n lines(mydensity, col='blue')\r\n abline(v=D, col='red')\r\n M<-mean(D_null>D)\r\n return(M)\r\n}", "Query": "Temp <-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Tempratures.csv\") #web load\n\nPermutation(Temp, \"Township\", \"Temprature\",1000, \"Trenton\", \"Princeton\")\nPermutation(Temp, \"Township\", \"Temprature\",1000, \"Newark\", \"Passaic\")\nPermutation(Temp, \"Township\", \"Temprature\",1000, \"New Brunswick\", \"Ocean Grove\")\nPermutation(Temp, \"Township\", \"Temprature\",1000, \"Holmdel\", \"New Brunswick\")\nPermutation(Temp, \"Township\", \"Temprature\",1000, \"Holmdel\", \"Red Bank\")", "Result": "", "Title": "Multiple Permutation Tests", "Type": "R", "Video": "", "Width": ""}, {"Details": "Above snippet: Price variation across room types or neighborhood groups? Any interesing hypotheses there? Just run the code and see.", "PreExCode": "", "Query": "airbnb<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/airbnb.csv') \ntapply(airbnb$price, airbnb$room_type, mean)\ntapply(airbnb$price, airbnb$neighbourhood_group, mean)", "Result": "", "Title": "Hypothesis hunting", "Type": "R", "Video": "", "Width": ""}, {"Details": "Above snippet:Price variation across neighborhoods?", "PreExCode": "", "Query": "airbnb<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/airbnb.csv') \nresult<-tapply(airbnb$price, airbnb$neighbourhood, mean)\nv<-as.data.frame(result)\ncolnames(v)[1]<-'Neighborhood'\nv", "Result": "", "Title": "Hypothesis hunting for many-valued attribute", "Type": "R", "Video": "", "Width": ""}, {"Details": "Above snippet: How about looking at mean price variation for different floors?", "PreExCode": "", "Query": "airbnb<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/airbnb.csv') \nresult<-tapply(airbnb$price, airbnb$floor, mean)\nv<-as.data.frame(result)\ncolnames(v)[1]<-'Floor'\nv", "Result": "", "Title": "Hypothesis huntin for many-valued attribute", "Type": "R", "Video": "", "Width": ""}, {"Details": "Above snippet:Any variation of mean imdb score across genres?", "PreExCode": "", "Query": "movies<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Movies2022F-4-new.csv')\ntapply(movies$imdb_score, movies$genre, mean)", "Result": "", "Title": "Hypothesis hunting for movies data set", "Type": "R", "Video": "", "Width": ""}, {"Details": "Above snippet:Any variation of mean imdb score across content ratings?", "PreExCode": "", "Query": "movies<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Movies2022F-4-new.csv')\ntapply(movies$imdb_score, movies$content, mean)", "Result": "", "Title": "Hypothesis hunting for movies data set", "Type": "R", "Video": "", "Width": ""}, {"Details": "Above snippet:Any variation of mean imdb score across countries?", "PreExCode": "", "Query": "movies<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Movies2022F-4-new.csv')\nresult<-tapply(movies$imdb_score, movies$country, mean)\nv<-as.data.frame(result)\ncolnames(v)[1]<-'Country'\nv", "Result": "", "Title": "Hypothesis hunting for movies data set", "Type": "R", "Video": "", "Width": ""}, {"Details": "Above snippet:This is example of deeper hunt for difference of mean of imdb score. Here we look at UK comedies and see if there is any variation accross ratings", "PreExCode": "", "Query": "movies<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Movies2022F-4-new.csv')\nmovies <- movies[movies$country=='UK' & movies$genre=='Comedy',]\ntapply(movies$imdb_score, movies$content, mean)", "Result": "", "Title": "Deeper Hypothesis hunting for movies data set", "Type": "R", "Video": "", "Width": ""}, {"Details": "Above snippet:This is example of deeper hunt for difference of mean of imdb score. Here we look at US movies which are rated PG and see if there is any variation accross genres", "PreExCode": "", "Query": "movies<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Movies2022F-4-new.csv')\nmovies <- movies[movies$content=='PG' & movies$country=='USA',]\ntapply(movies$imdb_score, movies$genre, mean)", "Result": "", "Title": "Deeper Hypothesis hunting for movies data set", "Type": "R", "Video": "", "Width": ""}]}, "12.3": {"Name": "Benjamini-Hochberg method to control FDR", "Details": "\n Chat about Benjamini-Hochberg method\n $\"\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "12", "snippets": [{"Details": "", "PreExCode": "", "Query": "# Given data\r\npvals <- c(0.003, 0.007, 0.015, 0.029, 0.041, 0.076)\r\nm <- length(pvals)\r\nalpha <- 0.05\r\n\r\n# Bonferroni threshold\r\nbonf_threshold <- alpha / m\r\nbonf_reject <- pvals <= bonf_threshold\r\n\r\n# BH thresholds\r\nranks <- 1:m\r\nbh_thresholds <- (ranks / m) * alpha\r\nbh_reject <- pvals <= bh_thresholds\r\n\r\n# BH rule: reject all p-values up to the largest i such that p <= bh_threshold\r\nmax_bh_i <- max(which(bh_reject), na.rm = TRUE)\r\nbh_final_reject <- rep(FALSE, m)\r\nif (!is.infinite(max_bh_i)) {\r\n bh_final_reject[1:max_bh_i] <- TRUE\r\n}\r\n\r\n# Combine into a data frame\r\nresult <- data.frame(\r\n Rank = ranks,\r\n p_value = pvals,\r\n Bonferroni_Reject = bonf_reject,\r\n BH_Threshold = round(bh_thresholds, 4),\r\n BH_Reject = bh_final_reject\r\n)\r\n\r\nprint(result)\r\n", "Result": "", "Title": "Benjamini-Hochberg", "Type": "R", "Video": "", "Width": ""}]}, "12.4": {"Name": "Quinlan 2025 Spring", "Details": "Discussion of Prediction Challenges ", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "12", "snippets": {}}}, "parent": "12", "snippets": {}}, "13": {"Name": "\ud83c\udfa6Test of Independence \ud83c\udfa6", "Details": "

We would like to test if a student\u2019s final grade in Professor Moody\u2019s class depends on the student\u2019s major. The null hypothesis in this case is the hypothesis of independence. Independence means that the distribution of final grades is the same no matter what the major is. The alternative hypothesis is that the distribution of final grades changes from major to major, rejecting the null hypothesis of independence.

Notice that we do not specify how the grades depend on students\u2019 majors. Do CS students get better grades than Psychology majors? Do Economics majors get lower grades than Statistics majors? We do not care about this. We are only testing here whether there is a relationship between Major and the final grade distribution.

We will describe the permutation test which scrambles our data randomly in such a way that any relationship between grades and major (if it ever existed) is broken. We will run a permutation test a large number of times - possibly tens of thousands of times. Then we will determine how likely it is to randomly obtain the observed result. But what is the observed result? It is a bit more complex than the difference of means which we were observing in hypothesis testing.

The observed result in our case is calculated by so called chi-square statistic which is calculated on the contingency table, table(moody$Grade, moody$Major)

To explain it, let us first define two tables: The observed contingency table and the expected contingency table. It is calculated by table() function.

OBSERVED CONTINGENCY TABLE

\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n

Grade/Major	CS	Economics	Psychology	Statistics
A	46	54	69	42
B	46	12	2	35
C	51	33	30	34
D	41	37	29	34
F	108	99	99	99

The second table we need is called the expected table. This is the hypothetical table of the relationship between grade and major, assuming grade and major are completely independent. Such a table would be the result of the same distribution of grades for each of the majors. Notice that we have 1000 students in the data set the expected table (i.e. table which have grades completely independent from majors) would have the same distribution of grades for each major that over all students - which is shown by the TOTAL column.

EXPECTED CONTINGENCY TABLE

\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n

Grade/Major	CS	Economics	Psychology	Statistics
A	61	50	48	51
B	28	22	22	23
C	43	35	34	36
D	41	33	32	34
F	118	96	93	99
TOTAL	291	236	229	244	1000

We kept fractions - although these are number of students - therefore would have to be rounded up to integers

The chi-square formula calculates the \u201cdistance\u201d between the observed contingency table and the expected contingency table.

$\"\"$

For the two tables above,

$\"\"$

To evaluate how far off is this observed result assuming that Grades are independent from Major, we run a permutation test which scrambles Grades and Majors randomly and every time computes the chi-square formula with the new observed table (the expected table is always the same). Then, we see how many times out of, say 10,000 iterations of permutation test we obtain a result larger than the observed result of 60.03? This is the p-value.

Permutation test for independence hypothesis gives us again a better feeling about the impact of randomness and whether the observed chi-square result for \u201csimilarity\u201d of grade distributions for different majors can be obtained randomly. In the following snippet we run the chi-square test which is based on the so-called chi square distribution. Here we simply show you a function which can calculate p-value, just like the z-test function does. The explanation of the chi-square test is provided in attached link to the excellent Khan Academy video.

Permutation tests in both cases of difference of means and independence hypotheses give a better intuitive sense of how we answer the question - can the observed result be obtained randomly?

Notice that the independence test is looking globally at two vectors and whether one depends on another. If we wanted to be more specific and know if psychology majors are more likely to get an A than CS majors, we can frame this as a difference of means hypothesis. Testing this hypothesis will be using the difference of means of frequencies of A\u2019s among CS majors and psychology majors. This could be done again by permutation test in section 9 or the z-test.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1IEDyLk5qKG27G2f2q4P0YsuNXWuw9c4E/preview", "child": {"13.1": {"Name": "Reporting Odds Ratio (ROR)", "Details": "ROR calculation\r\n\r\nThe ROR for a given Adverse Event Y and Drug X depends on four values:\r\n\r\na = the number of primary case reports involving Drug X that do report Adverse Event Y\r\nb = the number of primary case reports involving Drug X that do NOT report Adverse Event Y\r\nc = the number of primary case reports that do NOT involve Drug X but do report Adverse Event Y\r\nd = the number of primary case reports that do NOT involve Drug X and do NOT report Adverse Event Y", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "13", "snippets": [{"Details": "", "PreExCode": "", "Query": "#See the contingency table in the slides for section 11 - showing adverse effects x drug\na <- 5 # Replace with your value for 'a'\nb <- 100 # Replace with your value for 'b'\nc <- 200 # Replace with your value for 'c'\nd <- 9000 # Replace with your value for 'd'\nROR <- (a/b) / (c/d)\ncat('The value of ROR is')\nROR\neln_ROR <- log(ROR) # Natural logarithm of ROR\ncat('The logarithm of ROR is')\neln_ROR\n# Calculate the lower and upper bounds\nlower_CI <- exp(eln_ROR - 1.96 * sqrt(1/a + 1/b + 1/c + 1/d))\nupper_CI <- exp(eln_ROR + 1.96 * sqrt(1/a + 1/b + 1/c + 1/d))\ncat('Lower bound of 95 percent interval')\nlower_CI\ncat('Upper bound of 95 percent interval')\nupper_CI\nObserved<-matrix(c(a,b,c,d), nrow=2, ncol=2)\nObserved\nchisq.test(Observed)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "13", "snippets": [{"Details": "", "PreExCode": "", "Query": "Expected <-matrix(c(160,480,160, 40,120,40), nrow=3, ncol=2)\nObserved<-matrix(c(180,420,200,20,180,0), nrow=3, ncol=2)\nExpected\nObserved\nchisq.test(Observed)", "Result": "", "Title": "Chisq test", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "movies<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/Movies2022F-4.csv\")\ndata<-table(movies$content, movies$genre)\nchisq.test(data)", "Result": "", "Title": "Contingency table", "Type": "R", "Video": "", "Width": ""}]}, "16": {"Name": "\ud83c\udfa6 Bayesian Reasoning \ud83c\udfa6", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "16", "snippets": [{"Details": "To explain snippet above we asked GPT - passing to it this snippet as attached file: \n\n\n What does this code do?\n $\"ChatGPT$ \n\n\n\n", "PreExCode": "", "Query": "#Belief = \"Have Covid\"\n#Observation = Covid Test Result\n#How much the probability of having covid increases upon positive COVID-test?\n#We use the odds formulation of Bayesian Theorem\n# we begin with prior odds of having Covid: P(Covid)/(1-P(Covid)\nPriorHaveCovid<-0.01\nPriorCovidOdds<-PriorHaveCovid/(1-PriorHaveCovid)\ncat(\"The Prior Covid Odds are equal to:\",PriorCovidOdds)\n#True positive: Probability of having positive Covid test when having covid = P(PositiveCovidTest|HaveCovid)\nTruePositive<-0.99\n#False positive = Probability of having positive Covid test when not having covid = P(PostiveCovidTest/DoNotHaveCovid)\nFalsePositive<-0.001\nLikelihoodRatio<-TruePositive/FalsePositive\ncat(\"The Likelihood Ratio is equal to:\",LikelihoodRatio)\nPosteriorCovidOdds<-LikelihoodRatio*PriorCovidOdds\ncat(\"The Posterior Covid Odds are equal to:\",PosteriorCovidOdds)\nPosteriorHaveCovid<- PosteriorCovidOdds/(1+PosteriorCovidOdds)\n\n\n", "Result": "", "Title": "Covid Odds after positive Home Test", "Type": "R", "Video": "https://drive.google.com/file/d/1Jy4d7ZlkWgBa0HrEhks0AWbjKVYUPC1c/preview", "Width": ""}, {"Details": "", "PreExCode": "library(dplyr)\noptions(warn=-1)\nlikelihoods <- function(data, belief, observation) {\n\n contingency_table <- table(data[,belief], data[,observation])\n \n belief_values <- sort(unique(data[,belief]))\n observation_values <- sort(unique(data[,observation]))\n \n belief_values_indices <- seq(length(belief_values))\n observation_values_indices <- seq(length(observation_values))\n \n zeroes <- rep(0, times=length(contingency_table))\n likelihood_matrix <- matrix(zeroes, nrow = length(belief_values), ncol = length(observation_values))\n \n for(i in belief_values_indices) {\n for(j in observation_values_indices) {\n true_positive <- contingency_table[i,j] / sum(contingency_table[i,])\n rows_f <- belief_values_indices[-i]\n false_positive <- sum(contingency_table[rows_f,j]) / sum(contingency_table[rows_f,])\n likelihood_ratio <- true_positive/false_positive\n likelihood_matrix[i,j] <- likelihood_ratio\n }\n }\n \n rownames(likelihood_matrix) <- belief_values\n colnames(likelihood_matrix) <- observation_values\n likelihood_table <- as.table(likelihood_matrix)\n \n likelihood_table\n}", "Query": "d<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/MoodyMarch2022b.csv\")\nll <- likelihoods(d,'Seniority', 'Grade') \nll\n", "Result": "", "Title": "Table of likelihood ratios formed from contingency table assuming rows are beliefs and columns are observations. Transformation function and its application", "Type": "R", "Video": "", "Width": ""}]}, "17": {"Name": "Naive Bayes Classification ", "Details": "\n Let's chat about naive Bayesian classifier?\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "17", "snippets": [{"Details": "", "PreExCode": "", "Query": "install.packages(\"naivebayes\")\nlibrary(naivebayes)\nmoody<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022_new.csv')\n\nset.seed(123) # for reproducibility\ntrain_index <- sample(1:nrow(moody), 0.8 * nrow(moody)) # 80% for training\ntrain_data <- moody[train_index, ]\ntest_data <- moody[-train_index, ]\n\n# Train the Naive Bayes classifier\nnb_model <- naive_bayes(GRADE~., data = train_data)\n\n# Predict the 'major' attribute using the trained model\npred<- predict(nb_model, test_data)\nmean(pred==test_data$GRADE)", "Result": "", "Title": "Simple prediction using one tep of crossvalidation ", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "# Load and clean data\r\ndf <- read.csv(\"https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Movies2022F-4-new.csv\")\r\ndf_clean <- subset(df, !is.na(Budget) & !is.na(imdb_score))\r\ndf_clean$Budget <- factor(df_clean$Budget)\r\n\r\n# Filter to \"good movies\"\r\ngood_movies <- subset(df_clean, imdb_score > 7)\r\n\r\n# Count and compute odds\r\nbudget_counts <- table(good_movies$Budget)\r\nodds_vals <- sapply(names(budget_counts), function(b) {\r\n this <- budget_counts[b]\r\n others <- sum(budget_counts) - this\r\n this / others\r\n})\r\n\r\n# Bar plot of odds\r\nbarplot(odds_vals,\r\n main = \"Odds of Budget Type (IMDB > 7)\",\r\n ylab = \"Odds (This Budget / Other Budgets)\",\r\n col = \"skyblue\",\r\n ylim = c(0, max(odds_vals) + 0.2))\r\n", "Result": "", "Title": "BarPlots", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "# Load Naive Bayes\nlibrary(e1071)\ndf <- read.csv(\"https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Movies2022F-4-new.csv\")\ndf_clean <- subset(df, !is.na(Budget) & !is.na(imdb_score))\ndf_clean$Budget <- factor(df_clean$Budget)\n\n# Train model\nmodel <- naiveBayes(Budget ~ imdb_score, data = df_clean)\n\n# Create a range of imdb scores from 3 to 9.5\nimdb_seq <- seq(3, 9.5, by = 0.1)\npreds <- predict(model, newdata = data.frame(imdb_score = imdb_seq), type = \"raw\")\n\n# Convert to data frame for plotting\npred_df <- data.frame(\n imdb_score = imdb_seq,\n High = preds[, \"High\"],\n Medium = preds[, \"Medium\"],\n Low = preds[, \"Low\"]\n)\n\n# Load ggplot2\nlibrary(ggplot2)\nlibrary(reshape2)\n\n# Melt the data for plotting\nmelted_df <- melt(pred_df, id.vars = \"imdb_score\", variable.name = \"Budget\", value.name = \"Probability\")\n\n# Plot\nggplot(melted_df, aes(x = imdb_score, y = Probability, color = Budget)) +\n geom_line(size = 1.2) +\n labs(title = \"Predicted Budget Probabilities vs IMDB Score\",\n x = \"IMDB Score\", y = \"Predicted Probability\") +\n theme_minimal()\n", "Result": "", "Title": "LinePlots", "Type": "R", "Video": "", "Width": ""}]}, "18": {"Name": "Free Style: Prediction ", "Details": "Chapter 18 of the active textbook discusses freestyle prediction models and the structure of prediction challenges. It emphasizes the importance of building prediction models from scratch to gain a deeper understanding of the data and combining multiple models for better accuracy. The chapter also introduces the concept of training, testing, and submission datasets in prediction challenges.\n\nIn section 18.1, the chapter defines a prediction model as a collection of rules that determine the predicted value of a dependent variable based on independent variables. It provides examples of such rules and highlights the significance of freestyle prediction in understanding data and improving prediction models.\n\nSection 18.2 outlines the general structure of prediction challenges and introduces the training, testing, and submission datasets. It explains that the training dataset is used to train and cross-validate the prediction model, while the testing dataset is used to apply the trained model to new data. The submission dataset is used for submitting the predicted attribute column to platforms like Kaggle. The section also provides guidance on reading and writing datasets in R.\n\nIn section 18.3, an example of a simple freestyle prediction model is presented using the Moody dataset. The chapter demonstrates the process of developing a basic prediction model based on grade cutpoints and highlights the need for significant effort to improve the model's accuracy.\n\nSection 18.3.1 discusses how to build a more advanced freestyle prediction model by identifying discriminative subsets of data. It suggests using exploratory queries and analyzing relationships between attributes to create complex rules for the prediction model.\n\nIn section 18.4, the chapter introduces a synthetic data set called \"Hire\" and demonstrates the process of developing prediction models for hiring job candidates. It emphasizes the importance of exploratory analysis using functions like table() to identify relevant attributes and formulate effective rules for decision-making.\n\nSection 18.5 discusses the concept of one-step cross-validation to assess the quality of prediction models. It explains the process of dividing the training data into training and testing subsets and emphasizes the need for multiple iterations to evaluate the model's performance on different subsets.\n\nOverall, this chapter provides an overview of freestyle prediction models, the structure of prediction challenges, and techniques for improving prediction accuracy through exploratory analysis and cross-validation.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"18.1": {"Name": "Introduction", "Details": "A prediction model is a collection of rules that, based on the values of independent variables (predictors), determine the predicted value of a dependent variable. Here are some examples of such rules:\n\nIf the score is greater than 80 and the participation is above 0.6, then the grade is 'A'.\nIf the score is between 60 and 70, the major is 'Psychology', and the frequency of asking questions is 'always', then the grade is 'B'.\nIf the score is between 40 and 50 and the tendency to doze off is 'always', then the grade is 'F'.\nFreestyle prediction refers to constructing a prediction model without relying on R library functions such as rpart and other machine learning packages. In freestyle prediction, models are developed from scratch based on plots and exploratory queries. Freestyle prediction holds significance for two reasons: First, building prediction models from scratch allows aspiring data scientists to gain an intimate understanding of the data, as opposed to blindly applying these library functions. Second, even when using prediction models based on library functions, the best models often result from combining multiple models. These combinations arise from skillfully partitioning datasets and applying different models to different subsets.\n\nAs demonstrated by our prediction challenge competitions, the most successful prediction models (those with the lowest error) typically consist of combinations of different models applied to distinct subsets of the data. Therefore, the process of constructing a prediction model typically involves incorporating freestyle prediction. In the upcoming chapters, we will develop prediction models using a freestyle approach and also utilize library functions such as rpart(), which is based on decision trees created through recursive partitioning. Additionally, we may explore other machine learning techniques. During class, we will engage in prediction challenges where students will compete using platforms like Kaggle to create prediction models that yield the lowest error on the testing data. In the next section, we will outline the structure of prediction challenges by explaining fundamental concepts such as training data, testing data, as well as different measures of accuracy and error.\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "18", "snippets": {}}, "18.2": {"Name": "Freestyle Prediction - recording ", "Details": "Note: the reference for Hire data puzzle in ppts is incorrect (16.4) - just ignore.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1ft1zDaKeQBWQrTtfbK64IXlcsU3i-3fK/preview", "child": {}, "parent": "18", "snippets": {}}, "18.3": {"Name": "Example of a simple freestyle prediction model", "Details": "In the following snippet, we are working with the Moody data set to develop a simple prediction model for the target attribute \"GRADE.\" Our objective is to predict the GRADE using the remaining columns of the train data set. In the snippet provided below, the first two lines involve uploading the train and test data sets.\r\n\r\nTypically, in practice, the train and test data sets are different to ensure a realistic evaluation of the prediction model. It is important for the train and test data sets to come from the same distribution, but with different instances, as using the same data set for both training and testing would yield unreliable predictions.\r\n\r\nHowever, in our example, we intentionally made the train and test data sets identical to highlight the challenges involved in building prediction models. This demonstrates that even when testing the model on the same data set it was trained on, our simple prediction model produces a relatively large error.\r\n\r\nIn line 3, we display all the column names of the train data set. Then, in lines 6-10, we construct a basic prediction model based on familiar grade cutpoints. For this purpose, we use a vector called \"decision,\" which is initially set with all \"Fs,\" and then the cutpoints are established line by line.\r\n\r\nOn line 11, we instantiate the variable \"myprediction,\" which provides the predicted grades. Subsequently, we calculate the error on the test data. Even though the test data set is identical to the train data set, the error of this model is determined to be 0.3, which indicates poor performance. This highlights the need for significant effort to develop a high-accuracy prediction model if we aim to perform well on real test data sets that differ from the training data set.\n\nNotice that in order to print the error in understandable format we use the print() and paste() functions which allows us to produce a sentence, rather than just a number.\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"18.3.1": {"Name": "How to build a freestyle (your own code) prediction model?", "Details": "Now, let's move on to a more advanced freestyle prediction model in the hopes of reducing the error from the previous simple model described in the previous snippet. In the following section, we will create rules that are more complex, surpassing the basic grade cutpoints used previously.\n\nThe fundamental concept behind developing increasingly precise freestyle prediction models involves subsetting the data and selecting the most frequently occurring value of the predicted variable as the prediction. Naturally, we aim to identify subsets of data that are highly discriminative, with one value being highly dominant and occurring most frequently. Choosing such a frequent value as the prediction leads to a smaller error. However, the task of finding data subsets with such dominant and most frequent values can be a trial and error process.\n\nIn the next snippet, we demonstrate a sequence of exploratory queries, consisting of a single line each, which the programmer can rely on to identify these discriminative subsets of data. This process involves experimenting with different approaches until subsets with the desired dominant and most frequent values are discovered.\n\nIn the upcoming section, we will showcase how the rpart() package automates the generation of such discriminative subsets of data using recursive partitioning. This advanced technique streamlines the process by automatically identifying the subsets that contribute significantly to prediction accuracy.\n\nBy employing these more sophisticated techniques, we aim to improve the error rate of our prediction models and achieve better results.\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "18", "snippets": [{"Details": "Based on the insights presented in the previous snippet, we can further enhance our prediction model by exploring the possibility that the cutpoints for different grades are not only dependent on the major but also on the remaining columns of the data set. To investigate this hypothesis, we can perform exploratory queries using functions like table() and subset() or create subframes.\r\n\r\nBy running these exploratory queries, we can examine the relationship between the remaining columns and the grade cutpoints. This analysis will help us determine if there are dependencies between the grade and other attributes in the data set.\r\n\r\nOnce we establish more complex rules, either separate rules for each major or potentially combinations of other attributes, we can proceed with constructing a decision vector that incorporates these rules. It's important to note that constructing such a vector may require tens or even hundreds of lines of R code, each assigning specific values to the decision vector for different subsets of the data set.\r\n\r\nBy incorporating these complex rules into our prediction model, we aim to capture the intricate dependencies and relationships within the data set. This approach allows for a more nuanced and accurate prediction of the target variable, taking into account various factors that influence the grade outcomes", "PreExCode": "", "Query": "moody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/MoodyMarch2022b.csv\")\n\n# How do we build a freestyle prediction model? Definitely start with plots like the boxplot from the section 5 (data exploration). But then follow up with exploratory queries as in the recent quizzes. Examples here use table() functon and look for situations when one grade is absoutely dominant. This would be your prediction. Thus, the goal is to slice the data using subsetting in such a way that for each slice you get a clear \"winner grade\". Then combine these subset rules into decision vector - just as we did in snippet 14.1.\n\n# Below some examples of such exploratory queries with clear grade winners.\nprint(\"The most frequent grade for all students is F.This would be the most primitive predicion model.It would be correct 405 times out of 1000 cases on the train data as the table below shows\")\ntable(moody$Grade)\nprint(\"The most frequent grade for all students who scored above 80 is A. We can predict it in 184 cases from the total of 286 cases. Better but not great\")\ntable(moody[moody$Score>80,]$Grade)\nprint(\"But if we restrict to Psychology majors we get A in 50/51 - which is fantastic predition as the table below indicates\")\ntable(moody[moody$Score>80 & moody$Major=='Psychology',]$Grade)\nprint(\"The score exceeding 80 is not great predictor anymore for CS students though - it is just 46 out of 111! Clearly we need separate cutpoints for different majors\")\ntable(moody[moody$Score>80 & moody$Major=='CS',]$Grade)\nprint(\"We need to be more demanding for CS students Only the score exceeding 95 is a good predictor of A for CS students though - it is 27 out of 33! Reinforces the main message - we need separate cutpoints for different majors\")\ntable(moody[moody$Score>95 & moody$Major=='CS',]$Grade)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "18", "snippets": [{"Details": "", "PreExCode": "", "Query": "train<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/MoodyMarch2022b.csv\")\ntest<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/MoodyMarch2022b.csv\")\nprint(paste(\"The column names are:\"))\ncolnames(train)\nmyprediction<-train\ndecision <- rep('F',nrow(myprediction))\ndecision[myprediction$Score>40] <- 'D'\ndecision[myprediction$Score>60] <- 'C'\ndecision[myprediction$Score>70] <- 'B'\ndecision[myprediction$Score>80] <- 'A'\nmyprediction$Grade <-decision\nerror <- mean(test$Grade!= myprediction$Grade)\nprint(paste(\"The value of error computed on testing data is \", error))", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "18.4": {"Name": "Building simple prediction model on Hire table", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"18.4.1": {"Name": "Hidden pattern embedded in Hire data set by the professor. ", "Details": "In the spring semester of 2023, we created a data puzzle around a synthetic dataset called \"Hire,\" which provided information about a hypothetical group of job candidates within a company. Each candidate entry included standard attributes such as coding skills, the impression they made, and the college they attended. The target variable in this dataset was a boolean decision indicating whether the candidate was hired or not.\n\nTo generate the synthetic dataset, we utilized a tool called DataMaker, which we developed at Rutgers University. We embedded specific rules into the data to create patterns for analysis. For instance, candidates who did not attend the college named \"Best College\" and were not weak coders would be hired. Additionally, candidates who excelled in coding, did not display the \"Outgoing\" attribute (as the company preferred serious and introverted developers), and did not major in IT would also receive positive hiring decisions. None of other candidates was hired. However, it is important to note that the synthetic data introduced some level of randomness or noise to reflect real-world variability. Thus the embedded rules had \njust high probability of Hiring embedded and were not deterministic.\n\nThe objective for the students was to either uncover the embedded rules or develop a more accurate prediction model. It is worth mentioning that the professor's prediction model was based on the embedded rules within the dataset. Therefore, this posed a challenging benchmark for the students, as they were unaware of the underlying patterns and had to discover them independently.\n\nThe first snippet presented below demonstrates the prediction model based on the embedded rules. The subsequent snippet showcases the prediction model created by the top-performing student, Howard Luo, which surprisingly not only matched the benchmark but significantly surpassed its accuracy. All in all, nearly 200 prediction models were submitted and quite a few came really close in accuracy to Howard's model. \n\nThis challenging puzzle and the subsequent achievements of the students highlight the complexity of the task at hand. Uncovering the embedded pattern or developing a superior prediction model required diligent analysis, innovative thinking, and a deep understanding of the dataset. The success of Howard Luo's model showcases the students' exceptional skills and their ability to surpass the professor's benchmark.\n\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "18", "snippets": [{"Details": "in the next snippet we show the best prediction model in class. The student, Howard Luo's best solution- discovering even deeper pattern in the data than the pattern embedded by professor (due to noise and a bit of overfitting). Again the snippet below show the accuracy of Howard's model on training and testing data respectively. Much higher than professor's model!", "PreExCode": "", "Query": "train<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/HireTrainApr10.csv')\ntest<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/HireTestFull.csv')\ncolnames(train)\ndecision<-rep('No',nrow(train))\ndecision[train$College!='BestCollege'& train$Coding!='Weak']<-'Yes'\ndecision[train$Major!='IT' & train$Impression!='Outgoing' &train$Coding=='Excellent']<-'Yes'\naccuracy<-round(mean(train$Hired==decision),2)\nprint(paste(\"The accuracy of professor model on traing data is\",accuracy))\ndecision<-rep('No',nrow(test))\ndecision[test$College!='BestCollege'& test$Coding!='Weak']<-'Yes'\ndecision[test$Major!='IT' & test$Impression!='Outgoing' &test$Coding=='Excellent']<-'Yes'\naccuracy<-round(mean(test$Hired==decision),2)\nprint(paste(\"The accuracy of professor model on test data is\",accuracy))\n", "Result": "", "Title": "Building simple prediction model on Hire table - using the pattern embedded by the professor", "Type": "R", "Video": "", "Width": ""}]}, "18.4.2": {"Name": "Best student prediction model (by Howard Luo) ", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "18", "snippets": [{"Details": "\r\n What does the above code snippet do?\r\n $\"ChatGPT$ \r\n", "PreExCode": "", "Query": "train<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/HireTrainApr10.csv')\ntest<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/HireTestFull.csv')\ndecision = rep('Yes', nrow(train))\ndecision[train$Impression == \"Nerdy\" & train$Major == \"IT\" & train$College == \"BestCollege\"] <- \"No\"\ndecision[train$Coding == \"OK\" & train$Impression == \"Shy\" & train$College == \"BestCollege\"] <-\"No\"\ndecision[train$Coding == \"OK\" & train$Impression == \"Nerdy\" & train$College == \"BestCollege\"] <- \"No\"\ndecision[train$Coding == 'Weak']<-'No'\ndecision[train$Coding == \"Weak\" & train$Impression == \"Nerdy\" & train$College ==\"Redbrick\"] <- \"Yes\"\naccuracy<-round(mean(train$Hired==decision),2)\nprint(paste(\"The accuracy of student model on training data is\",accuracy))\ndecision = rep('Yes', nrow(test))\ndecision[test$Impression == \"Nerdy\" & test$Major == \"IT\" & test$College == \"BestCollege\"] <- \"No\"\ndecision[test$Coding == \"OK\" & test$Impression == \"Shy\" & test$College == \"BestCollege\"] <-\"No\"\ndecision[test$Coding == \"OK\" & test$Impression == \"Nerdy\" & test$College == \"BestCollege\"] <- \"No\"\ndecision[test$Coding == 'Weak']<-'No'\ndecision[test$Coding == \"Weak\" & test$Impression == \"Nerdy\" & test$College ==\"Redbrick\"] <- \"Yes\"\naccuracy<-round(mean(test$Hired==decision),2)\nprint(paste(\"The accuracy of student model on test data is\",accuracy))\n\n\n\n", "Result": "", "Title": "The student, Howard Luo's best solution- discovering even deeper pattern in the data\nthan the pattern embedded by professor (due to noise and a bit of overfitting). ", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "18", "snippets": {}}, "18.5": {"Name": "One-step crossvalidation", "Details": "How can we determine the quality of our prediction model? It's important to consider that while a model may perform exceptionally well on the training data, it could perform poorly when faced with new, unseen testing data. This scenario, known as a \"Kaggle surprise,\" is a nightmare for any prediction model builder. Kaggle surprises frequently occur in prediction competitions, where participants develop models that are excessively tailored to the training data, providing them with a false sense of accuracy, only to fail when tested on new data, resulting in a significantly higher error rate.\r\n\r\nTo avoid or mitigate this issue, cross-validation is crucial. Let's explore cross-validation in the following explanation. We divide our training data into two parts: the actual training data and the testing data, which comprise the remaining portion of our training dataset. By doing so, we utilize a portion of the training data for testing purposes. This split is performed randomly. It's important to note that the demonstration provided here depicts only a single step of cross-validation, but in practice, it should be repeated multiple times. This iterative process allows us to assess how our model performs across various randomly selected subsets of the training data, enabling us to identify inconsistent results (high variance of error). Such inconsistencies serve as warning signs for potential future Kaggle surprises.\r\n\r\nIn the snippet below, we illustrate one step of the cross-validation process. By running this snippet repetitively, you can see how each time it produces different results. This is done by scrambling (permuting) the rows of the dataframe randomly and testing your model each time on a different random subset of the dataframe. In the cross-validation function introduced next, you train your model on a random subset of the training data and test it on the complement of this \"pretended\" training model. This will ensure better stability of the model and more assurance that your model will perform well on the new testing data, thus avoiding the Kaggle surprise.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "18", "snippets": [{"Details": "", "PreExCode": "", "Query": "train<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n#scramble the train frame\nv<-sample(1:nrow(train))\ntrainScrambled<-train[v, ]\n#one step crossvalidation\ntrainSample<-trainScrambled[(nrow(trainScrambled)-10):nrow(trainScrambled), ]\nmyprediction<-trainSample\n\n#prediction model - free style\n#How to test how good your model is?\n#Crossvalidation: Divide train data set into two disjoint subsets T (train) and train MINUS T, the complement of T. \n#You use T to derive your prediction model and the complement of T (train MINUS T) to validate (test it).\n# We assume that you created prediction model looking just at the subset of training data T=trainScrambled[1:990, ]. \n#Since for crossvalidation we train on a subset T of the training data set and validate (test) on the complement of T. \n#In this case T= trainScrambled[1:990, ] and complement of T (to validate/test) is stored as trainSample.\n#You can do it multiple times. And observe the error and its stability.\n#You build your model using the decision vector. Here is very SIMPLISTIC MODEL which is just illustration. Your model should have much better error and be more sophisticated. \n\ndecision <- rep('F',nrow(myprediction))\ndecision[myprediction$Score>40] <- 'D'\n\ndecision[myprediction$Score>60] <- 'C'\n\ndecision[myprediction$Score>70] <- 'B'\n\ndecision[myprediction$Score>80 ] <- 'A'\n\nmyprediction$Grade <-decision\nerror <- mean(trainSample$Grade!= myprediction$Grade)\ncat('The is error rate for random subset of training data defined by last 10 tuples of scramble of training data')\nerror ", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "18.6": {"Name": "Confusion Matrix ", "Details": "A confusion matrix is a table that is often used to describe the performance of a classification model on a set of test data for which the true values are known. It allows visualization of the performance of an algorithm.\n\nIn a confusion matrix, each row represents the instances in an actual class, while each column represents the instances in a predicted class. The cells of the matrix show the counts of true positive (TP), false positive (FP), true negative (TN), and false negative (FN) predictions made by the model.\n\nHere's a breakdown of the components:\n\nTrue Positive (TP): The number of instances correctly predicted as positive (e.g., correctly identified as belonging to the positive class).\nFalse Positive (FP): The number of instances incorrectly predicted as positive (e.g., incorrectly identified as belonging to the positive class when they actually belong to the negative class).\nTrue Negative (TN): The number of instances correctly predicted as negative (e.g., correctly identified as belonging to the negative class).\nFalse Negative (FN): The number of instances incorrectly predicted as negative (e.g., incorrectly identified as belonging to the negative class when they actually belong to the positive class).\nA confusion matrix helps to evaluate the performance of a classification model by providing insights into the types of errors it makes.\n\nThis simple code does it - assuming predctions as column of values predicted by classification method. \n\nconfusion_matrix <- table(predictions, test_data$Target)", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "18", "snippets": [{"Details": "", "PreExCode": "", "Query": "train<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/HireTrainApr10.csv')\ntest<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/HireTestFull.csv')\ndecision = rep('Yes', nrow(train))\ndecision[train$Impression == \"Nerdy\" & train$Major == \"IT\" & train$College == \"BestCollege\"] <- \"No\"\ndecision[train$Coding == \"OK\" & train$Impression == \"Shy\" & train$College == \"BestCollege\"] <-\"No\"\ndecision[train$Coding == \"OK\" & train$Impression == \"Nerdy\" & train$College == \"BestCollege\"] <- \"No\"\ndecision[train$Coding == 'Weak']<-'No'\ndecision[train$Coding == \"Weak\" & train$Impression == \"Nerdy\" & train$College ==\"Redbrick\"] <- \"Yes\"\naccuracy<-round(mean(train$Hired==decision),2)\nprint(paste(\"The accuracy of student model on training data is\",accuracy))\ndecision = rep('Yes', nrow(test))\ndecision[test$Impression == \"Nerdy\" & test$Major == \"IT\" & test$College == \"BestCollege\"] <- \"No\"\ndecision[test$Coding == \"OK\" & test$Impression == \"Shy\" & test$College == \"BestCollege\"] <-\"No\"\ndecision[test$Coding == \"OK\" & test$Impression == \"Nerdy\" & test$College == \"BestCollege\"] <- \"No\"\ndecision[test$Coding == 'Weak']<-'No'\ndecision[test$Coding == \"Weak\" & test$Impression == \"Nerdy\" & test$College ==\"Redbrick\"] <- \"Yes\"\naccuracy<-round(mean(test$Hired==decision),2)\nprint(paste(\"The accuracy of student model on test data is\",accuracy))\ncat(\"Confusion matrix is\")\nconfusion_matrix <- table(decision, test$Hired)\nconfusion_matrix\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "18.7": {"Name": "Summary of section 18", "Details": "Chapter 18 of the active textbook delves into the fascinating world of freestyle prediction models and prediction challenges. In this chapter, the readers are taken on a journey to explore the intricacies of prediction modeling and the art of building accurate models from scratch. The chapter sheds light on the structure of prediction challenges and the significance of combining multiple models for improved accuracy.\n\nSection 18.1, titled \"Introduction,\" lays the foundation for understanding prediction models. It defines a prediction model as a collection of rules that determine the predicted value of a dependent variable based on independent variables. The examples provided in this section illustrate how these rules can be formulated to predict various outcomes. The chapter stresses the importance of freestyle prediction, where aspiring data scientists develop models from scratch rather than relying solely on pre-existing library functions like rpart and machine learning packages. By building prediction models from scratch, data scientists gain a deeper understanding of the data and its underlying patterns, leading to more insightful and accurate models. Furthermore, the chapter advocates for combining multiple models, a practice that often results in the best-performing prediction models. These combinations arise from skillfully partitioning datasets and applying different models to distinct subsets, enabling a more comprehensive analysis of the data and predictions.\n\nSection 18.2, titled \"General Structure of Prediction Challenges,\" provides an in-depth overview of the structure of prediction challenges and introduces the concept of training, testing, and submission datasets. The training dataset is the foundation of the prediction challenge, used for building and cross-validating prediction models. It contains all the attributes necessary for training, including the target attribute. On the other hand, the testing dataset is employed to apply the trained model to new and unseen data. While it shares the attributes of the training dataset, it omits the target attribute, making it a valuable tool for assessing the model's performance on previously unseen data. The submission dataset is designed for participants to submit their predicted attribute columns to platforms like Kaggle. It only contains two columns: an index column and the predicted attribute column. The chapter provides practical guidance on reading and writing datasets in R to ensure smooth participation in prediction challenges.\n\nSection 18.3, titled \"Example of a Simple Freestyle Prediction Model,\" presents a practical illustration of a freestyle prediction model using the Moody dataset. This section allows readers to witness the step-by-step process of constructing a prediction model based on grade cutpoints. The goal is to predict the GRADE attribute using the remaining columns of the train dataset. Despite using identical train and test datasets in this example, the model's error remains relatively high at 0.3. This demonstrates the challenges involved in developing high-accuracy prediction models, even when the model is tested on the same data it was trained on. The section emphasizes that significant effort is required to achieve accurate predictions on new and unseen data.\n\nIn Section 18.3.1, titled \"How to Build a Freestyle Prediction Model?,\" the chapter delves deeper into constructing more advanced freestyle prediction models by identifying discriminative subsets of data. The process involves experimentation with exploratory queries and analysis of relationships between attributes. The goal is to create complex rules for the prediction model that are highly discriminative and lead to more accurate predictions. The section acknowledges that building such complex models may require tens or even hundreds of lines of R code, each assigning specific values to the decision vector for different subsets of the data. The chapter stresses the importance of capturing intricate dependencies and relationships within the data to achieve more nuanced and accurate predictions.\n\nSection 18.4, titled \"Building Simple Prediction Model on Hire Table,\" introduces the synthetic dataset \"Hire\" and demonstrates the process of building prediction models for hiring job candidates. The professor embedded specific rules into the synthetic data to create patterns for analysis. The students' task was to either uncover these embedded rules or develop more accurate prediction models. The chapter highlights the challenge posed by the hidden patterns in the dataset and showcases the achievements of students who attempted to surpass the professor's benchmark. Nearly 200 prediction models were submitted, with some students coming remarkably close to achieving accuracy comparable to the professor's model.\n\nIn Section 18.4.1, titled \"Hidden Pattern Embedded in Hire Dataset by the Professor,\" the chapter further elaborates on the synthetic dataset \"Hire\" and the rules embedded by the professor. The professor's model served as a benchmark for the students, and they were tasked with either discovering the embedded rules or developing superior prediction models. The section underscores the complexity of the task, as students had to independently uncover the underlying patterns in the data and develop models that outperformed the professor's embedded rules. The chapter celebrates the exceptional skills of students who successfully surpassed the professor's benchmark, showcasing their diligence, innovative thinking, and deep understanding of the dataset.\n\nSection 18.4.2, titled \"Best Student Prediction Model (by Howard Luo),\" highlights the achievements of Howard Luo, the student who developed the best prediction model in class. Howard's model not only matched the professor's benchmark but significantly surpassed its accuracy. The chapter commends Howard's exceptional model, which showcased his exceptional skills and ability to uncover deeper patterns within the data. His success exemplifies the students' ingenuity and dedication to surpassing the benchmarks set by the professor.\n\nIn Section 18.5, titled \"One-Step Cross-Validation,\" the chapter delves into the importance of cross-validation to assess the quality of prediction models. It introduces the concept of dividing the training data into two parts: the actual training data and the testing data. By utilizing a portion of the training data for testing, data scientists can assess how the model performs on various randomly selected subsets of the training data, thus identifying any inconsistencies or high variance of error. The chapter emphasizes the potential pitfalls of overfitting and \"Kaggle surprises,\" where models perform well on the training data but fail on new, unseen testing data. To avoid such surprises, the chapter recommends implementing multiple iterations of cross-validation to evaluate model performance more comprehensively.\n\nIn conclusion, Chapter 18 of the active textbook offers a comprehensive exploration of freestyle prediction models and prediction challenges. It equips readers with essential knowledge and practical insights into building accurate prediction models from scratch. The chapter emphasizes the significance of understanding data and uncovering hidden patterns to develop precise prediction models. It showcases the complexity of prediction challenges and celebrates the achievements of students who surpassed benchmarks and achieved remarkable accuracy in their prediction models. Throughout the chapter, readers gain valuable knowledge and tools to excel in prediction modeling and develop models that deliver exceptional results in real-world applications.\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "18", "snippets": {}}}, "parent": "18", "snippets": {}}, "19": {"Name": "\ud83c\udfa6 Predictions with rpart (Video) \ud83c\udfa6", "Details": "This section discusses the use of the rpart package in R for predictions with decision trees. Decision trees are popular for classification and prediction because they generate rules that are easy to understand and require less computation. The rpart library is used to implement decision trees, and the rpart() function is used to build the decision tree model.\n\nTo build a decision tree using rpart(), the syntax is as follows:\nrpart(formula, method, data, control, ...)\n\nThe formula specifies the prediction column and the predictor columns.\nThe method describes the type of decision tree (e.g., \"class\" for classification).\nThe data parameter specifies the dataset.\nThe control parameter provides control parameters for the decision tree.\nThe rpart() function automates the process of building prediction models by recursively splitting and partitioning the data to find optimal subsets. The resulting decision tree can be represented as a binary tree. The output of the rpart() function provides information about the decision tree's structure and attributes used for splitting.\n\nTo visualize the decision tree, the rpart.plot library can be used with the rpart.plot() function. However, the code examples for using rpart.plot() are not supported in the interactive code blocks. The output of the decision tree can be interpreted to understand the splitting conditions and probabilities of outcomes.\n\nThe rpart.control() function is used to set control parameters for the rpart() function. Parameters like minsplit, minbucket, and cp can be specified to control the splitting process and the size of the tree.\n\nCross-validation is discussed as a method to evaluate the model's performance and prevent overfitting. By partitioning the training dataset into a training set and a validation set, the model's accuracy on the validation set can be compared with the training accuracy to assess overfitting.\n\nPlease note that the code examples and plots provided in the original section are not directly runnable here.\n\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "https://drive.google.com/file/d/1Ac27aLqTkWN5uXZX6sfqBfBWJYj2w-D-/preview", "child": {"19.1": {"Name": "Introduction", "Details": "Decision trees are widely regarded as one of the most powerful and popular tools in the realm of classification and prediction. One of the primary reasons behind their immense popularity is their ability to generate rules that are remarkably easier to understand when compared to other modeling techniques. This attribute makes decision trees an attractive option, particularly when interpretability and explainability of the model are crucial considerations.\r\n\r\nOne of the key advantages of decision trees lies in their efficient computational requirements. Unlike some other complex models, decision trees necessitate fewer computations during the modeling and prediction processes. This efficiency not only reduces the computational burden but also enables faster processing and decision-making. Consequently, decision trees can be highly valuable in scenarios where there are time constraints or the need for real-time predictions.\r\n\r\nAnother notable strength of decision trees is their versatility in handling different types of variables. Whether the dataset comprises continuous/numerical variables or categorical variables, decision trees are adept at accommodating both. This flexibility allows decision trees to effectively capture various types of information present in the data, leading to more comprehensive and accurate predictions.\r\n\r\nFurthermore, decision trees excel in handling missing values and outliers. By employing appropriate splitting criteria and considering multiple attributes, decision trees can effectively handle missing data points by making informed decisions during the tree-building process. This adaptability contributes to the robustness of decision trees in dealing with real-world datasets, which often exhibit missing values or outliers.\r\n\r\nThe interpretability of decision trees is another significant advantage. The hierarchical structure of decision trees presents a clear visualization of the decision-making process. Each branch represents a decision based on specific features or attributes, leading to a final outcome at the leaf nodes. This transparent representation enables users to comprehend the logic behind the predictions made by the model, making decision trees particularly useful in domains where explainability is of paramount importance, such as healthcare, finance, or legal sectors.\r\n\r\nMoreover, decision trees can handle both binary and multi-class classification problems. By leveraging different splitting criteria and decision rules, decision trees can effectively classify instances into multiple categories, making them suitable for a wide range of classification tasks. Additionally, decision trees can be extended to handle regression problems by predicting continuous values instead of discrete categories. This flexibility and applicability across various problem domains contribute to the widespread adoption of decision trees in diverse fields.\r\n\r\nDespite their numerous strengths, decision trees also have some limitations. They are prone to overfitting when the tree becomes too complex, which can result in poor generalization to unseen data. Overfitting occurs when the model becomes too specific to the training data, capturing noise or irrelevant patterns. Regularization techniques such as pruning, limiting the tree depth, or setting a minimum number of samples per leaf can help mitigate overfitting and improve the generalization performance of decision trees.\r\n\r\nIn conclusion, decision trees offer a powerful and popular approach for classification and prediction tasks. Their ability to generate interpretable rules, handle various types of variables, and require fewer computations makes them highly desirable in many practical applications. By understanding the strengths and limitations of decision trees, practitioners can effectively utilize these versatile models to gain valuable insights and make accurate predictions in diverse domains.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": {}}, "19.2": {"Name": "Use of Rpart", "Details": "The library associated with this RPART is called rpart. Install this library using install.packages(\"rpart\").\r\n\r\n\r\nSyntax for building the decision tree using rpart():\r\n\r\n\r\nrpart( formula , method, data, control,...)\r\nformula: here we mention the prediction column and the other related columns(predictors) on which the prediction will be based on.\r\nprediction ~ predictor1 + predictor2 + predictor3 + ...\r\nmethod: here we describe the type of decision tree we want. If nothing is provided, the function makes an intelligent guess. We can use \u201canova\u201d for regression, \u201cclass\u201d for classification, etc.\r\ndata: here we provide the dataset on which we want to fit the decision tree on.\r\ncontrol: here we provide the control parameters for the decision tree. Explained more in detail in the section further in this chapter.\r\n\r\nFor more info on the rpart function visit rpart documentation\r\n\r\n\r\nLets look at an example on the Moody 2022 dataset.\r\n\r\n\r\nWe will use the rpart() function with the following inputs:\r\nprediction -> Grade\r\npredictors -> Score, Major,Seniority,GPA\r\ndata -> moody dataset\r\nmethod -> \u201cclass\u201d for classification.\r\ndata -> moody dataset\r\nmethod -> \u201cclass\u201d for classification.\r\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"19.2.1": {"Name": "rpart()", "Details": "Recursive partitioning (rpart) automates the process of building prediction models, eliminating the need for manual data exploration. It follows the prediction principle described earlier: for a subset of the dataset, the predicted value of the categorical target variable is the most frequent value within that subset. Instead of manually searching for subsets where one target variable value dominates (a laborious task!), rpart automates this process. It recursively splits and partitions the data to discover nearly optimal subsets. You no longer have to painstakingly construct a decision vector for numerous subframes. The rpart() function takes care of it all. Simply run it and later use the predict() function to generate the decision vector. It's as simple as that, requiring just two lines of code!\n\nMany students may find the rpart output confusing, mistakenly thinking they need to construct the decision vector line by line. However, this is far from the truth. The output of rpart() is purely informational and none of its lines need to be used in your code.\n\nThis process is similar to the step-by-step approach we previously described using the table() function to identify the most frequent values of the target attribute through data subsetting. However, that process was time-consuming and exhaustive. With rpart, you can bypass such arduous procedures. For example, the code snippet below demonstrates how to build a prediction model for the Moody dataset, predicting the GRADE attribute using the Score, Grade, Major, Seniority, GPA attributes. Just execute the following function:\n\nrpart(Grade ~ Score+Major+Seniority+GPA , data = moody, method = \"class\")\n\nHere are the core functions performed by rpart: First, it selects the \"best\" attribute to split the data on. This is done recursively, resulting in the creation of a decision tree. For instance, it determines which attribute should be the first to split the data: Score,Major, Seniority, GPA Note that two of these attributes are numerical, while the other two are categorical. In each case, rpart evaluates each attribute and its values, searching for the split that produces the most discriminative subset. By \"discriminative,\" we mean a subset where the most frequent value of the target attribute (Grade) dominates. For example, a subset may consist of 99% A's or 95% F's. The higher the ratio, the greater the potential for a highly accurate prediction model.\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": [{"Details": "Let's provide an explanation for part of the rpart output:\r\n\r\nThe first row corresponds to the root node of the decision tree. It contains 828 instances. The numbers in parentheses, (0.19 0.18 0.19 0.22 0.21), indicate the distribution of the target attribute GRADE. The most frequent grade is D, representing 22% of all cases. If we were to stop here without further partitioning, our predicted grade would be D, as it minimizes the error. However, the number 642 represents the error, indicating the number of cases that would be classified incorrectly if we labeled them all as D. In this case, the error would be 1 - 0.22 = 0.78, resulting in an accuracy of only 22%. Therefore, further partitioning is necessary. The attribute selected by rpart for the split at this node is SCORE, with a splitting value of 80.025.\r\n\r\nMoving on, the second row indicates a split based on the condition \"SCORE >= 80.025\". There are 176 instances that satisfy this condition, and A is the most frequent grade among them. In fact, there are only 16 cases that would be misclassified if we labeled this node as A. This misclassification rate is deemed acceptable, and the asterisk (*) denotes the end of this branch, indicating that A is the final prediction.\r\n\r\nThe third row represents the alternative branch when the condition \"SCORE < 80.025\" is true. This branch contains 652 instances, with the majority of them belonging to grade D. However, the error rate is still substantial, as 466 cases would be misclassified if we labeled this node as D. Therefore, stopping at this point would result in a high error rate and low accuracy. Thus, further splitting is required.\r\n\r\nAs the tree progresses, you can observe that although SCORE initially appears to be the most influential attribute, other attributes eventually play a role in the decision-making process. For instance, PARTICIPATION becomes influential, and as the tree goes deeper, even the categorical attributes, such as DOZING_OFF and TEXTING_IN_CLASS, contribute to determining the final grade in Professor Moody's class.\r\n\r\nIn summary, the rpart output reveals how the decision tree is constructed by iteratively splitting the data based on attribute conditions. The rows with asterisks represent cases where a decision can be made, while the rows without asterisks indicate the need for further splitting. The attribute importance may vary at different levels of the tree, and ultimately, multiple attributes contribute to the final predictions.", "PreExCode": "", "Query": "library(rpart)\nmoody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n# Use of the rpart() function.\nrpart(Grade ~., data = moody,method = \"class\")", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "19", "snippets": {}}, "19.3": {"Name": "Visualize the Decision tree ", "Details": "

To visualize and understand the rpart() tree output in the easiest way possible, we use a library called rpart.plot. The function rpart.plot() of the rpart.plot library is the function used to visualize decision trees.

NOTE: The online runnable code block does not support rpart.plot library and functions, thus the output of the following code examples are provided directly.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"19.3.1": {"Name": " rpart.plot()", "Details": "ln your R studio - use library(rpart.plot)\n\nUse of the rpart.plot() function to visualize the decision tree.\nrpart.plot(tree)\n\nThe tree presented below provides a more user-friendly representation of the rpart output. The numbers displayed below each node directly correspond to the numbers discussed earlier when explaining the rpart output. Each node description begins with the most frequent grade within the partition it represents, which can be one of the grades A, B, C, D, or F. Following that is the grade distribution for that particular node, with the top grade being the most frequent. Lastly, the percentage number below the grade distribution represents the proportion of the entire dataset covered by that specific node. For instance, if the number is 60%, it indicates that the corresponding subset (partition) covers 60% of the entire dataset. As we traverse down the tree to more specific nodes, this percentage will gradually decrease.\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": [{"Details": "The tree presented below provides a more user-friendly representation of the rpart output. The numbers displayed below each node directly correspond to the numbers discussed earlier when explaining the rpart output. Each node description begins with the most frequent grade within the partition it represents, which can be one of the grades A, B, C, D, or F. Following that is the grade distribution for that particular node, with the top grade being the most frequent. Lastly, the percentage number below the grade distribution represents the proportion of the entire dataset covered by that specific node. For instance, if the number is 60%, it indicates that the corresponding subset (partition) covers 60% of the entire dataset. As we traverse down the tree to more specific nodes, this percentage will gradually decrease.\n\n

$\"\"$ \r\n

Output Plot of rpart.plot() function

\r\n

We can see that after plotting the tree using rpart.plot() function, the tree is more readable and provides better information about the splitting conditions, and the probability of outcomes. Each leaf node has information about

\r\n

the grade category.
the outcome probability of each grade category.
the records percentage out of total records.

\r\n

To study more in detail the arguments that can be passed to the rpart.plot() function, please look at these guides rpart.plot and Plotting with rpart.plot (PDF)

\r\n

NOTE: In this chapter, from this point forward, the rpart.plots() generated in any example below will be shown as images, and also the code to generate those rpart.plots will be commented in the interactive code blocks. If you want to generate these plots yourself, please use a local Rstudio or R environment.

", "PreExCode": "", "Query": "# First lets import the rpart library\nlibrary(rpart)\n\n# Import dataset\nmoody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n# Use of the rpart() function.\nrpart(Grade ~., data = moody,method = \"class\")\n\n# Now lets import the rpart.plot library to use the rpart.plot() function.\n#library(rpart.plot)\n\n# Use of the rpart.plot() function to visualize the decision tree.\n#rpart.plot(tree)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "19", "snippets": {}}, "19.4": {"Name": "Rpart Control", "Details": "[This text has been generated by GPT 3.5]\n\nBy specifying appropriate control parameters using the rpart.control() function, users can influence the structure and complexity of the decision tree, ensuring a better fit for the data and avoiding overfitting.\n\n\nThe rpart.control() function is used in the R programming language to specify control parameters for building decision trees using the rpart() function from the rpart package. These control parameters allow users to customize the behavior and structure of the decision tree.\n\nThe rpart.control() function takes several arguments that define the control parameters:\n\nminsplit: This parameter specifies the minimum number of observations required in a node for a split to be attempted. If the number of observations in a node is less than minsplit, the node will not be further split. This helps prevent overfitting by avoiding excessive partitioning of small nodes. The default value is usually 20.\n\nminbucket: This parameter sets the minimum number of observations required in any terminal (leaf) node of the tree. If a potential split would result in a leaf node with fewer observations than minbucket, the split is not attempted, and the node becomes a terminal node. Similar to minsplit, this parameter helps control the size of the tree and prevent overfitting. The default value is typically one more than minsplit.\n\ncp: The complexity parameter (cp) is used to control the complexity of the decision tree. It specifies the minimum improvement in the model's accuracy that must be achieved for a split to be considered. If the improvement is below cp, the split is not attempted, leading to a simpler tree. Higher values of cp result in smaller trees. The default value is 0.01.\n\nOther arguments: The rpart.control() function also accepts additional arguments that can further customize the behavior of the decision tree. These arguments include maxdepth to limit the maximum depth of the tree, xval to enable cross-validation for model selection, and parms to pass additional arguments to the rpart() function.\n\nIn the next subsections 17.4.1-17.4.x we just illustrate the consequences of choices of different parameters such as Minsplit, Minbucket and cp on the final decision tree - which we will display every time, using rplot() function. \n\n\n\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"19.4.1": {"Name": "rpart(): Minsplit = 200", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": [{"Details": "

$\"\"$ \r\n

Output tree plot of after setting minsplit=200 in rpart.control() function

", "PreExCode": "", "Query": "library(rpart)\nmoody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n# Use of the rpart() function with the control parameter minsplit=200\ntree <- rpart(Grade ~., data = moody, method = \"class\",control=rpart.control(minsplit = 200))\n\ntree\n\n#library(rpart.plot)\n#rpart.plot(tree,extra = 2)", "Result": "", "Title": "", "Type": "R", "Video": "https://drive.google.com/file/d/1eQAwPwW5-mPoEjQ6g9puFy3y4XRQOabf/preview", "Width": ""}]}, "19.4.2": {"Name": "rpart(): Minsplit = 100", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": [{"Details": "", "PreExCode": "", "Query": "library(rpart)\nmoody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n# Use of the rpart() function with the control parameter minsplit=100\ntree <- rpart(Grade ~., data = moody, method = \"class\",control=rpart.control(minsplit = 100))\n\ntree\n\n#library(rpart.plot)\n#rpart.plot(tree,extra = 2)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "19.4.3": {"Name": " rpart(): Minbucket = 100", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": [{"Details": "", "PreExCode": "", "Query": "library(rpart)\nmoody<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv')\n\n# Use of the rpart() function with the control parameter Minbucket=100\ntree <- rpart(Grade ~. , data = moody, method = \"class\",control=rpart.control(minbucket = 100))\n\ntree\n\n#library(rpart.plot)\n#rpart.plot(tree,extra = 2)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "19.4.4": {"Name": "rpart(): Minbucket = 200", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": [{"Details": "", "PreExCode": "", "Query": "library(rpart)\nmoody<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv')\n\n# Use of the rpart() function with the control parameter Minbucket=200\ntree <- rpart(Grade ~., data = moody, method = \"class\",control=rpart.control(minbucket = 200))\n\ntree\n\n#library(rpart.plot)\n#rpart.plot(tree,extra = 2)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "19.4.5": {"Name": "rpart(): cp = 0.05", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": [{"Details": "", "PreExCode": "", "Query": "library(rpart)\nmoody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n\n# Use of the rpart() function with the control parameter cp=0.2\ntree <- rpart(Grade ~ ., data = moody,method = \"class\",control=rpart.control(cp = 0.05))\n\ntree\n\n#library(rpart.plot)\n#rpart.plot(tree)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "19.4.6": {"Name": " rpart(): cp = 0.005", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": [{"Details": "", "PreExCode": "", "Query": "library(rpart)\nmoody<-read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv\")\n# Use of the rpart() function with the control parameter cp=0.005\ntree <- rpart(Grade ~ ., data = moody,method = \"class\",control=rpart.control(cp = 0.005))\n\ntree\n\n#library(rpart.plot)\n#rpart.plot(tree)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "19", "snippets": {}}, "19.5": {"Name": "Cross Validation ", "Details": "We use cross-validation to prevent our prediction model from surprising us with poor performance (high error) on the testing data, even if it performs well on the training data. This phenomenon is known as overfitting, where the model has high accuracy on the training dataset but low accuracy on the test dataset. However, without access to the test dataset, it becomes challenging to determine if overfitting is occurring. This is where cross-validation comes into play.\n\nSince we cannot directly assess accuracy on the test dataset, we partition our training dataset into a training set and a validation set. We train our model (e.g., rpart or lm) on the training partition and test it on the validation partition. The partitioning is defined by a split ratio. For example, if the split ratio is 0.7, 70% of the training dataset will be used for actual training, and the remaining 30% will be used for validation (or testing). The accuracy of this validation data is referred to as cross-validation accuracy.\n\nTo determine if overfitting is occurring, we compare the training accuracy with the cross-validation accuracy. If the training accuracy is high while the cross-validation accuracy is low, it indicates overfitting.\n\nThe cross_validate() function is used for cross-validation and has the following parameters:\n\ndata: The dataset on which cross-validation is performed.\ntree: The decision tree generated using rpart.\nn_iter: Number of iterations.\nsplit_ratio: The splitting ratio of the data into the training data and validation data.\nmethod: The prediction method, such as \"class\" for classification.\nThe function works as follows:\n\nIt randomly partitions the data into the training and validation sets.\nIt constructs two decision trees on the training partition:\nThe tree that is passed to the function.\nA tree constructed with all attributes as predictors and no control parameters.\nIt determines the accuracy of both trees on the validation partition and returns the accuracy values for both trees.\nThe values in the first column (accuracy_subset) returned by the cross_validate() function are crucial for detecting overfitting. If these values are significantly lower than the training accuracy, it indicates overfitting.\n\nIdeally, the values in accuracy_subset should be close to each other (low variance). If the values differ significantly, it suggests that the model or tree has high variance, which is undesirable.\n\nEach iteration of cross-validation creates a different random partition of the training and validation sets, resulting in possibly different accuracy values for each iteration.\n\nYou can see the cross_validate() function in action in the example provided below. The example uses the tree with the formula Grade ~ . and the control parameter minsplit=100. For the cross_validate() function, n_iter is set to 5, and split_ratio is set to 0.7.\n\nNote: The Cross-Validation repository is already preloaded for the interactive code block below, allowing you to use the cross_validate() function directly. However, if you want to use the cross_validate() function locally, please use the following commands:\n\n\ninstall.packages(\"devtools\") \ndevtools::install_github(\"devanshagr/CrossValidation\")\nCrossValidation::cross_validate()\n\n\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"19.5.1": {"Name": "cross_validate()", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": [{"Details": "You can see that the cross-validation accuracies for the tree that was passed (accuracy_subset) are fairly high and close to our training accuracy of 84%. This means we are not overfitting. Also, observe that accuracy_subset and accuracy_all have the same values, which means that the only relevant attributes are score and participation, and adding more attributes doesn't make any difference to the tree. Finally, the values in accuracy_subset are reasonably close to each other, which means low variance. This is again desirable, as it indicates that the prediction model is stable and is more likely to perform close to these numbers on the test data. All in all, what we desire is accuracy that is close to the one on the entire training dataset and does not vary much across the different runs of cross-validation.", "PreExCode": "cross_validate <- function(df, tree, n_iter, split_ratio, method = 'class')\n{\n # training data frame df\n df <- as.data.frame(df)\n\n # mean_subset is a vector of accuracy values generated from the specified features in the tree object\n mean_subset <- c()\n\n # mean_all is a vector of accuracy values generated from all the available features in the data frame\n mean_all <- c()\n\n # control parameters for the decision tree\n contro = tree$control\n\n # the following snippet will create relations to generate decision trees\n # relation_all will create a decision tree with all the features\n # relation_subset will create a decision tree with only user-specified features in tree\n dep <- all.vars(terms(tree))[1]\n indep <- list()\n relation_all = as.formula(paste(dep, '.', sep = \"~\"))\n i <- 1\n while (i < length(all.vars(terms(tree)))) {\n indep[[i]] <- all.vars(terms(tree))[i + 1]\n i <- i + 1\n }\n b <- paste(indep, collapse = \"+\")\n relation_subset <- as.formula(paste(dep, b, sep = \"~\"))\n\n # creating train and test samples with the given split ratio\n # performing cross-validation n_iter times\n for (i in 1:n_iter) {\n sample <-\n sample.int(n = nrow(df),\n size = floor(split_ratio * nrow(df)),\n replace = F)\n train <- df[sample,]\n testing <- df[-sample,]\n type = typeof(unlist(testing[dep]))\n\n # decision tree for regression if the method specified is \"anova\"\n if (method == 'anova') {\n first.tree <-\n rpart(\n relation_subset,\n data = train,\n control = contro,\n method = 'anova'\n )\n second.tree <- rpart(relation_all, data = train, method = 'anova')\n pred1.tree <- predict(first.tree, newdata = testing)\n pred2.tree <- predict(second.tree, newdata = testing)\n mean1 <- mean((as.numeric(pred1.tree) - testing[, dep]) ^ 2)\n mean2 <- mean((as.numeric(pred2.tree) - testing[, dep]) ^ 2)\n mean_subset <- c(mean_subset, mean1)\n mean_all <- c(mean_all, mean2)\n }\n\n # decision tree for classification\n # if the method specified is not \"anova\", then this block is executed\n # if the method is not specified by the user, the default option is to perform classification\n else{\n first.tree <-\n rpart(\n relation_subset,\n data = train,\n control = contro,\n method = 'class'\n )\n second.tree <- rpart(relation_all, data = train, method = 'class')\n pred1.tree <- predict(first.tree, newdata = testing, type = 'class')\n pred2.tree <-\n predict(second.tree, newdata = testing, type = 'class')\n mean1 <-\n mean(as.character(pred1.tree) == as.character(testing[, dep]))\n mean2 <-\n mean(as.character(pred2.tree) == as.character(testing[, dep]))\n mean_subset <- c(mean_subset, mean1)\n mean_all <- c(mean_all, mean2)\n }\n }\n\n # average_accuracy_subset is the average accuracy of n_iter iterations of cross-validation with user-specified features\n # average_acuracy_all is the average accuracy of n_iter iterations of cross-validation with all the available features\n # variance_accuracy_subset is the variance of accuracy of n_iter iterations of cross-validation with user-specified features\n # variance_accuracy_all is the variance of accuracy of n_iter iterations of cross-validation with all the available features\n cross_validation_stats <-\n list(\n \"average_accuracy_subset\" = mean(mean_subset, na.rm = T),\n \"average_accuracy_all\" = mean(mean_all, na.rm = T),\n \"variance_accuracy_subset\" = var(mean_subset, na.rm = T),\n \"variance_accuracy_all\" = var(mean_all, na.rm = T)\n )\n\n # creating a data frame of accuracy_subset and accuracy_all\n # accuracy_subset contains n_iter accuracy values on cross-validation with user-specified features\n # accuracy_all contains n_iter accuracy values on cross-validation with all the available features\n cross_validation_df <-\n data.frame(accuracy_subset = mean_subset, accuracy_all = mean_all)\n return(list(cross_validation_df, cross_validation_stats))\n}", "Query": "# First lets import the rpart library\nlibrary(rpart)\n# Import dataset\nmoody<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv',stringsAsFactors = T)\n# Use of the rpart() function.\ntree <- rpart(Grade~., data = moody,method = \"class\",control = rpart.control(minsplit = 100))\ntree\n# Now lets predict the Grades of the Moody Dataset.\npred <- predict(tree, moody, type=\"class\")\nhead(pred)\n# Lets check the Training Accuracy\nmean(moody$Grade==pred)\n# Lets us the cross_validate() function.\ncross_validate(moody,tree,5,0.7)", "Result": "", "Title": "", "Type": "R", "Video": "https://drive.google.com/file/d/1zZyfbZpmnX9ZS6Ty_Kmzea0oUGGNYz0p/preview", "Width": ""}]}, "19.5.2": {"Name": "Chat about crossvalidation", "Details": "\n Lets chat about crossvalidation\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": {}}}, "parent": "19", "snippets": {}}, "19.6": {"Name": "Prediction using rpart.", "Details": "

Now that we have seen the process to create a decision tree and also plot it, we will like to use the output tree to predict the required attribute.

From the moody example, we are trying to predict the grade of students. Lets look at the predict() function to predict the outcomes.

predict(*object*,*data*,*type*,...)\n
- object: the generated tree from the rpart function.
- data: the data on which the prediction is to be performed.
- type: the type of prediction required. One of \u201cvector\u201d, \u201cprob\u201d, \u201cclass\u201d or \u201cmatrix\u201d.
\n

Now lets use the predict function to predict the grades of students using the tree generated on the Moody dataset.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"19.6.1": {"Name": "predict()", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": [{"Details": "Notice that rpart creates the prediction model, and predict applies it to a dataset. Therefore, these two functions, leading to two lines of code, will replace your entire construction of the decision vector from the freestyle prediction section earlier.", "PreExCode": "", "Query": "# First lets import the rpart library\nlibrary(rpart)\n\n# Import dataset\nmoody<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv')\n\n# Use of the rpart() function.\ntree <- rpart(Grade~., data = moody ,method = \"class\")\ntree\n\n# Now lets predict the Grades of the Moody Dataset.\npred <- predict(tree, moody, type=\"class\")\nhead(pred)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "19", "snippets": {}}, "19.7": {"Name": "Combining multiple prediction models", "Details": "How to build highly predictive models?\n\nThis is the key question that many students ask in the context of our Prediction Challenges. These challenges involve building several prediction models, each for a different dataset. Students who achieve the lowest cumulative error make it to the top of the leaderboard. It's a competition. What is the secret of building a competitive prediction model? It's not simply blindly applying machine learning library functions such as rpart(). Even with well-configured parameter values and careful cross-validation, a singular model usually won't be very competitive. Typically, the top prediction models combine human ingenuity and domain knowledge with machine learning library functions. But how do we combine different prediction models to build the \"supermodel\"? First, you need to know your data. Conduct some preliminary freestyle data exploration, create plots, and analyze the data distribution. It's possible to identify subsets of data that may behave differently and require different prediction models, whether they are handcrafted or generated through machine learning. Simply put, the approach is to combine several prediction models. These prediction models will be built using rpart(). However, the way to combine them comes from the human element involved.\n\nLet's start by demonstrating how to combine two different prediction models applied to different partitions of the dataset. We assume that the partition is already defined, usually as a result of preliminary data exploration and plotting. In the next section, we will present an elegant and generic method of combining an arbitrary number of prediction models using the rpart() function.\n\nFor now, let's assume that we have partitioned the \"moody\" dataset based on the \"Score\" attribute into two subsets: one with a Score greater than 50 and another with a Score less than or equal to 50. Furthermore, we have trained separate rpart() prediction models for each of these two partitions. Now, we want to combine these two models into a single combined model and apply it to the testing dataset, \"moodyTest\".\n\nThe following code snippet (17.7.1) illustrates how to accomplish this. Two models, model1 and model2, are trained by running rpart() on the two partitions of \"moody\" - the training dataset, based on the \"Score\" attribute. Then, we use the predict() function to apply model1 to the partition where Score > 50 and model2 to the subset defined by Score <= 50 of the testing dataset - \"moodyTest\". Finally, lines 11-14 build the decision vector that combines the predictions of models 1 and 2 into a single prediction vector for \"moodyTest\".\n\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"19.7.1": {"Name": "Combining rpart prediction models", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": [{"Details": "\r\n What does the above code snippet do?\r\n $\"ChatGPT$ \r\n", "PreExCode": "", "Query": "library(rpart) \n\n# Import dataset \n\nmoody<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv')\nmoodyTest<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv') \n\n# We need two sets here: training moody and testing moody (the full testing like Kaggle stores but does not give to students - here just for simplicty we take the same set as training and testing\nmodel1<-rpart(Grade~., data=moody[moody$Score>50,]); \nmodel2<-rpart(Grade~., data=moody[moody$Score<=50,]); \n\n\npred1 <- predict(model1, newdata=moodyTest[moodyTest$Score>50,], type=\"class\") \npred2 <- predict(model2, newdata=moodyTest[moodyTest$Score<=50,], type=\"class\") \nmyprediction<-moodyTest \n\ndecision <- rep('F',nrow(myprediction)) \n\ndecision[myprediction$Score>50] <- as.character(pred1) \n \ndecision[myprediction$Score<=50] <-as.character(pred2) \n \nmyprediction$Grade<-decision \n\nerror <- round(mean(moody$Grade!= myprediction$Grade),2)\nprint(paste(\"The error of combined model is equal to\", error))", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "19.7.2": {"Name": "Combining multiple prediction models using rpart", "Details": "We will now present an elegant approach that enables the construction of a combined model in multiple phases. Let's start with two prediction models: pred1, which is a freestyle model, and pred2, which is an rpart() model. This situation arose during our prediction challenges in the spring of 2022, where students were tasked with creating two prediction models: one using their own code (freestyle prediction) and another using the rpart() function. Interestingly, the top-performing freestyle prediction models exhibited lower error rates on the testing data compared to rpart(). The challenge then became how to merge these two models and leverage the best features from each to create a combined model that outperforms both the freestyle and rpart() models. But how can we accomplish this? In the previous section, we discussed the mechanics of combining two models by dividing the dataset into two separate partitions and applying each model to one partition. However, the question remains: how do we determine these partitions? Fortunately, we can rely on rpart() to assist us in this process.\n\nFirst, we will informally describe the proposed method and later provide a code snippet that combines two specific prediction models. To begin, we will expand the training and testing datasets by incorporating two additional derived attributes, one for each prediction model. These attributes will be named model1 and model2. The attribute model1 will store the predicted value of the target attribute (in our snippet below, it is \"Grade\" from the freestyle model. On the other hand, the attribute model2 will store the predicted value of the target attribute from rpart(). These derived attributes are calculated using the rules of the freestyle model and the rpart() model, respectively. Next, we will employ rpart() on the expanded dataset, considering the model1 and model2 attributes. This final application of rpart() will guide us in combining model1 and model2, as both attributes will contribute to the definition of nodes in the combined decision tree. This tree integrates the predictions from both models. By following this approach, we can easily combine any number of prediction models by adding multiple attributes, such as model1, model2, and so on. rpart() provides the best method for merging all these prediction models. The code snippet below demonstrates how to combine a simple score cutpoint model of \"Grade\" with the rpart-generated model, extracting the best aspects from both models. \n\nOnly the last two lines of the below snippet 17.7.2 output matter. The number of disagreements between two models and the error of combined model. \n\n\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": [{"Details": "", "PreExCode": "", "Query": "# First lets import the rpart library\nlibrary(rpart)\n#install.packages('rpart.plot')\n#library(rpart.plot)\n\n# Use of the rpart.plot() function to visualize the decision tree.\n\n\n# Import dataset\nmoody<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv')\nmoodyTest<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/moody2022.csv') \n\n#These are the same two sets - but I wanted to distinguish that one of them is training and another is testing.\n#Maybe we can then use moody and moodyTest as two partitions of moody, like in cross validation.\n\n# Use of the rpart() function.\ntree <- rpart(Grade~., data = moody ,method = \"class\")\ntree\n\n# Now let's predict the Grades of the Moody Dataset.\nmoody$model1<-rep('F',nrow(moody))\nmoody$model2<-rep('F',nrow(moody))\nmoody$model1<- predict(tree, moody, type=\"class\")\ndecision <- rep('F',nrow(moody))\ndecision[moody$Score>40] <- 'D'\ndecision[moody$Score>60] <- 'C'\ndecision[moody$Score>70] <- 'B'\ndecision[moody$Score>80] <- 'A'\nmoody$model2 <-decision\ncolnames(moody)\nmoodyTest$model1<-rep('F',nrow(moody))\nmoodyTest$model2<-rep('F',nrow(moody))\nmoodyTest$model1<- predict(tree, moodyTest, type=\"class\")\ndecision <- rep('F',nrow(moody))\ndecision[moodyTest$Score>40] <- 'D'\ndecision[moodyTest$Score>60] <- 'C'\ndecision[moodyTest$Score>70] <- 'B'\ndecision[moodyTest$Score>80] <- 'A'\nmoodyTest$model2 <-decision\ncolnames(moody)\ntree_combined<-rpart(Grade~., data=moody, method='class')\nprint(\"Notice that the combined tree has in it both model1 and model2 attributes\")\ntree_combined\ncolnames(moodyTest)\n#rpart.plot(tree_combined)\npredict(tree_combined, moodyTest, type='class')\nm<-nrow(moody[moody$model1!=moody$model2,])\nn<-nrow(moody)\nprint(paste('The number of disagreements between model 1 and model 2 is', m, 'out of', n))\nerror<-round(mean(moodyTest$Grade!=predict(tree_combined, moodyTest, type='class')),2)\nprint(paste(\"The error of combined prediction model is equal to :\", error))", "Result": "", "Title": "Combining two prediction models using rpart() for moody data set", "Type": "R", "Video": "https://drive.google.com/file/d/15DfTzEGj3sSHzQziQwqia7-qYUG2QZGJ/preview", "Width": ""}]}, "19.7.3 ": {"Name": "Building prediction models using GPT.", "Details": "GPT PROMPT: \"Build prediction model in R to predict attribute GRADE using remaining attributes for the following training data set Score, Grade. Seniority, Major, GPA + <> \"\n\nThis makes your prompt pretty large but still acceptable by Chat GPT. \n\n Interestingly, the code produced by GPT includes crossvalidation as well! However what it is lacking - is the feature engineering and more sophisticated techniques as described in previous subsections - like combining multiple prediction models. Thus, such model may have hard time in competing for accuracy with best models created by students. ", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": {}}}, "parent": "19", "snippets": {}}, "19.8": {"Name": "How to run rpart and how to submit to results", "Details": "How to run rpart and how to submit results", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": [{"Details": "HOW TO SUBMIT TO KAGGLE. HOW TO POPULATE SUBMISSION FILE?", "PreExCode": "", "Query": "library(rpart)\nhw8<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/Miderm.csv')\ncolnames(hw8)\ntree<-rpart(Party~.,data=hw8, method='class')\ntree\npred<-predict(tree, newdata = hw8, type='class')\naccuracy<-mean(pred==hw8$Party)\ncat('Accuracy of rpart prediction model')\naccuracy\n#Free style - embedded model by Professor\ndecision<-rep('Republican', nrow(hw8))\ndecision[hw8$Education=='College' & hw8$Age <40]<-'Democrat'\ndecision[hw8$Education=='PrimarySchool' & hw8$Age >=40]<-'Republican'\ndecision[hw8$NetWorth=='High']<-'Republican'\ncat('Accuracy of embedded model')\nmean(decision==hw8$Party)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "movies<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/movies2023Tr.csv')\nmoviesT<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/movies2023Test_Student.csv')\nsubmission<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/submissionMovies2023.csv')\nlibrary(rpart)\n#Assume that you build a simple model using rpart - the so called 3-line solution\ntree<-rpart(RATING~.,data=movies, method='class')\npre<-predict(tree, newdata =moviesT, type='class')\nsubmission$RATING <-pre\n#Run it pn your R studio\n#write.csv(submission, 'mysubmission.csv' , row.names = FALSE)\n#Submit mysubmission.csv to Kaggle on the link provided on late Friday - Saturday. Kaggle opens ar 7PM on Saturday", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "19.9": {"Name": "How to build prediction model using chat GPT?", "Details": "\r\n Lets talk about prediction challenge!\r\n $\"ChatGPT$ \r\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "19", "snippets": {}}}, "parent": "19", "snippets": {}}, "20": {"Name": "Predicting numerical variables: linear regression and other techniques", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"20.1": {"Name": "Introduction", "Details": "When it comes to building prediction models for numerical variables, there are several techniques that can be employed. In addition to using the \"rpart\" package, which we previously discussed for categorical variables, linear regression is also a widely used approach.\n\nLinear regression is a statistical modeling technique that aims to establish a linear relationship between a dependent variable (the target variable we want to predict) and one or more independent variables (also known as predictor variables). This method assumes a linear relationship between the variables and seeks to find the best-fitting line that minimizes the differences between the predicted values and the actual values in the training data.\n\nTo build a prediction model using linear regression, the first step is to identify the relevant independent variables that may have an impact on the dependent variable. These independent variables can be numerical or categorical, and their selection is typically based on domain knowledge, data exploration, and statistical significance. Once the relevant independent variables are identified, the linear regression model is constructed by estimating the coefficients for each variable, which represent the strength and direction of their influence on the dependent variable.\n\nThe \"lm\" function in R is commonly used for fitting linear regression models. It takes the form:\n\n\nmodel <- lm(dependent_variable ~ independent_variable1 + independent_variable2 + ..., data = training_data)\nIn this formula, \"dependent_variable\" refers to the numerical variable we want to predict, while \"independent_variable1\", \"independent_variable2\", and so on, represent the relevant independent variables. The \"data\" argument specifies the training data that will be used to build the model.\n\nOnce the linear regression model is fitted, we can use it to make predictions on new data by utilizing the \"predict\" function in R. This function takes the model and the new data as inputs and generates predictions for the dependent variable based on the learned relationship from the training data.\n\nWhile linear regression is a powerful and widely used method for predicting numerical variables, it assumes linearity and may not capture more complex relationships between variables. In such cases, decision tree-based approaches can be employed using the \"rpart\" package in R.\n\nThe \"rpart\" package provides functionality for fitting decision tree models, which can handle both numerical and categorical variables. Similar to the process described earlier, the \"rpart\" function takes the form:\n\n\nmodel <- rpart(dependent_variable ~ independent_variable1 + independent_variable2 + ..., data = training_data)\nHere, the structure is similar to linear regression, with the dependent variable and independent variables specified accordingly. The resulting decision tree model can be visualized and interpreted, allowing for a clear understanding of the decision-making process.\n\nBy leveraging the \"rpart\" package for decision tree models and the \"lm\" function for linear regression models, practitioners have flexible tools at their disposal to build prediction models for numerical variables. The choice between these methods often depends on the specific characteristics of the data, the nature of the relationship between variables, and the interpretability requirements. Experimentation and exploration are essential in determining the most suitable approach for a given prediction task.\n\nIn conclusion, when building prediction models for numerical variables, both linear regression and decision tree-based approaches, such as the \"rpart\" package, can be utilized. Linear regression establishes a linear relationship between the dependent variable and independent variables, while decision tree models offer flexibility and interpretability. By leveraging these techniques in R, practitioners can develop accurate prediction models tailored to the specific requirements of their numerical data.\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "20", "snippets": {}}, "20.2": {"Name": "Linear regression using lm() function", "Details": "

Syntax for building the regression model using the lm() function is as follows:

lm(formula, data, ...)\n
- formula: here we mention the prediction column and the other related columns(predictors) on which the prediction will be based on.
  - prediction ~ predictor1 + predictor2 + predictor3 + ...
  \n
- data: here we provide the dataset on which the linear regression model is to be trained.
\n

For more info on the lm() function visit lm()

Lets look at the example on the Moody dataset.

Table 18.1: Snippet of Moody Num Dataset
Midterm	Project	FinalExam	ClassScore
73	8	70	39.60000
61	100	20	68.20000
58	88	38	67.00000
93	41	46	52.47565
85	52	85	68.50000
97	48	19	49.10000
26	59	22	41.30000
58	62	25	50.10000
53	56	27	46.70000
66	27	17	34.80494

Imagine that we do not know the weights of midterm, project and final exam. However we have the data from the previous semesters. Can we find these weights out? The answer is yes - by using linear regression.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"20.2.1": {"Name": "Example of linear regression - guessing weights of grading elements in class", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "20", "snippets": [{"Details": "Linear regression provides us with weights of roughly 20% for midterm, 50% for the project and nearly 30% for the final exam. The MSE is very small - and returned by the code in the snippet as well.", "PreExCode": "", "Query": "moodyNUM<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/MoodyNUM.csv')\nsplit<-0.7*nrow(moodyNUM)\nsplit\nmoodyNUMTr<-moodyNUM[1:split,]\nmoodyNUMTr\nmoodyNUMTs<-moodyNUM[split:nrow(moodyNUM),]\n#We use linear regression to #find out the weights of #Midterm, Project and Final #Exam in calculation of the #final class score. Each of #them are scored out of 100 and #the final class score is also #scored out of 100 as weighted #sum of Midterm, Project and #Final Exam scores.\ntrain <- lm(ClassScore~., data=moodyNUMTr)\ntrain\npred <- predict(train,newdata=moodyNUMTs)\nmean((pred - moodyNUMTs$ClassScore)^2)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "20", "snippets": {}}, "20.3": {"Name": "Calculating the Error using mse()", "Details": "

As was the simple case in the categorical predictions of the classification models, where we could just compare the predicted categories and the actual categories, this type of direct comparison as an accuracy test won\u2019t prove useful now in our numerical predictions scenario.

We don\u2019t want to eyeball every time we predict, to find the accuracy of our predictions each row by row, so lets see a method to calculate the accuracy of our predictions, using some statistical technique.

To do this we will use the Mean Squared Error(MSE).

The MSE is a measure of the quality of an predictor/estimator
It is always non-negative
Values closer to zero are better.

\n\n

The equation to calculate the MSE is as follows:

$\"\"$ \n\nwhere n is the number of data points, Y_i are the observed value and ^Yi are the predicted values\n\n

To implement this, we will use the mse() function present in the Metrics Package, so remember to install the Metrics package and use library(Metrics) in the code for local use.

The syntax for mse() function is very simple:

mse(actual,predicted)\n
- actual: vector of the actual values of the attribute we want to predict.
- predicted: vector of the predicted values obtained using our model.
\n

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "20", "snippets": {}}, "20.4": {"Name": "Cross Validate your prediction", "Details": "Crossvalidation of linear regression models relies on the same principles as crossvalidation for rpart() and any othe rprediction model. Here we show one step corssvalidation when every time we run this code, different rnadom subset of the data frame is selected and the error is computed/.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "20", "snippets": [{"Details": "Crossvalidation of linear regression models relies on the same principles as crossvalidation for rpart() and any othe rprediction model. Here we show one step corssvalidation when every time we run this code, different rnadom subset of the data frame is selected and the error is computed/.", "PreExCode": "", "Query": "library(ModelMetrics)\n\ntrain <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/MoodyNUM.csv\")\n#scramble the train frame\nv<-sample(1:nrow(train))\nv[1:5]\ntrainScrambled<-train[v, ]\n\n#one step crossvalidation\nn <- 100\ntrainSample<-trainScrambled[nrow(trainScrambled)-n:nrow(trainScrambled), ]\ntestSample <- trainScrambled[1:n,]\n\nlm.tree <- lm(ClassScore~., data=trainSample)\nlm.tree\n\npred <- predict(lm.tree,newdata=testSample)\npred\n\nmse(testSample$ClassScore,pred)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "20.5": {"Name": "Submission with lm", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "20", "snippets": [{"Details": "", "PreExCode": "", "Query": "library(rpart)\ntest<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/MoodyNUM_test.csv')\nsubmission<-read.csv('https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/M2022submission.csv')\ntrain <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/MoodyNUM.csv\")\n\ntree <- lm(ClassScore~., data=train)\ntree\n\nprediction <- predict(tree, newdata=test)\n\n#Now make your submission file - it will have the IDs and now the predicted grades\nsubmission$Grade<-prediction \n\n# use write.csv(submission, 'submission.csv', row.names=FALSE) to store submission as csv file on your machine and subsequently submit it on Kaggle", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "20.6": {"Name": "Using rpart for Numerical Variables - compete with lm", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "20", "snippets": [{"Details": "", "PreExCode": "", "Query": "library(rpart)\ntrain <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/MoodyNUM.csv\")\ncolnames(train)\ntree<-rpart(Midterm~.,data=train, method='anova')\ntree\npred<-predict(tree, newdata=train)\nmean((pred - train$ClassScore)^2)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "20.8": {"Name": "Feature Engineering and Prediction Models", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "20", "snippets": [{"Details": "", "PreExCode": "", "Query": "MoodyNUM <- read.csv(\"https://raw.githubusercontent.com/dev7796/data101_tutorial/main/files/dataset/MoodyNUM.csv\")\nlibrary(rpart)\nMoodyN<-subset(MoodyNUM, select = -ClassScore)\ncat('After removing ClassScore column')\ncolnames(MoodyN)\n#Define new feature - to help rpart!\nMoodyN$Delta<-(MoodyNUM$FinalExam-MoodyNUM$Midterm)\n#Define Grade as function of this new features. This is what DataMaker does normally when I prepare the challenges\nMoodyN$Grade<-'F'\nMoodyN[MoodyN$Delta>20,]$Grade<-'P'\n#Imagine that our testing data included generous extra credit for Midterm, Project and FinalExam!\nMoodyTest<-MoodyN\n#Extra Credit is random valie for max of 30 to max of 50!\nExtraCredit <- runif(nrow(MoodyTest), min = 0, max = 30)\nMoodyTest$Midterm<-MoodyN$Midterm +ExtraCredit\nExtraCredit <- runif(nrow(MoodyTest), min = 0, max = 45)\nMoodyTest$Project<-MoodyN$Project +ExtraCredit\nExtraCredit <- runif(nrow(MoodyTest), min = 0, max = 50)\nMoodyTest$FinalExam<-MoodyNUM$FinalExam+ExtraCredit\nMoodyTest[1:10,]\n#If we trained without additional Delta Feature - how would our prediction model handle the MoodyTest data with extra credits? Notice ranges are now different \ntree<-rpart(Grade~Midterm+FinalExam+Project, data=MoodyN,method='class')\ntree\npre<-predict(tree, newdata=MoodyTest, type='class')\ncat('The accuracy of the prediction model without Delta attribute')\nmean(pre==MoodyTest$Grade)\ntree<-rpart(Grade~Midterm+FinalExam+Project+Delta, data=MoodyN,method='class')\ntree\npre<-predict(tree, newdata=MoodyTest, type='class')\ncat('The accuracy of the prediction model withDelta attribute')\nmean(pre==MoodyTest$Grade)\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "20.9": {"Name": "Combining linear regression models", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "20", "snippets": [{"Details": "This snippet above was just to test your combined model on incomeTrain. Now time to build submission file. For this you use incomeTest\n\nHOW TO BUILD SUBMISSION FILE TO KAGGLE", "PreExCode": "", "Query": "incomeTrain<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/incomeTrain2023.csv')\n\nmodel1<-lm(Salary~Tuition+GPA+LinkedIN, data=incomeTrain[incomeTrain$College_location=='WestCoast',]) \n\nmodel2<-lm(Salary~Tuition+GPA+LinkedIN, data=incomeTrain[incomeTrain$College_location!='WestCoast',]) \n\n\npred1 <- predict(model1, newdata=incomeTrain[incomeTrain$College_location=='WestCoast',])\npred2 <- predict(model2, newdata=incomeTrain[incomeTrain$College_location!='WestCoast',])\ndecision <- rep(0,nrow(incomeTrain)) \ndecision[incomeTrain$College_location=='WestCoast'] <-pred1\ndecision[incomeTrain$College_location!='WestCoast'] <-pred2\nmean((decision- incomeTrain$Salary)^2)\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "incomeTest<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/IncomeTest2023_Kaggle.csv')\nincomeTrain<-read.csv('https://raw.githubusercontent.com/paramshah4/data101_tutorial/main/files/dataset/incomeTrain2023.csv')\n\nmodel1<-lm(Salary~Tuition+GPA+LinkedIN, data=incomeTrain[incomeTrain$College_location=='WestCoast',]) \n\nmodel2<-lm(Salary~Tuition+GPA+LinkedIN, data=incomeTrain[incomeTrain$College_location!='WestCoast',]) \n\n\npred1 <- predict(model1, newdata=incomeTest[incomeTest$College_location=='WestCoast',])\npred2 <- predict(model2, newdata=incomeTest[incomeTest$College_location!='WestCoast',])\ndecision <- rep(0,nrow(incomeTest)) \ndecision[incomeTest$College_location=='WestCoast'] <-pred1\ndecision[incomeTest$College_location!='WestCoast'] <-pred2\nids<-c(1:nrow(incomeTest))\nsubmission<-data.frame(ID=ids)\nsubmission$Salary<-decision\nsubmission[1:10,]", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "20.10": {"Name": "Prediction Challenges - Guide", "Details": "Prediction Challenges Guide\n\nIn this short guide, we'll utilize two core functions: rpart() and lm(). Additionally, you can explore other R-library functions like random forest, SVM, naive Bayes, and even neural networks (refer to section 17.1 and subsequent subsections for insights into rpart and rpart.control).\n\nThe basic code is straightforward and comprises just a few lines. We initiate the prediction model on the training dataset using:\n\n\ntree <- rpart()\n\nWe may also employ the crossvalidate() function for cross-validation to bolster confidence in our prediction model's performance on testing data. Please refer to the subsection on cross-validation in chapter 17. Once the prediction model is finalized, we proceed with the testing dataset to generate our prediction vector and create a submission file for canvas submission.\n\nTo make predictions using our model, we utilize:\n\n\npre <- predict(tree, ...)\nsubmission$TargetAttribute <- pre\n\nThis fills the second column of the submission file (the first being ID). You can opt for the straightforward use of rpart or lm (for numerical variable prediction, though rpart can also handle this) solely on existing attributes of your training data frame. However, this may not yield top scores in prediction challenges. Feature engineering is crucial, involving the addition of derived attributes through data plotting or eyeballing and plotting. This requires human intervention to enhance the usage of R libraries.\n\nNone of the R library functions automatically create derived attributes; it's your responsibility. After adding such variables, rpart() optimizes the prediction model by integrating this new feature with existing ones to maximize prediction accuracy.\n\nTherefore, feature engineering is pivotal for competitiveness in prediction challenges. Combining multiple prediction models, as outlined in section 17.7, is also critical. This reflects practical scenarios where one model isn't sufficient, necessitating data partitioning into subsets, each covered by different prediction models, some of which may not use R library functions at all.\n\nSection 17.7.1 illustrates a simple example of combining two rpart() models, while section 17.7.2 demonstrates how to combine any number of models using rpart to define their integration. This is achieved by creating additional attributes m1, m2, etc., for each prediction model, with each attribute returning predictions of the target attribute from a prediction model.", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "20", "snippets": {}}}, "parent": "20", "snippets": {}}, "22": {"Name": "Power Law Distribution", "Details": "\n

Power law distributions have been known for a while but came to prominence with the advent of the world wide web where power law occurs quite frequently.

Naseem Taleeb in his book Black Swan has termed Power Law as Extremistan and Normal (Gaussian) distribution as Mediocristan. The Mandelbrotian and the Gaussian. from Mandelbrodt and his pioneering work in the field of fractals and power distribution.

Figure 1 displays power law which reflects acute concentration and so-called long tail characteristics to virtual quantities such as web traffic, sales data, wealth data, youtube views, word frequency etc. Gausian law (Bell curve) usually displays more physical quantities such as weight, height etc. The latter concentrate heavily around mean and decline quickly (so called Gausian decline). Long tail of power law distributions declines much slower and as we will see helps to explain much more wild deviation of wealth, sales etc.

$\"\"$ \n

Figure 19.1 Power Law

$\"\"$ \n

Figure 19.2 Bell curve \u2013 Gaussian Distribution

For example, take the height of men and women. Average height is 1.67 meters. It is distributed normally around this mean height. Assume that standard deviation of height is 10 cm.

Consider odds of being

\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n

10 centimeters taller than the average (i.e., taller than 1.77 m,	1 in 6.3
20 centimeters taller than the average (i.e., taller than 1.87 m, or 6 feet 2	1 in 44
30 centimeters taller than the average (i.e., taller than 1.97 m, or 6 feet 6): 1 in 740	1 in 740
40 centimeters taller than the average (i.e., taller than 2.07 m, or 6 feet	1 in 32000
50 centimeters taller than the average (i.e., taller than 2.17 m, or 7 feet	1 in 3,500,000
\n 60 centimeters taller than the average (i.e., taller than 2.27 m, or 7 feet 70 centimeters taller than the average (i.e., taller than 2.37 m, or 8 feet 9\n	1 in 1,000,000,0000
100 centimeters taller than the average (i.e., taller than 2.67 m, or 8 feet 9	1 in 130,000,000,000,000,000,000,000

\n $\"\"$ \n

Thus if one doubles x (the threshold) then the probability of exceeding x goes down $\"\"$ times. If exponent c=1, then x doubles the P(X>2x) = \u00bd *P(X>x)

The following snippet illustrates the radical differences between power law distribution and the normal distribution.

We first generate the power law distribution with the exponent -1. To this aim we first generate uniform distribution (runif() function does this) of real numbers between 0.001 and 1. We are also rounding these numbers up to 3 digits using round(). Power distribution is generated by one line of code

\n $\"\"$ \n\n

We generate 10000 data points which range from 1000 down to 1. The mean value of the power law distribution is 7. Now to compare the power law distribution and normal distributions we use rnorm() to create 10000 data points distributed normally around the same mean =7 and with standard deviation of 0.1. We show histograms of both distributions to illustrate the different shapes, long tail shape of the power law distribution and the characteristic Bell curve of the Gaussian distribution. Then we follow by illustrating the main point: the rapid descending of the Gaussian distribution. Just change the values in two last statements of the snippet: length(normal[normal>7.1]) and length(power[power>7.1]). Notice that the number of values which are larger than 7.1 (one standard deviation from the mean) is roughly the same for both distributions: it is around 1500. But now observe what happens when we move x higher in length(normal[normal>x]), and length(power[power>x]). The value of x=7.3 is the largest value when there are still 19 values left in length(normal[normal>x]), when we move x to 7.4, this number is already zero. Now compare it with the power law distribution. For x =7.3 we get 1319 values.

Try x=8, 16,32, 64 and there are still a substantial number of values left. We get 140 values for power>64. Notice that each time we double x, we halve the number of values above x.

length(power[power>8])

length(power[power>16])

length(power[power>32])

length(power[power>64]) \u2026 length(power[power>256]) still gets us 27 values (it is stochastic, so when you run it values may be slightly different.

You are encouraged to experiment with the exponent (-1) by increasing and decreasing it, as well as with standard deviation of the normal distribution (increase it and decrease it). Just remember to keep the mean of normal distribution the same as the mean of power law distribution for easier comparison. The rapid descent of normal distribution is clearly visible, when all values are between 6.61 and 7.35. while power law distribution values range from 1000 to 1.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"22.2": {"Name": "Power Law Distribution vs Normal Distribution snippet", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "22", "snippets": [{"Details": "

In general the probability of exceeding the mean by number of sigmas (standard deviations) is

0 sigmas: 1 in 2 times

1 sigmas: 1 in 6.3 times

2 sigmas: 1 in 44 times

3 sigmas: 1 in 740 times

4 sigmas: 1 in 32,000 times

5 sigmas: 1 in 3,500,000 times

6 sigmas: 1 in 1,000,000,000 times

7 sigmas: 1 in 780,000,000,000 times

8 sigmas: 1 in 1,600,000,000,000,000 times

9 sigmas: 1 in 8,900,000,000,000,000,000 times

10 sigmas: 1 in 130,000,000,000,000,000,000,000 times

and, skipping a bit:

20 sigmas: 1 in 36,000,000,000,000,000,000,000,000,000,000,000 ,000,000,000,000,000,000,000,000,000,000,000,0 00,000,000,000,000,000, 000 times

Soon, after about 22 sigmas, one hits a \u201cgoogol\u201d, which is 1 with 100 zeros behind it

We find that the odds of encountering a millionaire in Europe are as follows:

Richer than 1 million: 1 in 62.5

Richer than 2 million: 1 in 250

Richer than 4 million: 1 in 1,000

Richer than 8 million: 1 in 4,000

Richer than 16 million: 1 in 16,000

Richer than 32 million: 1 in 64,000

Richer than 320 million: 1 in 6,400,000

One of the most misunderstood aspects of a Gaussian is its fragility and vulnerability in the estimation of tail events. The odds of a 4 sigma move are twice that of a 4.15 sigma. The odds of a 20 sigma are a trillion times higher than those of a 21 sigma! It means that a small measurement error of the sigma will lead to a massive underestimation of the probability. We can be a trillion times wrong about some events

If wealth was distributed according to Gaussian

If Wealth Distribution was following Gaussian Law

People with a net worth higher than \u20ac1 million: 1 in 63

Higher than \u20ac2 million: 1 in 127,000

Higher than \u20ac3 million: 1 in 14,000,000,000

Higher than \u20ac4 million: 1 in 886,000,000,000,000,000

Higher than \u20ac8 million: 1 in 16,000,000,000,000,000,000,000,000,000,000,000

Very egalitarian!

Other differences between Mediocristan vs Extremistan

Take a random sample of any two people from the U.S. population who jointly earn $1 million per annum. What is the most likely breakdown of their respective incomes?

In Mediocristan, the most likely combination is half a million each.

In Extremistan, it would be between $50,000 and $950,000.

For two authors to have sold a total of a million copies of their books, the most likely combination is 993,000 copies sold for one and 7,000 for the other.

This is far more likely than that the books each sold 500,000 copies.

For any large total, the breakdown will be more and more asymmetric.

Similarly for Height of two people if Total height of two people is fourteen feet, the most likely breakdown is seven feet each, not two feet and twelve feet, not even eight feet and six feet!

Persons taller than eight feet are so rare that such a combination would be impossible

", "PreExCode": "", "Query": "v<-round(runif(10000, min = 0.001, max = 1), 3)\nv<-v[order(v)]\npower<-v^(-1)\ncat('mean of power law distribution')\nmean(power)\ncat('max of power law distribution')\nmax(power)\ncat('min of power law distribution')\nmin(power)\nnormal<-rnorm(10000,mean=7, sd=0.1)\nhist(normal,50)\nhist(power,50)\ncat('How many values are above 7.3 according to the normal distribution')\nlength(normal[normal>7.3])\ncat('How many values are above 7.25 according to the power distribution')\nlength(power[power>7.3])", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}}, "parent": "22", "snippets": {}}, "21": {"Name": "Boosting - technique to improve prediction models. ", "Details": " \n Lets chat about Boosting\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "21", "snippets": [{"Details": "", "PreExCode": "", "Query": "# Boosting Step-by-Step Example in R\n\n# Sample data\nx <- c(1, 2, 3, 4, 5)\ny <- c(1.5, 3.8, 7.2, 8.5, 11.3)\n\n# Learning rate\nlr <- 0.1\n\n# Step 1: Initial prediction (mean of y)\ny_pred_0 <- rep(mean(y), length(y))\nresidual_1 <- y - y_pred_0\n\ncat(\"\\n--- Step 1: Initial Prediction ---\\n\")\nprint(data.frame(x, y, y_pred_0, residual_1))\n\n# Step 2: Fit model to residuals\nmodel_1 <- lm(residual_1 ~ x)\nresidual_pred_1 <- predict(model_1, data.frame(x = x))\ny_pred_1 <- y_pred_0 + lr * residual_pred_1\nresidual_2 <- y - y_pred_1\n\ncat(\"\\n--- Step 2: After First Weak Learner ---\\n\")\nprint(data.frame(x, y, y_pred_1, residual_2))\n\n# Step 3: Fit second model to new residuals\nmodel_2 <- lm(residual_2 ~ x)\nresidual_pred_2 <- predict(model_2, data.frame(x = x))\ny_pred_2 <- y_pred_1 + lr * residual_pred_2\nresidual_3 <- y - y_pred_2\n\ncat(\"\\n--- Step 3: After Second Weak Learner ---\\n\")", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "23": {"Name": "Simpson Paradox", "Details": "\n GPT, Can you help me with Simpson Paradox?\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "23", "snippets": {}}, "24": {"Name": "How can data fool us? Let GPT rap about it!", "Details": "Don't let data fool ya\n\nGPT\n\n(Verse 1)\nYo, gather 'round, let me drop some knowledge in this rhyme,\nAbout data and stats, how they can mess with our minds,\nThey promise data 101 to keep us in the know,\nBut sometimes that data's twisted, gotta watch where you go.\n\n(Hook)\nHow can data fool us? It's a tricky game,\nNumbers on the surface might not be the same,\nSimpson Paradox, it's a mind-bending twist,\nProsecutorial, Ecological, things on the list.\n\n(Verse 2)\nBeware of randomness, p-values and all that,\nHypothesis testing, don't just wear a hat,\nBayesian reasoning, be an optimist at heart,\nPrior odds, stay smart, don't let illusions start.\n\nExtreme results, they might look so grand,\nBut it's the law of small numbers, take a stand,\nAvailability, high frequency in our view,\nFalse events seem common, gotta see what's true.\n\n(Narrative fallacy) False patterns, don't fall for the trap,\n\"Stocks down for costs,\" don't get caught in the crap,\nNow let's dive in deeper, to the paradoxes we see,\nSimpson's up first, let me break it down, you'll agree.\n\n(Chorus)\nSimpson's paradox got us twisted and turned,\nNumbers that we thought, now have to be relearned,\nAggregating data, trends might just flip,\nWhat we thought was the truth, might just start to slip.\n\n(Verse 3)\nTwo ballers on the court, Aaron and Barry's the name,\nFTP's their game, stats ain't the same,\nBarry's beating Aaron, in both years that we see,\nBut aggregated up, the truth becomes a mystery.\n\n(Pre-Chorus)\nTable 16.1, it's all laid out,\nNumbers in the seasons, no doubt,\nBut Table 16.2, it flips the script,\nAttempts and percentages, man, it's hard to grip.\n\n(Chorus)\nSimpson's paradox got us twisted and turned,\nNumbers that we thought, now have to be relearned,\nAggregating data, trends might just flip,\nWhat we thought was the truth, might just start to slip.\n\n(Verse 4)\nMore seasons, more confusion, this paradox is a ride,\nBarry beating Aaron, yet Aaron's got the pride,\nThe absolute attempts, that's the missing link,\nFlip the numbers 'round, now it's Aaron who's in sync.\n\n(Pre-Chorus)\nSo keep in mind, the truth's not what it seems,\nData's like a puzzle, full of tricks and dreams,\nSimpson's paradox, it's a lesson we learn,\nTables and numbers, gotta discern.\n\n(Chorus)\nSimpson's paradox got us twisted and turned,\nNumbers that we thought, now have to be relearned,\nAggregating data, trends might just flip,\nWhat we thought was the truth, might just start to slip.\n\n(Outro)\nBut don't be fooled, my friends, stay sharp and wise,\nIn this data-driven world, where deception tries,\nFrom Simpson to Ecological, they're all in the mix,\nData's a puzzle, so let's solve it and fix.\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"24.1": {"Name": "Introduction", "Details": "

How not to be fooled by data? In the description of this class we promised that data 101 will teach you this. Have we? I hope so. Please use our question roulette to test yourself. In this section we discuss Simpson Paradox, Prosecutorial and Ecological Fallacies. Before we proceed with the paradoxes let us summarize what we have learned so far:

\r\n

Beware of randomness (Hypothesis testing, p-values, Multiple hypothesis testing)
Be an optimist - use Bayesian reasoning. Remember about prior odds first!
Beware of extreme results - Apply law of small numbers
Remember we process information following availability - we assign high frequency falsely to events which are just talked about very often
Narrative fallacy - do not find false patterns - \u201cStocks went down due to concerns about rising cost of living\u201d

\r\n

Here we will discuss the Simpson paradox as well as Prosecutorial and Ecological fallacies. Let us start with the Simpson paradox. Here is a simple example of two basketball players, Aaron and Barry

\r\n

The Table 22.1 shows Aaron\u2019s and Barry\u2019s free throw point averages (FTP) over the 2021 and 2022 season respectively. Clearly for both seasons Barry beats Aaron in terms of FTP, in 2021 by 90% to 80% and in 2022 by 70% to 65%. Can Aaron still beat Barry over both seasons - that is get a higher FTP over the sum of two seasons, 2021+2022? First answer which comes to mind is absolutely not. How can Aaaron beat Barry over 2021+2022 when Barry beats him in each of the two seasons?

\r\n

Table 22.1

\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n

season/player	Aaron	Barry
2021 season	80%	90%
2022 season	65%	70%

\r\n

But table 22.2 explains that it is quite possible that Aaron can beat Barry over 2021+ 2022. Indeed, since we do not know the absolute number of attempts at free throws, we can easily pick any number of attempts for each of them in any of the two seasons.

\r\n

Indeed - here is the proof that Aaron can still beat Barry. If Barry made 100 attempts in 2021 and 20 attempts in 2022, while Barry made only 20 attempts in 2021 and 100 attempts in 2022, Aaron\u2019s overall FRP for both 2021+2022 will be higher than Barry\u2019s. And this is Simposon\u2019s paradox.

\r\n

Table 22.2

\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n

season/player	Aaron	Barry
2021 season	80% out of 100	90% out of 20
2022 season	65% out of 20	70% out of 100

\r\n

Indeed Aaron\u2019s FTP over 2021+2022 is

\r\n

$\"\"$

\r\n

which is larger than Barry\u2019s

\r\n

$\"\"$

\r\n

More generally, trends in subsets of data may reverse themselves after aggregation.

\r\n

In fact we can have any number of seasons and have Barry beat Aaron in FTP in each and every season and Aaron still wins with better FTP over all seasons. This is again simply because we do not know how many attempts each player made each season. This applies to many real world situations such as graduate admissions for example (the famous Berkeley admission bias case). There women may have a higher chance to be admitted than men in each single academic department and nevertheless, men beat women in overall acceptance ratio. This is again hard to comprehend at first but it is due to the fact that the absolute number of female and male applicants may be different for each department.

\r\n

Is such reversal always possible?

\r\n

Let\u2019s look at the table below:

\r\n

Table 22.3

\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n

season/player	Aaron	Barry
2021 season	65%	90%
2022 season	60%	70%

\r\n

In this case the Simpson paradox is not possible. Why? Because Aaron\u2019s highest FTP (65% in 2021 season is lower than Barry\u2019s lowest FTP in 2022). You can easily see that no matter what the absolute numbers of attempts in each season, Aaron can never beat Barry for 2021+2022. Thus the Simpson paradox was possible in this simple case only because Aaron\u2019s highest FTP was higher than Barry\u2019s lowest FTP.

\r\n

One also has to be careful with the Simpson paradox and not apply it to situations when both groups /individuals have the same absolute number of \u201cattempts\u201d. For example, the Simpson paradox is not possible for students and their individual scores on homework\u2019s and exams. If Barry scores higher than Aaron on each homework and on each exam then Barry will always have a higher score overall than Aaron. There is no Simpsonian trend reversal. Every homework and every exam counts the same for all students. This is as if players always made the same number of free throw attempts.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {"24.1.1": {"Name": "Ecological Paradox", "Details": "

Ecological paradox is kind of the reverse of the Simpson paradox. Let\u2019s assume that we consider net worth for each member of groups A and B. Even if average net worth of group A is higher than average net worth of group B, it may be possible that random individual member of the group B has higher net worth than random individual member of the group A. Thus the order of aggregates may be reversed when we look at the level of individuals.

\r\n

For example as table 22.3 illustrates, the average net worth of Group A dominates the average net worth of Group B due to the presence of one wealthy individual. However for 90% of pairs of individuals, group B members are more wealthy than Group A members.

\r\n

Table 22.3

\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n

Group A	Group B
$10,000,000	$210,000
$100,000	$290,000
$120,000	$220,000
$80,000	$210,000
$60,000	$270,000
$160,000	$210,000
$110,000	$240,000
$100,000	$210,000
$200,000	$240,000

\r\n

For example, it is well known that Democrats win the richest states, while (until recently), the richest individuals vote republican. How is it possible? Explanation is simple. Everyone\u2019s vote counts the same and there are few very rich people. Very rich people may contribute more to the average wealth of the state (due to their extreme wealth), but there are just very few of them.

\r\n

Do not be fooled by aggregates!

\r\n

Let\u2019s assume that a Democrat wins 70% of the vote and a Republican wins 30% of the vote in some state. Is it possible that, nevertheless, the republican candidate wins all 19 counties out of 20 in the state? This actually happens a lot when the population is heavily concentrated in a heavily populated urban county which has the vast majority of voters living there.

", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "24", "snippets": {}}, "24.1.2": {"Name": "Conclusion and summary", "Details": "In conclusion, this section has explored two important statistical phenomena: the Simpson paradox and the ecological fallacy. The Simpson paradox is characterized by a reversal of trends when data subsets are aggregated, challenging our initial assumptions based on individual observations. It was exemplified through the case of Aaron and Barry's free throw point averages in basketball, where Barry outperformed Aaron in both seasons individually, yet Aaron could still have a higher overall average due to varying numbers of attempts.\n\nOn the other hand, the ecological fallacy presents a scenario where the order of aggregates is reversed when examining individual-level data within different groups. This can lead to misleading interpretations when relying solely on average values. The example of average net worth in groups A and B highlighted how even though group A had a higher average, individuals from group B could possess higher net worth in the majority of cases.\n\nThese phenomena emphasize the importance of scrutinizing statistical relationships beyond the surface level. Simply aggregating data or making assumptions based on averages can mask underlying complexities. The examples presented caution against drawing conclusions without considering individual-level characteristics, such as the number of attempts or the presence of influential outliers.\n\nIt is crucial to exercise caution when interpreting statistical patterns and to delve deeper into the underlying factors at play. The Simpson paradox and ecological fallacy serve as reminders that context and individual-level analysis are vital for a comprehensive understanding of data. By embracing a more nuanced approach, we can avoid oversimplification and gain valuable insights into complex statistical relationships.\n\n\n\n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "24", "snippets": {}}}, "parent": "24", "snippets": {}}}, "parent": "24", "snippets": {}}, "25": {"Name": "Monte Carlo", "Details": "\n Let's chat about Monte Carlo?\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "25", "snippets": [{"Details": "", "PreExCode": "", "Query": "set.seed(123) # for reproducibility\ntrue_p <- 0.52 # true proportion in the population\nn <- 1000 # sample size\nB <- 10000 # number of simulations\n\n# Simulate B surveys\nsample_props <- rbinom(B, size = n, prob = true_p) / n\n\n# Compute 95% confidence intervals for each sample\nz <- 1.96 # z-score for 95% confidence\nmargins <- z * sqrt(sample_props * (1 - sample_props) / n)\nlower <- sample_props - margins\nupper <- sample_props + margins\n\n# Check how many intervals contain the true proportion\ncovered <- (lower <= true_p) & (upper >= true_p)\n\n# Results\nmean(covered) # estimated confidence level (should be close to 0.95)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "set.seed(123)\r\n\r\n# True weights (hidden from model)\r\nw1 <- 0.3 # Midterm 1\r\nw2 <- 0.3 # Midterm 2\r\nw3 <- 0.4 # Final\r\n\r\nB <- 1000 # Number of simulations\r\nn <- 100 # Number of students per simulation\r\ncoverage <- numeric(B) # Store if true values fall in prediction interval\r\n\r\nfor (b in 1:B) {\r\n # Step 1: Generate synthetic data\r\n mid1 <- runif(n, 50, 100)\r\n mid2 <- runif(n, 50, 100)\r\n final <- runif(n, 50, 100)\r\n \r\n # True total + noise\r\n total <- w1 * mid1 + w2 * mid2 + w3 * final + rnorm(n, 0, 5)\r\n \r\n df <- data.frame(mid1, mid2, final, total)\r\n \r\n # Step 2: Fit linear model (we don\u2019t know true weights)\r\n model <- lm(total ~ mid1 + mid2 + final, data = df)\r\n \r\n # Step 3: Predict with interval\r\n preds <- predict(model, newdata = df, interval = \"prediction\", level = 0.95)\r\n \r\n # Step 4: Check how many true totals are within predicted interval\r\n inside <- (total >= preds[, \"lwr\"]) & (total <= preds[, \"upr\"])\r\n coverage[b] <- mean(inside) # Proportion of covered cases\r\n}\r\n\r\n# Result: average coverage over B runs\r\nmean(coverage)\r\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "set.seed(123)\r\n\r\n# True weights (hidden from model)\r\nw1 <- 0.3 # Midterm 1\r\nw2 <- 0.3 # Midterm 2\r\nw3 <- 0.4 # Final\r\n\r\nB <- 1000 # Number of simulations\r\nn <- 100 # Number of students per simulation\r\ncoverage <- numeric(B) # Store if true values fall in prediction interval\r\n\r\nfor (b in 1:B) {\r\n # Step 1: Generate synthetic data\r\n mid1 <- runif(n, 50, 100)\r\n mid2 <- runif(n, 50, 100)\r\n final <- runif(n, 50, 100)\r\n \r\n # True total + noise\r\n total <- w1 * mid1 + w2 * mid2 + w3 * final + rnorm(n, 0, 5)\r\n \r\n df <- data.frame(mid1, mid2, final, total)\r\n \r\n # Step 2: Fit linear model (we don\u2019t know true weights)\r\n model <- lm(total ~ mid1 + mid2 + final, data = df)\r\n \r\n # Step 3: Predict with interval\r\n preds <- predict(model, newdata = df, interval = \"prediction\", level = 0.95)\r\n \r\n # Step 4: Check how many true totals are within predicted interval\r\n inside <- (total >= preds[, \"lwr\"]) & (total <= preds[, \"upr\"])\r\n coverage[b] <- mean(inside) # Proportion of covered cases\r\n}\r\n\r\n# Result: average coverage over B runs\r\nmean(coverage)\r\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "set.seed(42) # for reproducibility\n\n# Generate synthetic data\nn <- 100\nmid1 <- runif(n, 50, 100)\nmid2 <- runif(n, 50, 100)\nfinal <- runif(n, 50, 100)\n\n# True weights\nw1 <- 0.3; w2 <- 0.3; w3 <- 0.4\ntrue_total <- w1 * mid1 + w2 * mid2 + w3 * final + rnorm(n, 0, 5)\n\ndf <- data.frame(mid1, mid2, final, true_total)\n\n# Fit regression model\nmodel <- lm(true_total ~ mid1 + mid2 + final, data = df)\n\n# Get predictions and intervals\npreds <- predict(model, newdata = df, interval = \"prediction\", level = 0.95)\n\n# Prepare data for plotting\ndf$predicted <- preds[, \"fit\"]\ndf$lwr <- preds[, \"lwr\"]\ndf$upr <- preds[, \"upr\"]\n\n# Plot\nplot(df$predicted, df$true_total,\n pch = 19, col = \"blue\", xlab = \"Predicted Total\", ylab = \"True Total\",\n main = \"True vs Predicted Total with 95% Prediction Intervals\")\narrows(df$predicted, df$lwr, df$predicted, df$upr,\n angle = 90, code = 3, length = 0.05, col = \"gray\")\nabline(0, 1, col = \"red\", lwd = 2, lty = 2)\n", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}, {"Details": "", "PreExCode": "", "Query": "set.seed(123) # for reproducibility\ntrue_p <- 0.52 # true proportion in the population\nn <- 1000 # sample size\nB <- 10000 # number of simulations\n\n# Simulate B surveys\nsample_props <- rbinom(B, size = n, prob = true_p) / n\n\n# Compute 95% confidence intervals for each sample\nz <- 1.96 # z-score for 95% confidence\nmargins <- z * sqrt(sample_props * (1 - sample_props) / n)\nlower <- sample_props - margins\nupper <- sample_props + margins\n\n# Check how many intervals contain the true proportion\ncovered <- (lower <= true_p) & (upper >= true_p)\n\n# Results\nmean(covered) # estimated confidence level (should be close to 0.95)", "Result": "", "Title": "", "Type": "R", "Video": "", "Width": ""}]}, "26": {"Name": "Association Rules (Agrawal, Imielinski, Swami)", "Details": "\n Association Rules, frequent itemsets, Apriori Algoirthm?\n $\"ChatGPT$ \n\n\nExample of transactions (only 0's and 1's)\n\nMilk Butter Bread Tea Cofee\n1 0 1 1 0\n0 1 1 0 0\n1 1 0 1 1\n1 1 0 1 0 ", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "26", "snippets": {}}, "27": {"Name": "Boundless Analytics", "Details": "\n Boundless Analytics\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "27", "snippets": {}}, "28": {"Name": "How to produce submission file to submit to Kaggle", "Details": "", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "28", "snippets": {}}, "29": {"Name": "\ud83d\udcac Let's Chat about hypothesis testing", "Details": "\nGPT are you there? Can we review hypothesis testing?\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "29", "snippets": {}}, "30": {"Name": "\ud83d\udcac How do I start, GPT. I have a nice data set about\nparties, what can I find out using data science?", "Details": "\nGPT, I have a data set. How do I start?\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": "", "Ignore": "", "Video": "", "child": {}, "parent": "30", "snippets": {}}}