Data Science For Dummies®, 2nd Edition

Devices

Overviewing the Vocabulary and Technologies

Digging into the Data Science Approaches

Advancing Artificial Intelligence Innovation

Part 3: Creating Data Visualizations That Clearly Communicate Meaning

Chapter 9: Following the Principles of Data Visualization Design

Data Visualizations: The Big Three

Designing to Meet the Needs of Your Target Audience

Picking the Most Appropriate Design Style

Choosing How to Add Context

Selecting the Appropriate Data Graphic Type

Choosing a Data Graphic

Chapter 10: Using D3.js for Data Visualization

Introducing the D3.js Library

Knowing When to Use D3.js (and When Not To)

Getting Started in D3.js

Implementing More Advanced Concepts and Practices in D3.js

Chapter 11: Web-Based Applications for Visualization Design

Designing Data Visualizations for Collaboration

Visualizing Spatial Data with Online Geographic Tools

Visualizing with Open Source: Web-Based Data Visualization Platforms

Knowing When to Stick with Infographics

Chapter 12: Exploring Best Practices in Dashboard Design

Focusing on the Audience

Starting with the Big Picture

Getting the Details Right

Testing Your Design

Chapter 13: Making Maps from Spatial Data

Getting into the Basics of GIS

Analyzing Spatial Data

Getting Started with Open-Source QGIS

Part 4: Computing for Data Science

Chapter 14: Using Python for Data Science

Sorting Out the Python Data Types

Putting Loops to Good Use in Python

Having Fun with Functions

Keeping Cool with Classes

Checking Out Some Useful Python Libraries

Analyzing Data with Python — an Exercise

Chapter 15: Using Open Source R for Data Science

R’s Basic Vocabulary

Delving into Functions and Operators

Iterating in R

Observing How Objects Work

Sorting Out Popular Statistical Analysis Packages

Examining Packages for Visualizing, Mapping, and Graphing in R

Chapter 16: Using SQL in Data Science

Getting a Handle on Relational Databases and SQL

Investing Some Effort into Database Design

Integrating SQL, R, Python, and Excel into Your Data Science Strategy

Narrowing the Focus with SQL Functions

Chapter 17: Doing Data Science with Excel and Knime

Making Life Easier with Excel

Using KNIME for Advanced Data Analytics

Part 5: Applying Domain Expertise to Solve Real-World Problems Using

Data Science

Chapter 18: Data Science in Journalism: Nailing Down the Five Ws

(and an H)

Who Is the Audience?

What: Getting Directly to the Point

Bringing Data Journalism to Life: The Black Budget

When Did It Happen?

Where Does the Story Matter?

Why the Story Matters

How to Develop, Tell, and Present the Story

Collecting Data for Your Story

Finding and Telling Your Data’s Story

Chapter 19: Delving into Environmental Data Science

Modeling Environmental-Human Interactions with Environmental Intelligence

Modeling Natural Resources in the Raw

Using Spatial Statistics to Predict for Environmental Variation across Space

Chapter 20: Data Science for Driving Growth in E-Commerce

Making Sense of Data for E-Commerce Growth

Optimizing E-Commerce Business Systems

Chapter 21: Using Data Science to Describe and Predict Criminal

Activity

Temporal Analysis for Crime Prevention and Monitoring

Spatial Crime Prediction and Monitoring

Probing the Problems with Data Science for Crime Analysis

Part 6: The Part of Tens

Chapter 22: Ten Phenomenal Resources for Open Data

Digging through data.gov

Checking Out Canada Open Data

Diving into data.gov.uk

Checking Out U.S. Census Bureau Data

Knowing NASA Data

Wrangling World Bank Data

Getting to Know Knoema Data

Queuing Up with Quandl Data

Exploring Exversion Data

Mapping OpenStreetMap Spatial Data

Chapter 23: Ten Free Data Science Tools and Applications

Making Custom Web-Based Data Visualizations with Free R Packages

Examining Scraping, Collecting, and Handling Tools

Looking into Data Exploration Tools

Evaluating Web-Based Visualization Tools

About the Author

Connect with Dummies

End User License Agreement

Introduction

The power of big data and data science are revolutionizing the world. From the modern business

enterprise to the lifestyle choices of today’s digital citizen, data science insights are driving

changes and improvements in every arena. Although data science may be a new topic to many, it’s

a skill that any individual who wants to stay relevant in her career field and industry needs to

know.

This book is a reference manual to guide you through the vast and expansive areas encompassed

by big data and data science. If you’re looking to learn a little about a lot of what’s happening

across the entire space, this book is for you. If you’re an organizational manager who seeks to

understand how data science and big data implementations could improve your business, this book

is for you. If you’re a technical analyst, or even a developer, who wants a reference book for a

quick catch-up on how machine learning and programming methods work in the data science

space, this book is for you.

But, if you are looking for hands-on training in deep and very specific areas that are involved in

actually implementing data science and big data initiatives, this is not the book for you. Look

elsewhere because this book focuses on providing a brief and broad primer on all the areas

encompassed by data science and big data. To keep the book at the For Dummies level, I do not go

too deeply or specifically into any one area. Plenty of online courses are available to support

people who want to spend the time and energy exploring these narrow crevices. I suggest that

people follow up this book by taking courses in areas that are of specific interest to them.

Although other books dealing with data science tend to focus heavily on using Microsoft Excel to

learn basic data science techniques, Data Science For Dummies goes deeper by introducing the R

statistical programming language, Python, D3.js, SQL, Excel, and a whole plethora of open-source

applications that you can use to get started in practicing data science. Some books on data science

are needlessly wordy, with their authors going in circles trying to get to the point. Not so here.

Unlike books authored by stuffy-toned, academic types, I’ve written this book in friendly,

approachable language — because data science is a friendly and approachable subject!

To be honest, until now, the data science realm has been dominated by a few select data science

wizards who tend to present the topic in a manner that’s unnecessarily overly technical and

intimidating. Basic data science isn’t that confusing or difficult to understand. Data science is

simply the practice of using a set of analytical techniques and methodologies to derive and

communicate valuable and actionable insights from raw data. The purpose of data science is to

optimize processes and to support improved data-informed decision making, thereby generating an

increase in value — whether value is represented by number of lives saved, number of dollars

retained, or percentage of revenues increased. In Data Science For Dummies, I introduce a broad

array of concepts and approaches that you can use when extracting valuable insights from your

data.

Many times, data scientists get so caught up analyzing the bark of the trees that they simply forget

to look for their way out of the forest. This common pitfall is one that you should avoid at all

costs. I’ve worked hard to make sure that this book presents the core purpose of each data science

technique and the goals you can accomplish by utilizing them.

About This Book

In keeping with the For Dummies brand, this book is organized in a modular, easy-to-access

format that allows you to use the book as a practical guidebook and ad hoc reference. In other

words, you don’t need to read it through, from cover to cover. Just take what you want and leave

the rest. I’ve taken great care to use real-world examples that illustrate data science concepts that

may otherwise be overly abstract.

Web addresses and programming code appear in monofont. If you’re reading a digital version of

this book on a device connected to the Internet, you can click a web address to visit that website,

like this: www.dummies.com.

Foolish Assumptions

In writing this book, I’ve assumed that readers are at least technically minded enough to have

mastered advanced tasks in Microsoft Excel — pivot tables, grouping, sorting, plotting, and the

like. Having strong skills in algebra, basic statistics, or even business calculus helps as well.

Foolish or not, it’s my high hope that all readers have a subject-matter expertise to which they can

apply the skills presented in this book. Because data scientists must be capable of intuitively

understanding the implications and applications of the data insights they derive, subject-matter

expertise is a major component of data science.

Icons Used in This Book

As you make your way through this book, you’ll see the following icons in the margins:

The Tip icon marks tips (duh!) and shortcuts that you can use to make subject mastery

easier.

Remember icons mark the information that’s especially important to know. To siphon off

the most important information in each chapter, just skim the material represented by these

icons.

The Technical Stuff icon marks information of a highly technical nature that you can

normally skip.

The Warning icon tells you to watch out! It marks important information that may save you

headaches.

Beyond the Book

This book includes the following external resources:

Data Science Cheat Sheet: This book comes with a handy Cheat Sheet which lists helpful

shortcuts as well as abbreviated definitions for essential processes and concepts described in

the book. You can use it as a quick-and-easy reference when doing data science. To get this

Cheat Sheet, simply go to www.dummies.com and search for Data Science Cheat Sheet in the

Search box.

Data Science Tutorial Datasets: This book has a few tutorials that rely on external datasets.

You can download all datasets for these tutorials from the GitHub repository for this course at

https://github.com/BigDataGal/Data-Science-for-Dummies.

Where to Go from Here

Just to reemphasize the point, this book’s modular design allows you to pick up and start reading

anywhere you want. Although you don’t need to read from cover to cover, a few good starter

chapters are Chapters 1, 2, and 9.

Foreword

We live in exciting, even revolutionary times. As our daily interactions move from the physical

world to the digital world, nearly every action we take generates data. Information pours from our

mobile devices and our every online interaction. Sensors and machines collect, store, and process

information about the environment around us. New, huge data sets are now open and publicly

accessible.

This flood of information gives us the power to make more informed decisions, react more quickly

to change, and better understand the world around us. However, it can be a struggle to know where

to start when it comes to making sense of this data deluge. What data should one collect? What

methods are there for reasoning from data? And, most importantly, how do we get the answers

from the data to answer our most pressing questions about our businesses, our lives, and our

world?

Data science is the key to making this flood of information useful. Simply put, data science is the

art of wrangling data to predict our future behavior, uncover patterns to help prioritize or provide

actionable information, or otherwise draw meaning from these vast, untapped data resources.

I often say that one of my favorite interpretations of the word “big” in Big Data is “expansive.”

The data revolution is spreading to so many fields that it is now incumbent on people working in

all professions to understand how to use data, just as people had to learn how to use computers in

the 80’s and 90’s. This book is designed to help you do that.

I have seen firsthand how radically data science knowledge can transform organizations and the

world for the better. At DataKind, we harness the power of data science in the service of humanity

by engaging data science and social sector experts to work on projects addressing critical

humanitarian problems. We are also helping drive the conversation about how data science can be

applied to solve the world’s biggest challenges. From using satellite imagery to estimate poverty

levels to mining decades of human rights violations to prevent further atrocities, DataKind teams

have worked with many different nonprofits and humanitarian organizations just beginning their

data science journeys. One lesson resounds through every project we do: The people and

organizations that are most committed to using data in novel and responsible ways are the ones

who will succeed in this new environment.

Just holding this book means you are taking your first steps on that journey, too. Whether you are a

seasoned researcher looking to brush up on some data science techniques or are completely new to

the world of data, Data Science For Dummies will equip you with the tools you need to show

whatever you can dream up. You’ll be able to demonstrate new findings from your physical

activity data, to present new insights from the latest marketing campaign, and to share new

learnings about preventing the spread of disease.

We truly are on the forefront of a new data age and those that learn data science will be able to

take part in this thrilling new adventure, shaping our path forward in every field. For you, that

adventure starts now. Welcome aboard!

Jake Porway

Founder and Executive Director of DataKind

Part 1

Getting Started with Data Science

IN THIS PART …

Get introduced to the field of data science.

Define big data.

Explore solutions for big data problems.

See how real-world businesses put data science to good use.

Chapter 1

Wrapping Your Head around Data Science

IN THIS CHAPTER

Making use of data science in different industries

Putting together different data science components

Identifying viable data science solutions to your own data challenges

Becoming more marketable by way of data science

For quite some time now, everyone has been absolutely deluged by data. It’s coming from every

computer, every mobile device, every camera, and every imaginable sensor — and now it’s even

coming from watches and other wearable technologies. Data is generated in every social media

interaction we make, every file we save, every picture we take, and every query we submit; it’s

even generated when we do something as simple as ask a favorite search engine for directions to

the closest ice-cream shop.

Although data immersion is nothing new, you may have noticed that the phenomenon is

accelerating. Lakes, puddles, and rivers of data have turned to floods and veritable tsunamis of

structured, semistructured, and unstructured data that’s streaming from almost every activity that

takes place in both the digital and physical worlds. Welcome to the world of big data!

If you’re anything like me, you may have wondered, “What’s the point of all this data? Why use

valuable resources to generate and collect it?” Although even a single decade ago, no one was in a

position to make much use of most of the data that’s generated, the tides today have definitely

turned. Specialists known as data engineers are constantly finding innovative and powerful new

ways to capture, collate, and condense unimaginably massive volumes of data, and other

specialists, known as data scientists, are leading change by deriving valuable and actionable

insights from that data.

In its truest form, data science represents the optimization of processes and resources. Data

science produces data insights — actionable, data-informed conclusions or predictions that you

can use to understand and improve your business, your investments, your health, and even your

lifestyle and social life. Using data science insights is like being able to see in the dark. For any

goal or pursuit you can imagine, you can find data science methods to help you predict the most

direct route from where you are to where you want to be — and to anticipate every pothole in the

road between both places.

Seeing Who Can Make Use of Data Science

The terms data science and data engineering are often misused and confused, so let me start off

by clarifying that these two fields are, in fact, separate and distinct domains of expertise. Data

science is the computational science of extracting meaningful insights from raw data and then

effectively communicating those insights to generate value. Data engineering, on the other hand, is

an engineering domain that’s dedicated to building and maintaining systems that overcome data

processing bottlenecks and data handling problems for applications that consume, process, and

store large volumes, varieties, and velocities of data. In both data science and data engineering,

you commonly work with these three data varieties:

Structured: Data is stored, processed, and manipulated in a traditional relational database

management system (RDBMS).

Unstructured: Data that is commonly generated from human activities and doesn’t fit into a

structured database format.

Semistructured: Data doesn’t fit into a structured database system, but is nonetheless

structured by tags that are useful for creating a form of order and hierarchy in the data.

A lot of people believe that only large organizations that have massive funding are implementing

data science methodologies to optimize and improve their business, but that’s not the case. The

proliferation of data has created a demand for insights, and this demand is embedded in many

aspects of our modern culture — from the Uber passenger who expects his driver to pick him up

exactly at the time and location predicted by the Uber application, to the online shopper who

expects the Amazon platform to recommend the best product alternatives so she can compare

similar goods before making a purchase. Data and the need for data-informed insights are

ubiquitous. Because organizations of all sizes are beginning to recognize that they’re immersed in

a sink-or-swim, data-driven, competitive environment, data know-how emerges as a core and

requisite function in almost every line of business.

What does this mean for the everyday person? First, it means that everyday employees are

increasingly expected to support a progressively advancing set of technological requirements.

Why? Well, that’s because almost all industries are becoming increasingly reliant on data

technologies and the insights they spur. Consequently, many people are in continuous need of re-

upping their tech skills, or else they face the real possibility of being replaced by a more tech-

savvy employee.

The good news is that upgrading tech skills doesn’t usually require people to go back to college,

or — God forbid — get a university degree in statistics, computer science, or data science. The

bad news is that, even with professional training or self-teaching, it always takes extra work to

stay industry-relevant and tech-savvy. In this respect, the data revolution isn’t so different from any

other change that has hit industry in the past. The fact is, in order to stay relevant, you need to take

the time and effort to acquire only the skills that keep you current. When you’re learning how to do

data science, you can take some courses, educate yourself using online resources, read books like

this one, and attend events where you can learn what you need to know to stay on top of the game.

Who can use data science? You can. Your organization can. Your employer can. Anyone who has a

bit of understanding and training can begin using data insights to improve their lives, their careers,

and the well-being of their businesses. Data science represents a change in the way you approach

the world. When exacting outcomes, people often used to make their best guess, act, and then hope

for their desired result. With data insights, however, people now have access to the predictive

vision that they need to truly drive change and achieve the results they need.

You can use data insights to bring about changes in the following areas:

Business systems: Optimize returns on investment (those crucial ROIs) for any measurable

activity.

Technical marketing strategy development: Use data insights and predictive analytics to

identify marketing strategies that work, eliminate under-performing efforts, and test new

marketing strategies.

Keep communities safe: Predictive policing applications help law enforcement personnel

predict and prevent local criminal activities.

Help make the world a better place for those less fortunate: Data scientists in developing

nations are using social data, mobile data, and data from websites to generate real-time

analytics that improve the effectiveness of humanitarian response to disaster, epidemics, food

scarcity issues, and more.

Analyzing the Pieces of the Data Science Puzzle

To practice data science, in the true meaning of the term, you need the analytical know-how of

math and statistics, the coding skills necessary to work with data, and an area of subject matter

expertise. Without this expertise, you might as well call yourself a mathematician or a statistician.

Similarly, a software programmer without subject matter expertise and analytical know-how might

better be considered a software engineer or developer, but not a data scientist.

Because the demand for data insights is increasing exponentially, every area is forced to adopt

data science. As such, different flavors of data science have emerged. The following are just a few

titles under which experts of every discipline are using data science: ad tech data scientist,

director of banking digital analyst, clinical data scientist, geoengineer data scientist, geospatial

analytics data scientist, political analyst, retail personalization data scientist, and clinical

informatics analyst in pharmacometrics. Given that it often seems that no one without a scorecard

can keep track of who’s a data scientist, in the following sections I spell out the key components

that are part of any data science role.

Collecting, querying, and consuming data

Data engineers have the job of capturing and collating large volumes of structured, unstructured,

and semistructured big data — data that exceeds the processing capacity of conventional database

systems because it’s too big, it moves too fast, or it doesn’t fit the structural requirements of

traditional database architectures. Again, data engineering tasks are separate from the work that’s

performed in data science, which focuses more on analysis, prediction, and visualization. Despite

this distinction, whenever data scientists collect, query, and consume data during the analysis

process, they perform work similar to that of the data engineer (the role you read about earlier in

this chapter).

Although valuable insights can be generated from a single data source, often the combination of

several relevant sources delivers the contextual information required to drive better data-informed

decisions. A data scientist can work from several datasets that are stored in a single database, or

even in several different data warehouses. (For more about combining datasets, see Chapter 3.) At

other times, source data is stored and processed on a cloud-based platform that’s been built by

software and data engineers.

No matter how the data is combined or where it’s stored, if you’re a data scientist, you almost

always have to query data — write commands to extract relevant datasets from data storage

systems, in other words. Most of the time, you use Structured Query Language (SQL) to query data.

(Chapter 16 is all about SQL, so if the acronym scares you, jump ahead to that chapter now.)

Whether you’re using an application or doing custom analyses by using a programming language

such as R or Python, you can choose from a number of universally accepted file formats:

Comma-separated values (CSV) files: Almost every brand of desktop and web-based

analysis application accepts this file type, as do commonly used scripting languages such as

Python and R.

Scripts: Most data scientists know how to use either the Python or R programming language to

analyze and visualize data. These script files end with the extension .py or .ipynb

(Python) or .r (R).

Application files: Excel is useful for quick-and-easy, spot-check analyses on small- to

medium-size datasets. These application files have the .xls or .xlsx extension.

Geospatial analysis applications such as ArcGIS and QGIS save with their own proprietary

file formats (the .mxd extension for ArcGIS and the .qgs extension for QGIS).

Web programming files: If you’re building custom, web-based data visualizations, you may

be working in D3.js — or Data-Driven Documents, a JavaScript library for data visualization.

When you work in D3.js, you use data to manipulate web-based documents using .html,

.svg, and .css files.

Applying mathematical modeling to data science tasks

Data science relies heavily on a practitioner’s math skills (and statistics skills, as described in the

following section) precisely because these are the skills needed to understand your data and its

significance. These skills are also valuable in data science because you can use them to carry out

predictive forecasting, decision modeling, and hypotheses testing.

Mathematics uses deterministic methods to form a quantitative (or numerical)

description of the world; statistics is a form of science that’s derived from mathematics, but

it focuses on using a stochastic (probabilities) approach and inferential methods to form a

quantitative description of the world. More on both is discussed in Chapter 5.

Data scientists use mathematical methods to build decision models, generate approximations, and

make predictions about the future. Chapter 5 presents many complex applied mathematical

approaches that are useful when working in data science.

In this book, I assume that you have a fairly solid skill set in basic math — it would be

beneficial if you’ve taken college-level calculus or even linear algebra. I try hard, however,

to meet readers where they are. I realize that you may be working based on a limited

mathematical knowledge (advanced algebra or maybe business calculus), so I convey

advanced mathematical concepts using a plain-language approach that’s easy for everyone to

understand.

Deriving insights from statistical methods

In data science, statistical methods are useful for better understanding your data’s significance, for

validating hypotheses, for simulating scenarios, and for making predictive forecasts of future

events. Advanced statistical skills are somewhat rare, even among quantitative analysts, engineers,

and scientists. If you want to go places in data science, though, take some time to get up to speed in

a few basic statistical methods, like linear and logistic regression, naïve Bayes classification, and

time series analysis. These methods are covered in Chapter 5.

Coding, coding, coding — it’s just part of the game

Coding is unavoidable when you’re working in data science. You need to be able to write code so

that you can instruct the computer how you want it to manipulate, analyze, and visualize your data.

Programming languages such as Python and R are important for writing scripts for data

manipulation, analysis, and visualization, and SQL is useful for data querying. The JavaScript

library D3.js is a hot new option for making cool, custom, and interactive web-based data

visualizations.

Although coding is a requirement for data science, it doesn’t have to be this big scary thing that

people make it out to be. Your coding can be as fancy and complex as you want it to be, but you

can also take a rather simple approach. Although these skills are paramount to success, you can

pretty easily learn enough coding to practice high-level data science. I’ve dedicated Chapters 10,

14, 15, and 16 to helping you get up to speed in using D3.js for web-based data visualization,

coding in Python and in R, and querying in SQL (respectively).

Applying data science to a subject area

Statisticians have exhibited some measure of obstinacy in accepting the significance of data

science. Many statisticians have cried out, “Data science is nothing new! It’s just another name for

what we’ve been doing all along.” Although I can sympathize with their perspective, I’m forced to

stand with the camp of data scientists who markedly declare that data science is separate and

definitely distinct from the statistical approaches that comprise it.

My position on the unique nature of data science is based to some extent on the fact that data

scientists often use computer languages not used in traditional statistics and take approaches

derived from the field of mathematics. But the main point of distinction between statistics and data

science is the need for subject matter expertise.

Because statisticians usually have only a limited amount of expertise in fields outside of statistics,

they’re almost always forced to consult with a subject matter expert to verify exactly what their

findings mean and to decide the best direction in which to proceed. Data scientists, on the other

hand, are required to have a strong subject matter expertise in the area in which they’re working.

Data scientists generate deep insights and then use their domain-specific expertise to understand

exactly what those insights mean with respect to the area in which they’re working.

This list describes a few ways in which subject matter experts are using data science to enhance

performance in their respective industries:

Engineers use machine learning to optimize energy efficiency in modern building design.

Clinical data scientists work on the personalization of treatment plans and use healthcare

informatics to predict and preempt future health problems in at-risk patients.

Marketing data scientists use logistic regression to predict and preempt customer churn (the

loss or churn of customers from a product or service to that of a competitor’s). I tell you more

on decreasing customer churn in Chapters 3 and 20.

Data journalists scrape websites (extract data in-bulk directly off the pages on a website,

in other words) for fresh data in order to discover and report the latest breaking-news stories.

(I talk more about data journalism in Chapter 18.)

Data scientists in crime analysis use spatial predictive modeling to predict,...