5 - Choosing Database.pptx

1 - NoSQL.pptx

2 - HBase.pptx

3 - Cassandra.pptx

4 - MongoDB.pptx

CHOOSING YOUR

DATABASE

Integration considerations

Scaling requirements

Support considerations

Budget considerations? Probably not.

CAP considerations

Availability

Consistency

Partition-Tolerance

Simplicity

An example

■ You’re building an internal phone directory app

Scale: limited

–

– Consistency: Eventual is fine

– Availability requirements: not mission critical

– MySQL is probably already installed on your web server…

Another example

■ You want to mine web server logs for interesting patterns

■ What are the most popular times of day? What’s the average session length?

Etc.

Another example

■ You have a big Spark job that produces movie recommendations for end users

nightly

■ Something needs to vend this data to your web applications

■ You work for some huge company with massive scale

■ Downtime is not tolerated

■ Must be fast

■ Eventual consistency OK – it’s just reads

You try it!

■

You’re building a massive stock trading system

■ Consistency is more important than anything

■

“Big data” is present

It’s really, really important – so having access to professional support might be

a good idea. And you have enough budget to pay forit.

CAP considerations

Availability

Consistency

Partition-Tolerance

NoSQL

When RDBMS’s don’t cut it

Random Access to Planet-Size Data

Scaling up MySQL etc. to massive loads

requires extreme measures

■ Denormalization

■ Caching layers

■ Master/slave setups

■ Sharding

■ Materialized views

■ Removing stored procedures

Support nightmare

Do you really need SQL?

■ Your high-transaction queries are probably pretty simple once de-normalized

■ A simple get / put API may meet your needs

■ Looking up values for a given key is simple, fast, and scalable

■ You can do both…

Sample architecture

Client & request

router

Shard 1

Shard 2

Shard 3

Shard N

Shard 1

backup

Shard 2

backup

Shard 3

backup

Shard N

backup

Use the right tool for the job

■ For analytic queries, Hive, Pig, Spark, etc. work great.

■ Exporting data to MySQL is plenty fast for most applications too.

■ But if you work at giant scale – export your data to a non-relational database

for fast and scalable serving of that data to web applications, etc.

Sample application architecture (greatly

simplified!)

Happy customers

Internet

Web servers

MongoDB

Spark Streaming

Data

source(s)

Hadoop YARN / HDFS

Every box represents many machines!

Choose a database design that

meets your usage patterns

HBase

Non-relational, scalable database built

on HDFS

Based on Google’s BigTable

CRUD

■ Create

■ Read

■ Update

■ Delete

■ There is no query language, only CRUDAPI’s!

HBase architecture

Zookeeper

HMaster

Region

Server

Region

Server

Region

Server

Region

Server

Auto-sharding!

HDFS

HBase data model

■ Fast access to any given ROW

■ A ROW is referenced by a unique KEY

■ Each ROW has some small number of COLUMN FAMILIES

■ A COLUMN FAMILY may contain arbitrary COLUMNS

■ You can have a very large number of COLUMNS in a COLUMN FAMILY

■ Each CELL can have many VERSIONS with given timestamps

■ Sparse data is A-OK – missing columns in a row consume no storage.

Example: One row of a web table

Key

com.cnn.www

Contents column family

Anchor column family

Contents:

Anchor:cnnsi.com Anchor:my.look.ca

html><head>

NN…

C<html><head>

NN…

CNN…

“CNN”

“CNN.com”

Some ways to access HBase

■ HBase shell

■ Java API

– Wrappers for Python, Scala, etc.

■ Spark, Hive, Pig

■ REST service

■ Thrift service

■ Avro service

LET’S PLAY WITH

HBase

Creating a HBase table with Python via REST

What are we doing?

■ Create a HBase table for movie ratings by user

■ Then show we can quickly query it for individual users

■ Good example of sparse data

Column family: rating

Rating:50

Rating:33

Rating:223

UserID

How are we doing it?

Python client

REST service

HBase

HDFS

Let’s do this

HBase / Pig

Populating HBase at scale

Integrating Pig with HBase

■ Must create HBase table ahead of time

■ Your relation must have a unique key as its first column, followed by

subsequent columns as you want them saved in Hbase

■ USING clause allows you to STORE into an HBase table

■ Can work at scale – Hbase is transactional on rows

Let’s do this

CASSANDRA

A distributed database with no single point of failure

Cassandra – NoSQL with a twist

■ Unlike HBase, there is no master node at all – every node runs exactly the

same software and performs the same functions

■ Data model is similar to BigTable / Hbase

■ It’s non-relational, but has a limited CQL query language as its interface

Cassandra’s Design Choices

■ The CAP Theorem says you can only have 2 out of 3: consistency, availability,

partition-tolerance

– And partition-tolerance is a requirement with “big data,” so you really

only get to choose between consistency and availability

■ Cassandra favors availability over consistency

–

It is “eventually consistent”

But you can specify your consistency requirements as part of your

requests. So really it’s “tunable consistency”

Where Cassandra Fits in CAP tradeoffs

Availability

Consistency

Partition-Tolerance

Cassandra architecture

Node

Cassandra and your cluster

■ Cassandra’s great for fast access to rows of information

■ Get the best of both worlds – replicate Cassandra to a another ring that is used

for analytics and Spark integration

Node

CQL (Wait, I thought this was

NoSQL!)

■ Cassandra’s API is CQL, which makes it easy to look like existing database

drivers to applications.

■ CQL is like SQL, but with some big limitations!

– NO JOINS

■ Your data must be de-normalized

■ So, it’s still non-relational

– All queries must be on some primary key

■ Secondary indices are supported, but…

■ CQLSH can be used on the command line to create tables, etc.

■ All tables must be in a keyspace – keyspaces are like databases

Cassandra and Spark

■ DataStax offers a Spark-Cassandra connector

■

Allows you to read and write Cassandra tables as DataFrames

Is smart about passing queries on those DataFrames down to the appropriate

level

■ Use cases:

– Use Spark for analytics on data stored in Cassandra

– Use Spark to transform data and store it into Cassandra for

transactional use

Let’s Play

■ Install Cassandra on our virtual Hadoop node

■ Set up a table for MovieLens users

■ Write into that table and query it from Spark!

MongoDB

Managing HuMONGOus data

Where are we?

Availability

Consistency

Partition-Tolerance

Document-based data model

Looks like JSON. Example:

{

}

“_id” : ObjectID(“7b33e366ae32223aee34fd3”),

“title” : “A blog post about MongoDB”,

“content” : “This is a blog post about MongoDB”,

“comments”: [

{

}

]

“name” : “Samir”,

“email” : [email protected],

“content” : “This is the best article ever written!”

“rating” : 1

No real schema is enforced.

■ You can have different fields in every document if you want to

■ No single “key” as in other databases

–

But you can create indices on any fields you want, or even combinations

of fields.

If you want to “shard”, then you must do so on some index.

■ Results in a lot of flexibility

–

But with great power comes great responsibility

MongoDB terminology

■ Databases

■ Collections

■ Documents

It’s kinda corporate-y

Replication Sets

■

Single-master!

■ Maintains backup copies of your database instance

–

Secondaries can elect a new primary within seconds if your

primary goes

 down

–

But make sure your operation log is long enough to give you

time to recover the primary when it comes back…

PRIMARY

SECONDARY

Replica Set

Quirks

■ A majority of the servers in your set must agree on the primary

Even numbers of servers (like 2) don’t work well

–

■ Don’t want to spend money on 3 servers? You can set up an ‘arbiter’ node

–

But only one

■ Apps must know about enough servers in the replica set to be able to reach one to

learn who’s primary

■ Replicas only address durability, not your ability to scale

– Well, unless you can take advantage of reading from secondaries – which

generally isn’t recommended

And your DB will still go into read-only mode for a bit while a new primary is

elected

–

■ Delayed secondaries can be set up as insurance against people doing dumb things

Sharding

■ Finally – “big data”

■ Ranges of some indexed value you specify are assigned to different replica sets

App Server

Process

App Server

Process

App Server

Process

App Server

Process

mongos

Config

Server

Config

Server

Config

Server

PRIMARY

SECONDARY

PRIMARY

SECONDARY

PRIMARY

SECONDARY

RS1: users

Min -> 1000

RS2: users

1000 -> 5000

RS3: users

5000 -> max

Sharding Quirks

■ Auto-sharding sometimes doesn’t work

–

Split storms, mongos processes restarted too often

■ You must have 3 config servers

– And if any one goes down, your DB is down

– This is on top of the single-master design of replica sets

■ MongoDB’s loose document model can be at odds with effective sharding

Neat Things About MongoDB

■ It’s not just a NoSQL database – very flexible document model

■ Shell is a full JavaScript interpreter

■ Supports many indices

But only one can be used for sharding

–

– More than 2-3 are still discouraged

Full-text indices for text searches

–

Spatial indices

–

■ Built-in aggregation capabilities, MapReduce, GridFS

–

For some applications you might not need Hadoop at all

But MongoDB still integrates with Hadoop, Spark, and most languages

■ A SQL connector is available

–

But MongoDB still isn’t designed for joins and normalized data really.

Let’s Mess Around

■ We’ll integrate MongoDB with Spark

■ Then play around with the resulting database in the mongo shell