Showing posts with label NoSQL. Show all posts
Showing posts with label NoSQL. Show all posts

Sunday, August 9, 2015

Size doesnt matter. Speed does!

After beeing involved in about 30 projects in different sizes , companies , technologies and solutions i made a conclusion : data sizes are not the major issue in a project. Actually picking the right technology is just a matter of time and money. Most problematic part is the speed of the data: throughput of writes and reads of the data itself. And here the truth creativity comes to play : modelling the data in correct way can drasticly change ( in both ways) the project .

Friday, January 10, 2014

Order the mess : ACID ,CAP theorem and NoSQL



Database transaction
Lets define a transaction as unit  of work that is reliable and recoverable. By definition transaction must support ACID features :
 Atomic - if transaction contains several actions it can occur in all-or-nothing manner.
Consistent - if i commit some action - all other actions see the change immediately.
Isolated - 2 transactions that run on the same data will see different copies until on of them commits changes.
Example
1 transactions reads some value when other updates it.Untill transaction B commits - transaction A will see an old value.
Durable -Once transaction is committed -even if the system crushes, the transaction results not going to get lost.Its achieved by managing transaction logs that can be replayed,

Great! we have 4 properties of a transaction.

All those properties achieved in different ways .


CAP theorem
Its set of  requirements for distributed system
Consistency- all servers in the distributed system will be in the same state.
Availability - you will be always be able to get data out of the system ( even if isn't consistent)
Partition tolerance - service will stay available even if some parts of he system crushes.

So its proved that all 3 of them can't be achieved at the same time.And its required to have 2 out of 3.

So lets make some order :
1. CAP are properties of distributed system.
2. ACID are futures of transaction .

Transactions are futures of relational databases. like MySQL, Oracle etc.

Big Data era.

Amounts of data  that is being generated in last years caused  scale and performance to be most important over some ACID futures. Providing Consistency for example causes the system to be much slower. And sometimes we can live with an idea that the data will not be consistency at the same second ( but finally yes!)

That causes the idea of other DB systems to get popular . Some of them provide parts of ACID , some of them don't have SQL as main interface. And we call them NoSQL databases.
Ideas of Relational storage is avoided in some of them.
But as main concept they provide 2 out of 3 CAP requirements, avoid ACID and solve some predefined problem in data storage, like graph relations, columnar storage etc.

Which means the main focus now is on data instead of the system.You look, investigate the data then you look for storage solution instead of taking some Relational storage and starting modelling the data .




http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pdf
http://en.wikipedia.org/wiki/Eventual_consistency
http://www.cs.berkeley.edu/~istoica/classes/cs268/06/notes/20-BFTx2.pdf




Thursday, January 9, 2014

Couchbase : almost document-oriented database





After deep investigation of several NoSQL DB's Looks like couchbase is one of best options

1. It has JSON support 
like document-oriented DB but it doesn't indexes all the fileds on all levels of the document ( good and bad!).
It something between Document-Oriented DB and Key-Value store with an option for custom Indexes View creation which is actually a simple map reduce job that runs automaticly.

2. Auto-Sharding
In 1 click you can simply add servers.

3.Cross Cluster replication

4. Object level cache based on memcached

5.  Built in management and monitoring tool

6. Asynchronous persistence.


7. Benchmarks:
http://www.couchbase.com/presentations/benchmarking-couchbase
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-708169.pdf

8. Best practices

9. Great documentation

10. Supports ( not offically) GeoSearch

11. Good API's
Contains Bulk modes ( on Views)
CAS oprations ( optimistic locking)
and much more

It looks like it fits all avarage needs but during the work you need to pay attention to some things like:
1. Working with keys and values instead of indexes ( indexes defined by map - reduce jobs update incrementally  and you can not control it too much)
2. Compaction on views and buckets exist and can be controlled.
3. Don't define more than couple of buckets. ( Read attached best  practices paper)

have fun with couchbase.




Tuesday, September 24, 2013

Tipical NoSQL Big data solution ( part 1)

Big data components

In flow

This is actually the data that gets into the system. It can be files , any kind of events or web pages.. We dont care .

Distributor

When we recieve our in flow we need to distribute it. The distribution can be based on replication of the data to several destination or distributed according to some data details.
Example: If log record contains word : event - send it to HDFS only.
Examples: Apache flume,Logstash ,Fluentd


Storages - Long term, short term 

Then we save the data to storages. We have several types of storages and each one has its pros and cons.


Long term 

We need it to hold the whole data and analyze it by batch processing. In most of cases it will be hadoop based HDFS storage and we use Map-Reduce / Hive / Pig jobs run and create some reports.
As you can understand - its heavy and slow process.


Short term 

If we need our data to be easuly and fast accessible we will use some high scalable database.We have several types here:
Examples : Redis, Riak, Dynamo, Gemfire
Examples: Vertica,MonetDB

Examples : MongoDB, Cassandra,CouchDB
Examples : Neo4J

Data ModelPerformanceScalabilityFlexibilityComplexityFunctionality
Key–value Storeshighhighhighnonevariable (none)
Column Storehighhighmoderatelowminimal
Document Storehighvariable (high)highlowvariable (low)
Graph Databasevariablevariablehighhighgraph theory
Relational Databasevariablevariablelowmoderaterelational algebra.
The data is much faster acessible and much more structurized.



Real time processing

This component in most of cases will be Storm (http://storm-project.net/).It will pull the data ( in our case we use Kafka( http://kafka.apache.org/) and process it based on Short term and fast access data.
Probably it's decision should be sent to some external systems  to notify end user.


End User 

Will use some  stack for visualizing the data.
It also can contain a service for  querying data.In most of cases it will be against short term storages.

Next part is comming...

Thursday, August 22, 2013

NoSQL - types and use cases



In order to explain what NoSQL is and why its needed lets first introduce what was before it came into the big game...and before we knew RDBMS what provided us the main concept - ACID.
Atomicity- transactivity of actions - if we have a transaction that contains actions A,B and C - all those actions should success .If one of them fails - we shoudl rollback the previous one's to initial state.
Consistency - if tranascation A is ececuted and it should do 2 actions : 1. incease balance of account a1 in 200$  and action 2. should decrease account b1 in 100$ onec transaction is done (in success!) both numbers will be updated ( we dont care what happens during transaction execution. consistency is about what happens in the end)
Isolation - transactions that executed in parallel dont impact each other.
Durability - we dont care if the electiricty goes down in the whole area or the machine totally went down - once transaction was done - even if the system goes down - when it gets up again - the transaction result will be updated.

Great!the concept is clear and here main keyplayer come...
ORACLE , SQLServer, MySQL,DB2 etc..
Everything went fine..till the data started growing with huge speed.... then it started beeing clear that setting up 1 single machine is not enough anymore.In addition to that oracle licensing is based on CPU amount on the machine...making simple map shows us that something totally different should come and replace RDBMS....not yet! Let's shot a last bullet ..last nail to the grave of RDBMS...
CAP theorem(Brewer's theorem)
It says that its impossible in destributed computing provide Consistency,Isolation and Partition tolerance.
So lets summarize:
We have ACID (Oracle) and we want to add more machines to reach scalability.Then we go into destributed computing.And then comes CAP theorem and kicks our a**.
So what know?
We want ACID,we buy license but we cant buy it?
So one option is  to use open source RDBMS and save money...
But what if i say that you you are still in trouble? Your system based on RDBMS cant be infinitly scalible..
Once you reach to petabyte - you will be soooooo slow that you buisness will crush?
Then NoSQL comes to the game...
And what you have there?
*Open Source
*Greatly scalible
*Not Relational
*Destributed
system that during last 10 years proved itself in different companies.
We have 4 families:
And each family has its advantages and disadvantages:
Data ModelPerformanceScalabilityFlexibilityComplexityFunctionality
Key-Value Storeshighhighhighnonevariable (none)
Column Storehighhighmoderatelowminimal
Document Storehighvariable (high)highlowvariable (low)
Graph Databasevariablevariablehighhighgraph theory
Relational Databasevariablevariablelowmoderaterelational algebra.

So you have different databases and the you defenetly say : so which one to pick?
Thats what i've asked first time i saw it. Sometimes i needed graph and sometimes relational and sometimes key-value...
So the answer is simple : you can hold all of them... and combine them...

Sources;
http://en.wikipedia.org/wiki/NoSQL
http://db-engines.com/en/ranking
http://highlyscalable.wordpress.com/2012/03/01/nosql-data-modeling-techniques/


PS: if you look into db-engines link you can see top DB engines rank.Sure!Oracle Number 1. But im almost sure that NoSQL and other technologies will make with oracle the same Linux did to Windows.Sure!ORacle will stay forever...but world around it will change..

Wednesday, May 22, 2013

We don't need Big Data?!


Tipical Relational Database story begins like :
1. On proof of concept stage we use Local DB installation. 
2. Then, after the step we go into production with remote RDBMS on separate machine.
3. Then we grow and see that many read operation occures.In order to optimize this we add some memcahe solution to cache queries.Here we loos ACID on read operation.
4. Here we grow more and we relize that many write operation hit our system...Then we add additional servers and spend money for new hosts.
5. Then we start to add additional feautures to our system that increases query complexity.Many joins occure. As a solution for this - denormalization of the data.
6. Then the popularity grows more  and things are gettign slow...to solve his we try to reduce server side computation tasks and parametrize some unneeded ones...
7.Then we stabilize reads operations but write operations are still getting slow...And then to improve this we drop indexes and triggers...so what we got in the end?
Not fully ACID , many machines, denormalized data with no indexes...
Lets think together........















Material downloaded from Internet from different sources.





Wednesday, December 19, 2012

What is Big Data



What is Big Data?

The amount of data in our world has been exploding, and analyzing large data sets—so-called big data—will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus.
The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data for the foreseeable future.

Examples :

  • Walmart handles more than 1 million customer transactions every hour, which is imported into databases estimated to contain more than 2.5 petabytes of data – the equivalent of 167 times the information contained in all the books in the US Library of Congress.
  • Facebook handles 40 billion photos from its user base.
  • FICO Falcon Credit Card Fraud Detection System protects 2.1 billion active accounts world-wide.
  • The volume of business data worldwide, across all companies, doubles every 1.2 years, according to estimates


  • Facts:

    1. Data have swept into every industry and business function and are now an important factor of production, alongside labor and capital
    2. Analytics can substantially improve decision-making   and forcasting.
    3. The use of big data will become a key basis of competition and growth for individual firms
    5.The use of big data will underpin new waves of productivity growth and consumer surplus
    6. There will be a shortage of talent necessary for organizations to take advantage of big data.
    7. Several issues will have to be addressed to capture the full potential of big data. Policies related to privacy, security, intellectual property, and even liability will need to be addressed in a big data world.
    8. Companies will increasingly need to integrate information from multiple data sources, often from third parties


    Big Data technologies



    Sources
    Big Data on Wikipedia
    Mckinsey