After beeing involved in about 30 projects in different sizes , companies , technologies and solutions i made a conclusion : data sizes are not the major issue in a project. Actually picking the right technology is just a matter of time and money. Most problematic part is the speed of the data: throughput of writes and reads of the data itself. And here the truth creativity comes to play : modelling the data in correct way can drasticly change ( in both ways) the project .
Showing posts with label NoSQL. Show all posts
Showing posts with label NoSQL. Show all posts
Sunday, August 9, 2015
Friday, January 10, 2014
Order the mess : ACID ,CAP theorem and NoSQL
Database transaction
Lets define a transaction as unit of work that is reliable and recoverable. By definition transaction must support ACID features :
Atomic - if transaction contains several actions it can occur in all-or-nothing manner.
Consistent - if i commit some action - all other actions see the change immediately.
Isolated - 2 transactions that run on the same data will see different copies until on of them commits changes.
Example
1 transactions reads some value when other updates it.Untill transaction B commits - transaction A will see an old value.
Durable -Once transaction is committed -even if the system crushes, the transaction results not going to get lost.Its achieved by managing transaction logs that can be replayed,
Great! we have 4 properties of a transaction.
All those properties achieved in different ways .
CAP theorem
Its set of requirements for distributed system
Consistency- all servers in the distributed system will be in the same state.
Availability - you will be always be able to get data out of the system ( even if isn't consistent)
Partition tolerance - service will stay available even if some parts of he system crushes.
So its proved that all 3 of them can't be achieved at the same time.And its required to have 2 out of 3.
So lets make some order :
1. CAP are properties of distributed system.
2. ACID are futures of transaction .
Transactions are futures of relational databases. like MySQL, Oracle etc.
Big Data era.
Amounts of data that is being generated in last years caused scale and performance to be most important over some ACID futures. Providing Consistency for example causes the system to be much slower. And sometimes we can live with an idea that the data will not be consistency at the same second ( but finally yes!)
That causes the idea of other DB systems to get popular . Some of them provide parts of ACID , some of them don't have SQL as main interface. And we call them NoSQL databases.
Ideas of Relational storage is avoided in some of them.
But as main concept they provide 2 out of 3 CAP requirements, avoid ACID and solve some predefined problem in data storage, like graph relations, columnar storage etc.
Which means the main focus now is on data instead of the system.You look, investigate the data then you look for storage solution instead of taking some Relational storage and starting modelling the data .
http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pdf
http://en.wikipedia.org/wiki/Eventual_consistency
http://www.cs.berkeley.edu/~istoica/classes/cs268/06/notes/20-BFTx2.pdf
Thursday, January 9, 2014
Couchbase : almost document-oriented database
After deep investigation of several NoSQL DB's Looks like couchbase is one of best options
1. It has JSON support
like document-oriented DB but it doesn't indexes all the fileds on all levels of the document ( good and bad!).
It something between Document-Oriented DB and Key-Value store with an option for custom Indexes View creation which is actually a simple map reduce job that runs automaticly.
2. Auto-Sharding
In 1 click you can simply add servers.
3.Cross Cluster replication
4. Object level cache based on memcached
5. Built in management and monitoring tool
6. Asynchronous persistence.
7. Benchmarks:
http://www.couchbase.com/presentations/benchmarking-couchbase
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-708169.pdf
8. Best practices
10. Supports ( not offically) GeoSearch
11. Good API's
Contains Bulk modes ( on Views)
CAS oprations ( optimistic locking)
and much more
It looks like it fits all avarage needs but during the work you need to pay attention to some things like:
1. Working with keys and values instead of indexes ( indexes defined by map - reduce jobs update incrementally and you can not control it too much)
2. Compaction on views and buckets exist and can be controlled.
3. Don't define more than couple of buckets. ( Read attached best practices paper)
have fun with couchbase.
Tuesday, September 24, 2013
Tipical NoSQL Big data solution ( part 1)
Big data components
In flow
This is actually the data that gets into the system. It can be files , any kind of events or web pages.. We dont care .
Distributor
When we recieve our in flow we need to distribute it. The distribution can be based on replication of the data to several destination or distributed according to some data details.
Example: If log record contains word : event - send it to HDFS only.
Examples: Apache flume,Logstash ,Fluentd
Storages - Long term, short term
Then we save the data to storages. We have several types of storages and each one has its pros and cons.
Long term
We need it to hold the whole data and analyze it by batch processing. In most of cases it will be hadoop based HDFS storage and we use Map-Reduce / Hive / Pig jobs run and create some reports.
As you can understand - its heavy and slow process.
Short term
If we need our data to be easuly and fast accessible we will use some high scalable database.We have several types here:
Key - Value Databases - http://en.wikipedia.org/wiki/NoSQL#Document_store
Examples : Redis, Riak, Dynamo, Gemfire
Columnar databases - http://en.wikipedia.org/wiki/Column-oriented_DBMS
Examples: Vertica,MonetDB
Document databases - http://en.wikipedia.org/wiki/Document-oriented_database
Examples : MongoDB, Cassandra,CouchDB
Graph databases -http://en.wikipedia.org/wiki/Graph_database
Examples : Neo4J
| Data Model | Performance | Scalability | Flexibility | Complexity | Functionality |
|---|---|---|---|---|---|
| Key–value Stores | high | high | high | none | variable (none) |
| Column Store | high | high | moderate | low | minimal |
| Document Store | high | variable (high) | high | low | variable (low) |
| Graph Database | variable | variable | high | high | graph theory |
| Relational Database | variable | variable | low | moderate | relational algebra. |
The data is much faster acessible and much more structurized.
Real time processing
This component in most of cases will be Storm (http://storm-project.net/).It will pull the data ( in our case we use Kafka( http://kafka.apache.org/) and process it based on Short term and fast access data.
Probably it's decision should be sent to some external systems to notify end user.
End User
Will use some stack for visualizing the data.It also can contain a service for querying data.In most of cases it will be against short term storages.
Next part is comming...
Thursday, August 22, 2013
NoSQL - types and use cases
In order to explain what NoSQL is and why its needed lets first introduce what was before it came into the big game...and before we knew RDBMS what provided us the main concept - ACID.
Atomicity- transactivity of actions - if we have a transaction that contains actions A,B and C - all those actions should success .If one of them fails - we shoudl rollback the previous one's to initial state.
Consistency - if tranascation A is ececuted and it should do 2 actions : 1. incease balance of account a1 in 200$ and action 2. should decrease account b1 in 100$ onec transaction is done (in success!) both numbers will be updated ( we dont care what happens during transaction execution. consistency is about what happens in the end)
Isolation - transactions that executed in parallel dont impact each other.
Durability - we dont care if the electiricty goes down in the whole area or the machine totally went down - once transaction was done - even if the system goes down - when it gets up again - the transaction result will be updated.
Great!the concept is clear and here main keyplayer come...
ORACLE , SQLServer, MySQL,DB2 etc..
Everything went fine..till the data started growing with huge speed.... then it started beeing clear that setting up 1 single machine is not enough anymore.In addition to that oracle licensing is based on CPU amount on the machine...making simple map shows us that something totally different should come and replace RDBMS....not yet! Let's shot a last bullet ..last nail to the grave of RDBMS...
CAP theorem(Brewer's theorem)
It says that its impossible in destributed computing provide Consistency,Isolation and Partition tolerance.
So lets summarize:
We have ACID (Oracle) and we want to add more machines to reach scalability.Then we go into destributed computing.And then comes CAP theorem and kicks our a**.
So what know?
We want ACID,we buy license but we cant buy it?
So one option is to use open source RDBMS and save money...
But what if i say that you you are still in trouble? Your system based on RDBMS cant be infinitly scalible..
Once you reach to petabyte - you will be soooooo slow that you buisness will crush?
Then NoSQL comes to the game...
And what you have there?
*Open Source
*Greatly scalible
*Not Relational
*Destributed
system that during last 10 years proved itself in different companies.
We have 4 families:
- Column: Hbase, Accumulo
- Document: MongoDB, Couchbase
- Key-value : Dynamo, Riak, Redis, Cache, Project Voldemort
- Graph: Neo4J, Allegro, Virtuoso
And each family has its advantages and disadvantages:
| Data Model | Performance | Scalability | Flexibility | Complexity | Functionality |
|---|---|---|---|---|---|
| Key-Value Stores | high | high | high | none | variable (none) |
| Column Store | high | high | moderate | low | minimal |
| Document Store | high | variable (high) | high | low | variable (low) |
| Graph Database | variable | variable | high | high | graph theory |
| Relational Database | variable | variable | low | moderate | relational algebra. |
So you have different databases and the you defenetly say : so which one to pick?
Thats what i've asked first time i saw it. Sometimes i needed graph and sometimes relational and sometimes key-value...
So the answer is simple : you can hold all of them... and combine them...
Sources;
http://en.wikipedia.org/wiki/NoSQL
http://db-engines.com/en/ranking
http://highlyscalable.wordpress.com/2012/03/01/nosql-data-modeling-techniques/
PS: if you look into db-engines link you can see top DB engines rank.Sure!Oracle Number 1. But im almost sure that NoSQL and other technologies will make with oracle the same Linux did to Windows.Sure!ORacle will stay forever...but world around it will change..
Wednesday, May 22, 2013
We don't need Big Data?!
Tipical Relational Database story begins like :
1. On proof of concept stage we use Local DB installation.
2. Then, after the step we go into production with remote RDBMS on separate machine.
3. Then we grow and see that many read operation occures.In order to optimize this we add some memcahe solution to cache queries.Here we loos ACID on read operation.
4. Here we grow more and we relize that many write operation hit our system...Then we add additional servers and spend money for new hosts.
5. Then we start to add additional feautures to our system that increases query complexity.Many joins occure. As a solution for this - denormalization of the data.
6. Then the popularity grows more and things are gettign slow...to solve his we try to reduce server side computation tasks and parametrize some unneeded ones...
7.Then we stabilize reads operations but write operations are still getting slow...And then to improve this we drop indexes and triggers...so what we got in the end?
Not fully ACID , many machines, denormalized data with no indexes...
Lets think together........

Material downloaded from Internet from different sources.
Wednesday, December 19, 2012
What is Big Data
What is Big Data?
The amount of data in our world has been exploding, and analyzing large data sets—so-called big data—will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus.
The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data for the foreseeable future.
Examples :
Facts:
1. Data have swept into every industry and business function and are now an important factor of production, alongside labor and capital
2. Analytics can substantially improve decision-making and forcasting.
3. The use of big data will become a key basis of competition and growth for individual firms
5.The use of big data will underpin new waves of productivity growth and consumer surplus
6. There will be a shortage of talent necessary for organizations to take advantage of big data.
7. Several issues will have to be addressed to capture the full potential of big data. Policies related to privacy, security, intellectual property, and even liability will need to be addressed in a big data world.
8. Companies will increasingly need to integrate information from multiple data sources, often from third parties
Big Data technologies
Sources
Big Data on Wikipedia
Mckinsey
Subscribe to:
Posts (Atom)











