Showing posts with label In-Memory. Show all posts
Showing posts with label In-Memory. Show all posts

Friday, January 25, 2013

In-Memory Data Grids - what and why's

The need

As we discussed previously , data storage need increases exponentially . We need more and more place to hold more and more data. We want it to be high available, scalable and super fast.  Here is where In Memory data grids come into the picture .

The model

The data model is distributed across many servers in a single location or across multiple locations.  This distribution is known as a data fabric.  This distributed model is known as a ‘shared nothing’ architecture.
  • All servers can be active in each site.
  • All data is stored in the RAM of the servers.
  • Servers can be added or removed non-disruptively, to increase the amount of RAM available.
  • The data model is non-relational and is object-based. 
  • Distributed applications written on the .NET and Java application platforms are supported.
  • The data fabric is resilient, allowing non-disruptive automated detection and recovery of a single server or multiple servers.

Starting point

  • VMware Gemfire                                                 (Java)
  • Oracle Coherence                                             (Java)
  • Alachisoft NCache                                              (.Net)
  • Gigaspaces XAP Elastic Caching Edition            (Java)
  • Hazelcast                                                           (Java)
  • Scaleout StateServer                                          (.Net)
  • IBM eXtreme Scale
  • Terracotta Enterprise Suite
  • Jboss (Redhat) Infinispan
Links
http://www.infoq.com/articles/in-memory-data-grids

Wednesday, December 19, 2012

2013 Predictions of CA


1. Big Data Grows Up:  There will be an emergence of Big Data Administrators, who will play a critical role in using new technologies and processing power to take a cold, hard (and useful) look at data and its business application. In 2013, Big Data projects will begin to show demonstrable ROI.The risk of Big Data decision is much lower and the insights Big Data provides will increase IT’s leadership on innovation.
2. Enterprises Adopt Public Cloud: Enterprises will adopt public cloud services, spurred by the expansion of offerings from Service Providers (SP) like established telcos, who have already earned their trust. In addition, as SPs franchise their business models and technologies, cloud services will surge outside the US. Additionally, the buzz of “the cloud” will dim as people realize it’s just the way business is done. Vertical industries such as healthcare will lead this trend, realizing its impact on security, the value of specialized community cloud services and the ability to address compliance while driving down costs. Mainframe will move more rapidly to the cloud as companies that already have a mainframe will adopt cloud technologies and processes to get the most out of them. Externally hosted private cloud adoption will also increase in the next couple of years.
3. Identity is the New Perimeter: Enterprise users know no bounds of time and space.  As they adopt cloud services and collaborate globally with external customers and partners from multiple devices, they erase the traditional IT perimeter. Security professionals today find themselves in a borderless war on multiple fronts and one common ally – Identity. Strongly authenticated identity is the new perimeter. This places emphasis on reducing risk at the authentication point, signaling the end of the password as we know it today. As such, we expect to see advanced authentication models expand. We will see more risk-based authentication coming into play based on the device, transaction, location and more.  We will see industry move towards more intelligent methods of authentication such as pattern creation, image recognition, mobile phone-based authentication, audio and biometrics. But this is not enough. We will see more content-driven security based on what the data is, or how it’s classified, and that information, plus user identity and role, will be used to guide access rights.
4. The Seventh Sense: There will be increased exploitation of sensing technologies available in most modern mobile devices as the ‘Internet of Things’ takes hold. Everything will become intelligent as sensors are embedded into a wide array of devices from the home to those that drive applications that span disaster management, healthcare IT, transportation networks, Smart Grids, utility computing and more. These technologies will drive additional demand for IT to manage, store, analyze and secure the data traffic, privacy and end points.
5. Mobile/Social First in the Enterprise: Companies will start to build applications not just with mobile/social platforms in mind, but primarily for mobile/social platforms, with traditional platforms secondary or not supported at all. Consumerization will accelerate as we see the end of resistance from the enterprise embracing the rich, immersive user experience consumers are used to from mobile applications. In parallel, the management of mobile/social IT will become less about managing and securing the devices themselves, and more about managing and securing the mobile applications and mobile data, all while preserving the user experience.



Sources
CA

In-Memory computing - part 2


Hardware for Data in RAM solution cost

A typical Dell/HP/IBM/Cisco blade with 256GB of DRAM will cost below $20K if you just buy on the list prices That brings the total cost of 2TB cluster well below $200K (with all network and power equipment included and 100s TBs of disk storage).

According to Wikibon article - "Data in DRAM" - we can see that Data in RAM solution can be very expensive because of max 96GB per ndoe RAM limiation.
So in general having 2 - 1TB Blades defenetly redices the cost.

 Examples:



Hazelcast - in memory data grid. Open source.

Oracle Coherence - In memory data grid solution.Download avalible on Oracle website

Ifinispan -  Transactional in-memory key/value NoSQL datastore & Data Grid. Free/JBOSS comunity

MemSQL- For engineering teams who are embarking upon new projects and need a lightning fast database, this is free forever but limited to 32 GB in capacity.

VoltDB- Open Source Version of Enterprise Edition, Languages: English   Memory: 4 GB


VMware vFabric SQLFire
VMware vFabric SQLFire implements standard SQL as well as Java and ADO.NET. Leverage your existing database development skills to write modern applications designed for scale and distributed systems.
The VMware vFabric SQLFire trial software doubles as the no-fee developer software. Limited to 3 node connections, the trial software used for development is otherwise fully functional

VMware vFabric GemFire
 VMware vFabric GemFire delivers a highly scalable, low latency object-based data platform for modern applications.
GemFire’s APIs and integration with the popular Spring framework combine to greatly speed and simplify the development and support of scalable, transactional enterprise applications.
The VMware vFabric GemFire trial software doubles as the no-fee developer software. Limited to 3 node connections, the trial software used for development is otherwise fully functional.

Gigaspaces
GigaSpaces In Memory data grid  is much more than a memory based caching solution, it is a scalable enterprise grade system of record that supports complex data models and provides extensive set of features for queries, data manipulation and data store management. XAP Elastic Caching Edition is compatible with Database Data access APIs such as SQL, JDBC and JPA, Free trial avalible

SAP HANA
SAP HANA is a completely re-imagined platform for real-time business, no trial on the website

Gridgain
GridGain is Java-based middleware for in-memory processing of big data in a distributed environment. It is based on high performance in-memory data platform that integrates world’s fastest MapReduce implementation with In-Memory Data Grid technology delivering easy to use and easy to scale software. Using GridGain you can process terabytes of data, on 1000s of nodes in under a second.
GridGain typically resides between business, analytics or BI applications and long term data storage such as RDBMS, ERP or Hadoop HDFS, and provides in-memory data platform for high performance, low latency data processing and computations.Download avalible on the website


Sources:
Gridgain

Tuesday, December 18, 2012

In-Memory computing



In Memory Computing
Facts according to Gartner

  1. 64bits processor can address - up to 16 exabytes of memory
  2. DRAM prices drops 30% every 18 months
  3. 1gb of NAND flash memory avarage price is 0,84$ in 2011
  4. Commodity blades provides 1 terabyte of DRAM
  5. Multicore CPU enables parallel processing of In-memory data

History if RAM types

The two main forms of modern RAM are static RAM (SRAM) and dynamic RAM (DRAM). In SRAM, a bit of data is stored using the state of a flip-flop. This form of RAM is more expensive to produce, but is generally faster and requires less power than DRAM and, in modern computers, is often used as cache memory for the CPU. DRAM stores a bit of data using a transistor and capacitor pair, which together comprise a memory cell. The capacitor holds a high or low charge (1 or 0, respectively), and the transistor acts as a switch that lets the control circuitry on the chip read the capacitor's state of charge or change it. As this form of memory is less expensive to produce than static RAM, it is the predominant form of computer memory used in modern computers.
Both static and dynamic RAM are considered volatile, as their state is lost or reset when power is removed from the system. By contrast, Read-only memory (ROM) stores data by permanently enabling or disabling selected transistors, such that the memory cannot be altered. Writeable variants of ROM (such as EEPROM and flash memory) share properties of both ROM and RAM, enabling data to persist without power and to be updated without requiring special equipment. These persistent forms of semiconductor ROM include USB flash drives, memory cards for cameras and portable devices, etc. As of 2007, NAND flash has begun to replace older forms of persistent storage, such as magnetic disks and tapes, while NOR flash is being used in place of ROM in netbooks and rugged computers, since it is capable of true random access, allowing direct code execution.
ECC memory (which can be either SRAM or DRAM) includes special circuitry to detect and/or correct random faults (memory errors) in the stored data, using parity bits or error correction code.
In general, the term RAM refers solely to solid-state memory devices (either DRAM or SRAM), and more specifically the main memory in most computers. In optical storage, the term DVD-RAMis somewhat of a misnomer since, unlike CD-RW or DVD-RW it does not require to be erased before reuse. Nevertheless a DVD-RAM behaves much like a hard disc drive if somewhat slower.




All those facts make In-memory computing real

Everything in RAM

Advantages:
1. low latancy - < 1 microsecond
2. Enabling real time buisness intelligence
3. Dramatically shrotening batch execution time

What it gives us?
The ability to cache countless amounts of data constantly. This ensures extremely fast response times for searches.
The ability to store session data, allowing for the customization of live sessions and ensuring optimum website performance.
The ability to process events for improved complex event processing

In-memory is based on appliance solutions that combine software with optimized hardware

Data analyses speeds can be accelerated by up to 150 times

Queries with 500 billion data records in response times of less than one minute are possible 

Portfolio Scatch


                          




Source
Gartner
Wikipedia


Monday, December 17, 2012

Lets discuss future predcitions for IT and software

According to Gartner:


By Year-End 2014, three of the top five mobile handset vendors will be Chinese.
By 2015, big data demand will reach 4.4 million jobs globally, but only one-third of those jobs will be filled.
By 2017, 40 percent of enterprise contact information will have leaked into Facebook via employees' increased use of mobile device collaboration applications.
Through 2014, employee-owned devices will be compromised by malware at more than double the rate of corporate-owned devices.
By 2016, wearable smart electronics in shoes, tattoos and accessories will emerge as a $10 billion industry




CISCO

By 2029, 11 petabytes of storage will be available for $100—equivalent to 600+
years of continuous, 24-hour-per-day, DVD-quality video.

In the next 10 years, we will see a 20-time increase in home networking speeds

By 2013, wireless network traffic will reach 400 petabytes a month

By 2020, a $1,000 personal computer will have the raw processing power of a
human brain

By 2030, it will take a village of human brains to match a $1,000 computer.

By 2050 (assuming a global population of 9 billion), $1,000 worth of computing
power will equal the processing power of all human brains on earth.

Today, we know 5 percent of what we will know in 50 years. In other words, in 50
years, 95 percent of what we will know will have been discovered in the past 50
years.

By 2020 worldwide, the average person will maintain 130 terabytes of personal data
(today it is ~128 gigabytes).

By 2030, artificial implants for the brain will take place.

By 2020, universal language translation will be commonplace in every device


With IPv6, there will be enough addresses for every star in the known universe to
have 4.8 trillion addresses
 


CorbinBall

 New indoor positioning options will provide better event and exhibition indoor way finding and mapping.


Sources
Cisco
Gartner predictions
Corbinball