The RSP selected Atos Origin to design, build and operate the UK rail settlement system, which allocates more than £3.5 billion in annual passenger revenue to Britain’s passenger train operators.

The SAND solution has delivered a number of benefits to Atos Origin, the most important being that it has enabled the company to cost- effectively meet its contractual requirements and stringent SLA’s. The SAND solution has also brought a massive reduction in the overall cost of data management by efficiently partitioning the active data accumu- lated by the ATOC between an online server and a nearline archive from which it can be restored quickly and easily.

“We found that SAND prod- ucts were the natural choice for this project because, put simply, they allowed us to effectively archive, rapidly access and extensively analyze huge amounts of very complex data.”

Get the case study

To receive a PDF copy of this case study by return email, please complete and submit the following form:

“Enterprises want to take structured data – customer and transaction data – and combine it will [sic] all the unstructured data coming off their websites…that might not fit into a tabular schema well.” [...] “All of that activity is captured in web logs that can’t easily be digested using existing relational systems.”

I couldn’t agree more with the premise and I’m glad Mike is prepared to mention the elephant in the room. Given who he works for and given that Netezza is a relational database in a very fast box, that would seem fair.

I think of this as lawnmowers and duct tape. If you strap enough lawnmowers together they will go pretty quick. They won’t handle very well, the safety record’s spotty, they burn a lot of two-stroke, and keep running out of gas, but still they go fast in a straight line.

Again Mike is right when he says customers want to be able to merge this data with other data types such as customer and transaction data. His conclusion, however, is just plain wrong.

The answer isn’t to process some data in the cloud and then some in an appliance. The answer is to put them all in a common store that can process both.

Hadoop is a great technology. Cloudera is doing a great job and adds a lot of value. But the particular problem Mike refers to would be much better handled by putting all the data in a single Column-Oriented Database Management System (CDBMS).

Why push your data into the cloud, analyze some of it, pass some of it down into an appliance, analyze some more of it, and then start all over again? Even putting aside questions of bandwidth, security, and performance, it’s simply not efficient. You can pound screws with a hammer and wrench rather than using a screwdriver, but why would you?

Given the costs involved and results demanded, enterprises are looking to use the right technology for the right job. The right technology for this job efficiently combines all the data in one place and is optimized for performance.

The right technology is SAND CDBMS.

[The Register]

“There are a lot of people talking about massively parallel editions and lots of people talk about the cloud,” said Pilcher, adding that scalability doesn’t factor into the discussion. Customers operating 24/7 at the global level, for instance, will need the ability to shift processing capacity to certain regions at certain times, said Pilcher.

Pilcher also spoke about the continued expansion of enterprise data towards mobile access and cloud architectures:

“(Overall), it’s all about pushing data out into the ether [while] ensuring it is secure and ensuring despite bandwidth concerns you can still get access to it,” said Pilcher.

• Full itWorldCanada article
• SAND CDBMS V6 press release
• Mike Pilcher’s blog

Meanwhile, some of our competitors are waxing lyrical about how smart their first-generation technology has become, now that they have discovered and developed… wait for it… MPP. SAND architected MPP into our database from the outset — and our outset was 20 years ago, not barely 20 months.

Anyone can sit around smoking in their marketing departments with clouds billowing onto every PowerPoint deck, and without a doubt a number of our competitors massively do this and do it in parallel. It is an accepted fact that if you want to deliver an enterprise-class scalable solution in the cloud, it needs to be built using parallel technology. What makes the difference is how you use your scalability. The “cloud” is actually a very good analogy for some of our competitors, because what these vendors are selling looks light and fluffy from the outside, but inside turns out to be cold, wet and a long way from the ground.

What you need to navigate the cloud is robust, proven technology. Unlike other vendors in the market, we at SAND know what we are great at doing — providing analytic databases to support deep and wide analysis of big data. SAND is not trying to put Oracle out of business, nor is it trying to focus on developing OLTP databases. We are delighted dear Mike Stonebraker of Vertica agrees with us that the relational OLTP paradigm is completely unsuitable for analytic applications: using traditional technologies designed to support online transaction processing for analytics is like trying to build a car for the Indy 500 by stringing 700 1-hp lawnmowers together with duct tape.

Unlike Mr. Stonebraker, however, we don’t dream of building a better mousetrap for OLTP. Oracle does a great job of OLTP — they’ve kept Larry in designer suits and big yachts for years. At SAND we focus on the databases people use to make real-time operational business intelligence decisions. Some vendors have got good at duct-taping lawn mowers and putting them together in a big box (if people pay you enough you can buy good lawn mowers and the best quality duct tape). However, this doesn’t change the fact that you are still working with lawn mowers and duct tape.

At SAND, our goal is to deliver more information into the hands of users, as fast and as easily as possible, using existing hardware and driving unprecedented volumes of data. We have been doing this for over 20 years. In the past, unfortunately, we tried to be the ultimate stealth marketing company. We decided our technology was so good we didn’t need to tell anyone about it. We thought that customers would somehow find us through the database equivalent of osmotic viral marketing. While that is a great concept for the future, one I am sure we could get VC funding for (“osmovirus” – doesn’t that sound like it should be a Web 2.0 company?), we are in the present, and it’s time to tell people what our customers have known for years: that SAND simply has the most visionary development team, building the most scalable enterprise-class, mission-critical deep analytic databases on the market.

SAND’s Agile Cloud platform delivers seamless scalability, deep and wide across processors. We ensure that every available resource is fully used to deliver optimal performance to users. We deliver virtual database instances on the fly, with what we call “SAND boxes”: a user easily creates a SAND box, puts their data in it, uses it and moves on, then the SAND box is discarded and the computing resources are made available for the next user. Predictable linear scalability is the core benefit we set out to deliver with our software, and we have done precisely that.

Really good software should be predictable in using all the resources you want it to, when you want it — without the need for someone hanging around to hold the code together with duct tape and glue, or requiring a massive amount of dedicated hardware to be kept waiting for a user to ask the right question. We all know about software vendors who only manage to deliver performance benchmarks by stripping the data — oiling it down, and then massaging it to such an extent it looks like it was worked over by an East German physiotherapist shot-putter — and then sticking it on a box so big, the lights dim in Des Moines whenever they run a query. That’s not innovative — that’s 700 lawnmowers in a box.

SAND software works fast because it optimizes all aspects of disk, memory and CPU performance by seamlessly using every piece of computing power you want it to. That’s smart computing. It’s why I like software, and it’s why I joined SAND. Remember, if it isn’t smart computing it’s just plain dumb.

SAND V6 is smart, and it’s the future of business intelligence in the agile cloud.

SAND CDBMS (column-oriented database management system) is a specialized Business Intelligence (BI) software platform that streamlines and accelerates the development and deployment of high-performance multi-user analytic applications. Unlike repurposed online transaction processing (OLTP) systems, SAND CDBMS imple- ments a patented data structure specifically designed to provide speed and responsiveness for large and complex analytic tasks. This column- based, tokenized, bitmap-indexed data structure offers very efficient storage of data at a granular level, and provides 100%-indexing of the data without requiring imposition of specialized schemas. At the same time, it appears to users as a standard relational database and fully supports standard SQL.

This paper describes in detail the architecture of SAND CDBMS and shows how its various features can work as part of a powerful BI environ- ment for organizations of any size that need fast and flexible analytical access to masses of corporate data.

Get the white paper

To receive the white paper by return email, please complete and submit the following form:

Elastic Private Cloud Architecture

On-demand resource allocation ensures efficient use of shared processing and storage resources to deliver an agile and secure private cloud architecture that is fully transparent to the end-user community.

Augmented Data Security/Privacy Options

SAND CDBMS 6 extends the high level of data security offered by the internal architecture of SAND CDBMS by adding new algorithms for encrypting specified data at the column level. This option provides robust protection against external or internal intrusion, and supports compliance with PCI standards.

Advanced Text Search Functionality

SAND CDBMS was the first columnar database to offer the ability to blend text search functionality into standard SQL commands. Version 6 extends these capabilities with “relevance ranking” functionality along with support for SPELLED LIKE and SOUNDS LIKE comparisons. This functionality is essential for complex analytics on textual data such as customer feedback.

High-Performance Joins

Executing joins has always been an expensive operation for any RDBMS. SAND CDBMS 6 introduces a new technique to provide the high level of join performance associated with materialized join indexes, without the management and disk space costs usually associated with them.

Rich Mathematics Package

SAND CDBMS SQL syntax has been augmented with new mathematical functions specifically designed for complex data mining processes.

Distributed Load Process for Unlimited Scalability Addition of a new distributed load process permits a high level of load process scalability, enabling unlimited speed for data loads.

Mobile Computing Support

With SAND CDBMS version 6, enterprises with mobile users can easily manage the movement of data from the central data warehouse to local personal data marts on small personal computers such as a netbooks.

“We are very proud of the new version of our CDBMS product,” said Richard Grondin, Chief Technology Officer at SAND Technology. “For many years, SAND has been a pioneer in the Columnar Relational Database market, and our product is now really mature. SAND CDBMS version 6 introduces new functionality that will enable any enterprise to take maximum advantage of Columnar Database technology to satisfy their complex analytic requirements.”

“On-line transaction processing systems (OLTP) were designed for storing rapidly changing, volatile data, not for high-performance analytics on slowly changing, operational data,” said Mike Pilcher, Chief Operating Officer for SAND. “Transactional database vendors cannot address the challenge of making legacy software handle the massive data volumes, user populations, and complex analytics requirements of today’s enterprises. With over 15 years of research, development, and experience behind it, SAND CDBMS 6 was built from the ground up to handle these requirements simply, with sub-linear scalability, and with little to no disruption to existing systems and resources.”

Well, during a recent trip to provide training for a global energy company, one of our senior solution architects, Steven, sent me back the following:

Mike,

Aside from thinking SAND CDMBS Nearline for SAP was generally fantastic — [the customer] couldn’t stop talking about how much it made sense, especially when it comes to BWA and the new in-memory database.

I know we see tremendous value here, but it’s another thing entirely when our customers keep repeating it. [The customer's] point was — since nearline makes their online objects smaller, they can now add more of them to the existing BWA infrastructure with no additional costs.

Also, preparing for the new in-memory database — which will merge BWA with the online data — would typically create the need for huge hardware with lots of horsepower and a giant price tag to go with it.

With SAND CDBMS Nearline for SAP, however, the amount of online data is drastically reduced — while still staying immediately accessible — decreasing the hardware requirements, controlling costs, and improving performance.

Part of the performance boost comes because, with a smaller foot-print, even more objects can be kept online within the existing hardware. In addition, as new data continues to accumulate, they can seamlessly move the older data to nearline. That means the online system remains stable and users retain access to more data than before, with increased performance and lower ongoing costs. Of course there is the increased performance for backups and the nightly batch processes, but that’s another groups gain

Steven

It’s not the first time we’ve heard this kind of feedback from our customers. It’s not even the first time we’ve heard it as early on as the training phase. If you’re running SAP BW and are looking to enhance BWA and/or in-memory database, we’d love to hear it from you as well.

This white paper by Bill Inmon, father of the data warehouse, examines the value of “nearline” storage in the context of the information lifecycle. At the foundation it is a discussion of storage hierarchy and the early lifecycle of data in the corporation. It reviews the dynamics of disk storage; separation of data and the effects the process creates; the impact of the full lifecycle of data; and the limitations of the traditional DBMS to handle this activity.

The value of this new “nearline” technology will be reviewed in the context of storing data from any structured data source in a highy efficient software repository, managing bi- directional data flow, and enabling high-performance access to nearline data. Today – with the introduction of SAND Technology’s SAND CDBMS Nearline – a major step forward matches data’s lifecycle to technology through “nearline” storage.

Get the Bill Inmon SAND CDMBS Nearline ILM white paper

To receive Bill Inmon’s SAND CDBMS Nearline ILM white paper by return email, please complete and submit the following form:

1. Big data.

When the size of your SAS environment is starting to grow, it’s a sign you need SAND CDBMS Nearline. Data growth can creep up on you — even a terabyte of SAS data, once it’s indexed and prepared for analytic use, can balloon up by a factor of 3. If that data is replicated in production, development, and backup environments, it could be 9 times the original size before you roll it out. And when that original size grows to 2 terabytes, 5 terabytes, 10 terabytes, and beyond…? As I wrote in my last blog post, storage is getting cheaper but bandwidth isn’t. Moving around multiple terabytes isn’t cheap. SAND CDMBS Nearline for SAS has reduced our customers’ multi-terabyte environment down to gigabytes — 98% compression. Like any diet, however, it’s much easier to get control early on than to try and deal with it after it’s become a huge problem. Putting SAND CDBMS Nearline for SAS in place lets you keep massive amounts of data instantly accessible, in a small footprint, and gain all the extra benefits of agility and flexibility that come with it.

2. Big ETL

When you have issues with ETL processes, or you want to enhance the value of data by adding elements like demographic information, it’s a sign you need SAND CDBMS Nearline for SAS. With web log analytics for example, we’ve seen performance improvements that took the ETL process down from 24 hours to 30 minutes. And this is fully scalable. So if making batch windows and meeting SLA agreements are important — and we know from our customers that they are — SAND CDBMS Nearline for SAS provides significant additional value.

3. Big analytics.

When you need to get the most productivity possible out of your SAS analysts, it’s a sign you need SAND CDBMS Nearline. One of the benefits of SAND CDBMS Nearline for SAS is that the smaller footprint greatly improves I/O performance, which significantly improves the overall performance of the SAS environment. Query speeds also benefit — we’ve seen real world speed boosts of over 7 times and productivity increases of over 3 times. In addition to that, eliminating the need for ever-increasing indexing projects and removing ongoing data-preparation delays, lets your hard-to-find, harder-to-train analysts spend their time doing analysis. Your analysts get access to more data, so they can shop for it when they want, and slice away at it how they want.

4. Big change.

When you are dealing with new projects, new customers, mergers and acquisitions — anytime you want to integrate more data from different data sources, — it’s a sign you need SAND CDBMS Nearline for SAS. Not only can SAND CDBMS Nearline connect to almost any data source, it can run on Windows, Linux, or Unix, so it’s platform-agnostic and can fit into any architecture, be it legacy, inherited, or newly acquired. And because SAND data is stored in read-only format, your original data remains intact in compliance with corporate policy or government regulation. Analysts know the data they’re working on is the same data that was brought into the system. SAND also manages the Information Lifecycle Management (ILM), so specific pieces of information can be kept live in the online system for a pre-defined period of time and then automatically brought into SAND CDBMS Nearline.

5. Big updates.

When you migrate to a new version of SAS and need to re-open SAS data sets to do a conversion, it’s a sign you need SAND CDBMS Nearline. A powerful tool to simplify the process, once SAS data is moved to nearline, the resulting reduction in the footprint of the SAS environment (up to 98%) means a quicker and more efficient migration. Afterwards, data stored in SAND CDBMS Nearline for SAS remains just as accessible as it was before.

One more thing…

In my last blog post I talked about some of this as it pertains to one of our customers, dunnhumby, who handles retail analytics for over 350 million retail customers on behalf of companies like Tesco, Home Depot, and Kroger. In my next blog post I’ll tackle the final piece of our SAND CDBMS Nearline for SAS offering — result sets.

Montreal, Canada – June 24, 2010 SAND Technology Inc. (OTCBB: SNDTF.OB), provider of leading column-based database software, today announced the launch of its SAND CDBMS Nearline product for SAS environments. Based on SAND’s patented column-based database technology, SAND CDBMS Nearline for SAS efficiently stores and manages the large amounts of supporting data kept in SAS data sets, flat files and databases, making this information easily and quickly accessible on-demand for ad hoc analyses in SAS.

“Traditionally, making large volumes of corporate data available in a form usable by SAS has been very costly,” said Richard Grondin, Chief Technology Officer for SAND Technology. “Specific infrastructures had to be built to hold the data, management requirements were extensive, and the time required to identify, transform and extract the data for detailed analysis was longer than users were willing to wait. SAND CDBMS Nearline for SAS is designed to solve these issues by making all data immediately accessible for analysis without requiring extensive administration, and without displacing existing software, data management, operations and hardware investments.”

“In providing analytics on point-of-sale data for some of the world’s largest retailers, our biggest challenge is the sheer volume of data that we have to manage. We need a solution that is scalable, that can work on just about any hardware and which gives us good response times,” said Clive Humby, chairman and co-founder of leading customer analytics provider dunnhumby. “SAND CDBMS Nearline for SAS is very valuable to us. We use SAS as our analytic tool for retail data and SAND fits very nicely with this technology. It gives us fantastic performance at very modest cost, and enables us to build our tools in a scalable way.”

SAND CDBMS Nearline for SAS is natively integrated with the SAS environment. Data is saved with full definitions extracted from SAS, and can be accessed using data step or proc statements. SAND CDBMS Nearline supports automation of data migration and lifecycle management processes based on relevant business rules, and uses advanced data encryption algorithms to ensure protection of data privacy. SAND’s data storage requirements are typically 90% of flat file size, and the system can scale linearly using commodity hardware to support any volume of data.

For more information about SAND CDBMS Nearline for SAS, visit www.sand.com.