Tony Bain

Top 10 Data Management Issues for 2009

So it’s that time of year again when everyone puts out their predictions for the year ahead.  I think predictions are a bit of a waste of time because to be interesting predictions have to be big, but a year really isn’t all that long so actual changes over the course of 2009 are likely to be just small progressions.  So instead I have been thinking about the top issues that we face heading into 2009 and here is my Top 10 list for issues in Data Management.  In this post I avoid offering solutions to these issues, while I have several ideas on solutions these can be the subject of subsequent posts.

10 - Limits on Scalability

While scalability is on my list it is at number 10 because against popular belief, scalability is only an issue for a very small number of data based applications.  Almost all data based applications in use today can be scaled without major issue by increasing the underlying hardware resources.  But for those applications for which it is an issue it is usually a major issue, and the most common category of data based application that has such scalability limitations is internet applications and web based services.

The problems with scaling this type of application are firstly, the scalability requirements are hard to predict in advance and secondly they can change instantly based on sudden popularity (aka the Slashdot effect).  So currently you either over invest in infrastructure and hope for the growth or you under invest and hope you have time to rapidly add capacity if/when required.  When planning “corporate” applications you usually have the benefit of capacity planning projections and the ability to manage rollouts to ensure scalability limitations can be managed and while you try to do this in a web world by nature any projections you do today can be meaningless tomorrow. 

Some cloud database platform vendors are looking to help alleviate this by providing solutions that offer on demand scalability.  These solutions come in two flavors, firstly running existing database platforms on a virtualized infrastructure and secondly providing a non-relational key/value pair repository (SimpleDB, SSDS etc) where all underlying infrastructure and database operations are abstracted.  While these represent positive steps forward I think the issue of completely abstracting infrastructure scalability to an application remains an issue that will receive attention in 2009.

9 - Constantly changing landscape

Vendors are changing/updating platforms faster than organizations and their application vendors are willing or able to respond with migrations to the new platforms.  When combined with applications that only work with a particular vendor many organizations have multiple data management platforms with multiple versions of each platform in production use.  For example, enterprise organizations that use SQL Server often have SQL Server 2000, 2005 and 2008 servers in production (as well as multiple flavors of Oracle, DB2 and so on).  Different vendors and different versions within vendors create a major management overhead.  This impacts areas such as consolidation, required skill sets, management processes/policies, tool sets.   It also restricts the level of value adding exercises that are taking place by data management teams due to their limited resources being consumed with functions surrounding operational management.

8 – Data Recovery

Recovery is something that people have focused on since we first starting collecting electronic data so how is this still an issue?  Well as the demands for electronic data grow the factors that control how data is recovered in the event of failure become more complex.

It is easier to recover an entire data set than just a specific part of it.  This is fine if you lose an entire data set but not so fine if you need to recover only a part of a data set.  New factors such as pressure to have 24x7 availability and an increase in purely electronic transactions with no paper “backup” mean that recovery of individual records/transactions presently can be a very difficult and time consuming process.  Due to the complexity of doing transaction based recovery, most database applications no longer “delete” any data instead you a status flag to indicate that data is in a deleted state.  Changes to the data are also logged as change history which is also infrequently actually deleted.  This allows for application based recovery of that transaction but significantly contributes to the increasing data volumes being experienced.   Methods for ensuring individual pieces of data can be rolled back without keeping it within the highly used data set perpetually is a complex issue that will take time to resolve.

7 - Increasing Data Volumes

Data Volumes are growing rapidly.  This graph shows the average database size across 3000 databases for ~ 30 random organizations.  This graph shows a 12% increase in the average database size in the last 6 months of 2008.  

 

While storage is cheap such rapidly increasing data volumes brings issues other than just pure storage costs.  Backup/recovery time frames, maintenance timeframes, query/access performance, increased CPU & Memory requirements (for batch processing etc).  This would be less of a concern if the data was adding value (see the low leverage of data topic above) however today in most database applications, much of the data contained is non active historical data that is there solely for the purpose of future reference should it so be required.  Managing the vast volumes of data to ensure the balance between data availability and manageability is maintained moving forward presents an interesting issue.

6 - Lack of trained people

Finding qualified and experienced people in IT has always been an issue, in the area of data management this is especially true.  Data management is a niche field and the rapid increase in requirement for data management experts has not been met by an increase in the number of skilled people in the workforce.  This has led to skills shortages and organizations being forced to take progressively less experienced and qualified staff onboard.

Lack of skilled resources has a flow on effect which impacts all aspects of data management including performance, security, availability and recoverability.  Managing an increasingly complex set of data management problems with a depleting set of expertise is another interesting issue heading into 2009.

5 - Inefficient use of Resources

Modern database systems are typically implemented in a manner which sees a physical server provide a platform for one or a small number of applications.  This approach has had its benefits from an implementation perspective including, the complete costs (hardware, software, license, management) can be easily calculated and assigned to a project/department, the risk of impacting other applications is low, sizing an environment is easier as only a single workload is taking into consideration and so on.  However due to the continual increase in applications this has lead to large numbers of database servers being implemented.  There are commonly dozens of such servers in small organizations, hundreds in medium sized organizations and even thousands of database servers being implemented in large enterprises.

One of the several issues with this approach is that it leads to significant inefficiency in resource utilization across the entire infrastructure.  If we define “resource usage” to mean the percentage of available CPU, Memory and I/O capacity in a server we can graph a typical analysis of usage in an enterprise: 

 
 
This shows us that a typical picture is around 50% of all servers have a 10% or less average resource usage (i.e. half the servers are only having at most 10% of their available resources used).  30% have between 10 and 20% usage and so on down to 5% of all servers having 90% or more of their available resources utilized.  This means across the entire environment there is an average resource usage of 32% meaning 68% of resources are not being put to work.

4 - Auditability

Organizations are under increased pressure to audit every action that a user performs within a database.  This is due to increased focus on security, risk, accountability and avoidance of fraud and corruption.  While security prevention measures (logins, firewalls, tokens etc) are important to prevent unauthorized access to the data in the first place, as this survey shows, most breaches occur by users who are authorized but are either negligent or malicious.

 
image from ars technica

The problem with auditing is that itself generates a lot of data, potentially much more than even the database that it is monitoring.  Also the act of auditing a database can cause a significant performance load on the database being audited.  I think this is a particularly important issue facing enterprise organizations especially heading into 2009.

3 - Data Security

Security is always an issue for data.  The biggest issue facing data security I think is not actually securing the information that is contained within a database, but ensuring security is maintained on the data once that is has its original data source.  As we expand ways in which a piece of data may be consumed or modified, through methods such as API’s, web services and other integration means, currently it is up to each step in that integration chain to ensure appropriate security is provided. 

To date this is typically controlled by users making access decisions for services based on the data they own, and a mixture of specific copy/access protection systems in some cases for data the users don’t own (such as DVD’s, music and software).  However heading into 2009 a significant issue around how universally groups of data can be secured and have that security survive distribution remains open.

2 - Decentralized Data Management

When we used to talk about data management we almost always were talking about database management.  Even today this is mostly the case.  But this is changing.  Data is becoming more distributed and the source of data to a “data consumer” may not necessarily be a database but instead may be a “data service”.  How the data service is made up and if it has a database under the covers is often irrelevant to the consumer as the service rightly abstracts the underlying architecture.

Data services are commonly used today as data sources or destinations and data is pushed and pulled from service A to location B though integration processes (e.g. old world ETL processes).  Due to data volumes and the need to have timely data, pressure is mounting to instead build applications that use the data services directly as the data source and perform any necessary integration in real time.  This reduces unnecessary data duplication and increases the timeliness but creates a whole set of issues to do with consistency, recoverability and availability that will require resolution.

1 - Low leverage of data assets

Organizations have a lot of data.  A quick survey across some random business finds that on average a SMB has about 500GB of data in total in their database systems.  An “enterprise” is much more difficult to average as this will range from the 10’s of TB well into the PB’s (PetaBytes).  And of course there is large volumes of data outside of the database systems in files, email etc also.  But at the moment for most organizations much of this data is used purely operationally, which means the data is being used for the application in which it was created but no additional value is being derived from this data through wider analysis.

It used to be that competitive advantages were gained by moving inefficient manual processes to more efficient alternative through the use of technology, heading forward I think these gains will be less and the larger competitive gains will come from using the collective knowledge to understand and serve customers better.

And that is my list of the top 10 issues in data management.  I had a starting list of about 30 issues which I whittled down to 10, I am reasonably happy with this list but on a different day maybe a few in here would have been swapped out for some alternatives.  Anyway, I look forward to your comments.

NOTE: The reference data I use in this post is just data I have quickly pulled together or observed, it hasn’t been formally researched or validated and should not be considered fact.  This is a blog post not a research project!

Weny

Weny is a link shortner that is used to associate links with your twitter id.  Links remain associated with you even if they are forwarded or retweeted etc.  Weny has a lot of ideas in the pipeline, predominately in the area of link tracking data analysis.

http://weny.ws

SSD and DB usage

Image via Wikipedia

Recently I have seen several articles about Solid State Drives (SSD) and some articles are picking 2009 to be the year of the SSD.  While I agree SSD’s will bring some great benefits in the years to come, I wanted to clarify the current state of SSD’s in relation to usage in the database field.  A primary misconception is that because SSD’s are a type of “memory chip” storage that they are super fast, like system RAM and using them to house a database will make the database super fast.  SSD’s are very different from system RAM because of course their data must be persisted and this requirement carries a major performance impact. 

SSD’s offer high read performance benefits over current mechanical (magnetic) HDD’s.  As much as 20x’s as fast.  However write performance on SSD’s is slower than current HDD’s, in fact HDD’s can be up to 10x as fast to carry out random writes when compared to SSD’s.  This is due to a number of limitations involved in writing to persisted solid state memory, including the fact the SSD sectors cannot be overwritten but instead the whole sector has to be erased and rewritten. 

The benchmarks available are a problem and it is difficult to get clear results.  This is for a few reasons.  Firstly, because both the SSD and HDD are two technologies undergoing rapid evolution any benchmark published quickly becomes out of date and irrelevant.  Secondly the benchmarks are often published by advocates of either technology rather than completely independent bodies.  Some benchmarks I have seen are showing the speeds of high end SSD’s compared with low end HDD’s and vice versa.

So when considering SSD use for improving Database performance, make sure you consider the following points. 

• SSD’s generally have better read performance than HDD. But make sure you are purchasing current SSD technology as you may find older SSDs have slower read performance than current HDDs.

• SSD’s generally have worse write performance than current HDDs.  If you have a write heavy workload and your silver bullet is SSD you may be disappointed.  There are exceptions but ensure you are comparing current with current.

• If you are able to split data based on usage patterns such as partitioning or file placement, moving highly read, low write data to SSD storage can provide significant performance improvement.

Some proponents are picking that before we see pure SSD storage replace existing HDD’s we will see the emergence of hybrid devices that contain SSD and HDD technologies.  These devices will have hardware algorithms that make a determination about where a particular sector should site based on its usage profile.  Highly read low write sectors would sit on SSD and highly written sectors on HDD, of course all of this transparent to the system using such a disk.  If such a hybrid device actually becomes mainstream or is skipped altogether due to rapid advancement of SSD is still to be seen.

Related articles by Zemanta

• Intel, Hitachi to develop solid-state drives
• SanDisk cranks up solid-state drive speed
• Solid-state drives: No rush to widespread success

RockSolid Cloud Services Edition

  

Today we have announced our Cloud Services Edition of RockSolid.  This edition is specifically targeted to organizations that provide SQL Server services in the cloud from in a hosting or managed service capacity. 

RockSolid Cloud Services Edition is designed to allow for the standardization of monitoring, automated management, automated performance optimization and automated issue resolution across diverse environments that may be made up of thousands of SQL Server instances and tens of thousands of SQL Server databases.

The focus of the product is to significantly reduce operational costs associated with managing very large numbers of instances and databases through the automation of all routine DBA management tasks including performance optimization.  The policy driven framework also allows for provision of very high service levels guarantees to made to end customers with little associated overhead on the service provider in doing so.

RockSolid Cloud Services Edition has a web portal interface which allows end user customers to log in and review the state of their SQL Server databases from a performance, reliability, recovery and maintenance perspective.  The same interface when logged in to by the service provider allows the management of their SQL Server environment as a whole in a consolidated view.

Australian National Broadband Roll Out

Image via Wikipedia

As you may have heard Telstra, Australia’s largest Telco and ISP and the provider of almost all current physical telecommunication network assets across the country, has been excluded from the governments tender process for a $4.7b proposal to roll out a new broadband network.  The reason for this exclusion is apparently because Telstra left out one of the key requirements in its proposal, details on how it would make use of Australian and New Zealand SMB’s during the network roll out.
I am personally against the government’s broadband rollout plan.  For me this is not the role of government.  Every other example I can recall of government supported monopolies produce terrible results, even when those monopolies exist in the private sector but are government supported.  Typically those monopolies provide poor service, have long delays in acquiring services, are slow to upgrade services and are slow to innovate on top of such services.  The only fix for such monopolist habits is to break the monopolies by introducing completion.  The government’s objective shouldn’t be to fund the roll out of infrastructure itself but instead the government should manage regulation and resolve impediments to creating a commercially viable environment for such services, and leave the implementation of such services up to the commercial sector.

We have a current example of how having a competitive retail layer built on top of a monopoly infrastructure layer doesn’t work.  All the current ADSL services are built on top of Telstra’s existing copper network and while retailers have lots of competition and different pricing and engagement models, underneath we have a service with no price pressure, and little competitive push to innovate or add value.

This on the surface might seem a bad thing for Telstra but I have a glimmer of hope that it will actually provide some level of competition in the future.  Unless Telstra wants to give up its overwhelming market position as the country’s largest telecommunications infrastructure provider I can see opportunity for Telstra to expand its own network to be competitive with the governments network roll out.  Apparently other tender bidders have requested the Government to legislate to prevent Telstra from constructing a rival broadband network.  If the government decided to do this, this would a shocking day for the free market economy and a dark day for consumers.  Instead the government, if they continue with their own broadband network rollout, must ensure they are providing the same level playing field for any other provider who also wants to roll out a competitive broadband network.

Why PC’s still suck

Image via Wikipedia

Yesterday afternoon I was showing my son some funny songs he liked on YouTube.  The last one was the “Star Trekking Across the Universe” song, anyway near the end of it my laptop hung.  This started a 5 hour process to get it back up and running again.

Ok it’s hung, hard power off and restart.  Normally will fix.  Unfortunately on restart I got a Blue Screen of Death (BSOD) and an instant restart (before I could read the error message).

Ok, let’s try that again.  Restart, another BSOD and instant reboot.  Dam

Ok, I haven’t had any other issues so let’s try the “Last know good configuration”.  BSOD, restart.

Now I am getting a bit concerned, I better boot into Safe Mode and get the files that I had been working on the last couple of days off.  BSOD, restart.

Hmm.  It won’t boot under any boot option.  Fortunately as down time is such a problem for me, I keep spare replica laptop on hand at all times.  My laptop is a HP nx9420, and I have another one exactly the same.  So Plan B was to swap the hard drives over and boot the failing PC’s laptop on a new laptop.  BSOD, restart.  Dam same issue.  But at least I could rule out everything from a hardware perspective other than an issue with the HDD itself.  Swap HDD back and get OK laptop online to assist with issue resolution process.

Plan C, investigate the error.  First thing to do was see the error so reboot and press F8 to get the boot options menu.  Choose the Disable Auto Restart on Error option.  Boots, BSOD but this time I can read the error.  UNMOUNTABLE_BOOT_VOLUME. Crap.

Google the error and find a fix process for the issues.  Basically start Windows XP from setup disk, go into recovery console, run chkdsk /P and if necessary fixboot.  Great let’s give this a go.  Hang on, these HP laptops don’t come with any CD/DVD’s.  Everything is on the special recovery partition on the disk.  Ok, reboot, press F11 to boot from the recovery partition.  Here is where I was expecting to have the option to start the recovery console.  No, after about 10 minutes of hour glass a single msg comes up which said basically “All data will be lost and your system will be reset to factory configuration”.  And it only had an OK button, no cancel!  Ok power off.  Aghhh.

Ok, not to worry.  Fortunately I have an MSDN subscription so on working laptop log in and download the retail version of Windows XP Professional with SP2 in ISO format.  Download takes about 30 minutes, burn it to DVD.  On failed laptop boot, press F9 to boot from DVD and wait for Windows XP setup to start.  Takes a few minutes but eventually comes up with the menu and the option of pressing R to start the recovery console.  Finally, high spirits at this point.  Press R.  “Windows XP can’t find any Hard Disk Drives on this computer, press F3 to exit”.  You got to be kidding me.

At this point it would have been very easy to decide that the HDD was dead and to give up hope.  But the PC did actually go through the boot process and did constantly fail at the same point so obviously it was able to see and read the HDD.  Also I have had a number of HDD failures in my time and almost always failures start small and get progressively worse rather than just drop dead one day.  So back to Google.

Google turned up some interesting results.  Seems this laptop has a SATA drive and the Windows XP setup doesn’t support SATA drives natively so you need to install an additional SATA driver during the set up initialization phase.  Visit the HP site, enter the model number and hooray, find the SATA driver I need, download it.  Run the EXE and it extracts and a message pops up, please insert the Floppy Disk that you wish to install the driver on.  Floppy Disk!  These laptops don’t have Floppy drives and I haven’t used a Floppy Disk in over 5 years.  Dam, am I beat at this point?  Hang on, I remembered that many years ago I had a Toshiba with a USB Floppy disk drive.  Where would that be now?  Ok by this time it is late and dark, but I manage to feel my way around the garage (the light was out) and to my surprise was able to locate the Toshiba USB drive.  Next trick, finding a floppy disk.  Found a box of all software of mine that was filled with old CD’s but down the sides of the box found 3 floppy disks in apparent ok order.  Woo-hoo moving forward again.

Ok, plug the USB Floppy into the working laptop, dong ding, which means it recognized it and now my laptop had an A: for the first time.  Try to insert the Floppy Disk into the drive and the disk gets about half way in and stops.  Won’t go any further.  Try a few disks and nope, something was blocking it.  Sigh.  Remove plastic casing from USB floppy drive and then popped off the metal cover the protects the floppy drives internals.  Found a plastic connector of some sort stuck in the drive.  Not sure where that came from, probably from the box in which the floppy drive was living for the last 5 years but anyway removed it and inserted the disk ok.

First disk came up as unformatted.  "Did I want to format?". OK.  Windows tries to format and after a few minutes says “Windows can’t format this disk”.  Why not.  Oh forget it, here is another disk.  Tried disk 2 out of 3 and this one was formatted and appeared to be read OK.  Started the SATA driver set up again, clicked OK and it wrote all the necessary files to the A:.  Yay!

On failed laptop, plugged in the USB drive, booted from CD and this time when setup was initializing pressed F6.  After a few minutes was given the option to press S to load additional disk drive which I did and, with some astonishment from me, it worked and loaded the necessary SATA driver.  Continued through to the Set Up menu as before and this time when I pressed R we successfully went through to the recovery console.

At the recovery console typed chkdsk /P.  It started checking stuff.  Went away and came back an hour later when it had finished.  Seemed to indicate it may have fixed something.  Rebooted and crossed fingers.  With some disbelief Windows started up normally and I was able to log on and everything seemed in normal working order with no apparent data loss.  Unbelievable.

If I was a mainstream laptop user I would have had one option.  Wipe everything and go back to factory configuration.  Even if I had taken it into the local PC repairer I am 99% sure they would have decided it was a HDD failure and charged me for a replacement and rebuilt from scratch. 

So this is why PC’s still suck.  It is these sorts of issues that continue to frustrate everyday people about technology.  The PC is always pushed very hard in areas of bigger, faster & cheaper but we also need to push just as hard in the area of reliability. 

My definition of a Cloud service

Image by akakumo via Flickr

There has been some debate on a few sites the last couple of days about what a cloud service actually is. 

My definition is as follows:

• Infrastructure is abstracted (from applications and users)
• Scalability is unlimited (near linear scale out)
• Environment is multi tenant, but provides isolation

 

ReBlog: On Why I Don't Like Auto-Scaling in the Cloud

George Reese posted the following on the O'Reilly Blog

"I enter into sales meetings getting clients excited about dynamic scaling only having to vigorously talk them away from the idea of auto-scaling. I just don't like auto-scaling."

My reply to this was as follows (I am re-posting this here as it is relevant to the chain of posts I am working on at the moment).

George, you are dismissing a logical concept, “auto sizing”, based on current physical implementations of that concept.  A model where we don’t have to worry about manually planning application resources requirements because they are automatically allocated based on needs makes sense and certainly seems to the right direction to be heading.  One can predict a likely point in the not too distant future where planning I/O and other resource demands in detail will cease to be relevant and service providers charging metrics, such as based on network traffic, will become outdated just as charging based on number of CPU instructions or CPU time ceased to be (and charging based on number of database transactions used ceased to be, and charging based on disk space is starting to be).  Capacity planning as a discipline is becoming more holistic as the detail becomes less relevant to us humans, the focus will be on planning the capacity of an “environment” and leaving the individual allocations of applications that run in that environment up to it.

So I will agree with you that Auto Sizing systems are not necessarily fool proof or should be trusted, but only for now.

Surviving the downturn lesson #73431

Image by kris247 via Flickr

While there have been hundreds of posts recently about how to stay positive and how to survive a down turning economy in the tech sector, I thought I would also add my own perspective as someone involved in the technology innovation for the past 15 years.

So a significant crisis in the global economy is going to affect the dynamics of almost all industries.  You all know change is a constant force in business and in the tech sector change is also a constant reoccurrence over much shorter cycles than a lot of other industries.  But in a significant economic event such as we are seeing now the change is even more rapid and unexpected than usual and those who are too big to meet the change quickly, fail to adapt or can’t adapt for other reasons are going hurt significantly if not fail completely.

Too many businesses try and ignore change.  I have countless organisations that were previously successful fail when change occurred, by attempting to ignore the change or ride it out.  Their approach was to keep trying to do what they had done before.  And when that didn’t work they continued trying to do what they were doing before but cut more and more staff, slash marketing and new initiatives until eventually this approach killed their ability to produce, innovate and generate sales and they went out of business.  On the flip side radicalism is risky also, I once observed a large timber company try and become a web site company when the bottom fell out of the lumber business with dire consequences.  Recognizing change, having the guts to acknowledge change and making changes in strategy to adapt to change is a difficult but an important process that is required to survive long term.  Of course slash the fat and cull unnecessary expenditure, this is also what your customers are wanting to be and will be looking for providers who can help them achieve this. 

Creating technology that significantly helps organisations do more with less should do well right now.  As an example, since the start of the current downturn we have seen interest in our RockSolid product steadily increase.  Our customers are trying to do what everyone is trying to do, improve efficiency through cost reductions and increasing value from existing assets.  For our customers DBA staff are significant costs and reducing/re-deploying or gaining additional value from them is a primary issue.  Our product helps remove the high staff cost component from the operational roles (low value producing roles) allowing the staff to be used in a higher value generating capacity.  The managers in the field themselves are driving this initiative as they know if they don’t they will likely just receive a blanket order to cut staff by n% which will result in reduced costs but also a reduced ability to deliver the current level of service.

Doesn’t all technology look to reduce costs or increase value in some way?  Sure in theory, but the characteristics of what will do well right now is more specific.  The current climate has come on quickly and a lot of businesses weren’t expecting it and weren’t prepared.  Therefore people are looking at improving efficiency (without destroying value) using solutions that have an immediate or short term path to ROI.  It is too late to put in a new system that will improve efficiency in 2 years if you have significant cost pressure now.  The ROI that you are offering to your customers has to be proven, significant and measurable.  By nature SaaS offerings are well positioned to gain ground.  Most SaaS offerings have a low capital outlay (a monthly fee usually) and a quick path to producing a positive ROI so are by nature much lower risk than a full perpetual purchase of a product that may take 2-3 years to see that realized.

So my advice at this point is simple.  Improve your own efficiency but don’t lose your ability to produce/compete.  Put a focus on your existing technologies that provide significant short term returns.  If those returns are not clear then invest in making them clear through customer case studies, market analysis etc.  If your current technologies have a high initial investment cost (infrastructure etc) look at ways you can help mitigate that by, for example, providing your customers a SaaS alternative.  Also if you have “mammoth” products that have a long implementation timeframe, look at ways you can componentize these and provide your customers small, quick and practical measures to help them achieve their own efficiency goals quickly but can be built upon longer term.