Tony Bain

Twitter @ddata Discuss Data bot

Image via CrunchBase

If you use twitter then this may be interesting, I have launched a twitter based “Discuss Data” blog and bot.  This is a forum where thoughts, ideas, questions and links can quickly be posted using twitter and discussed through twitter, or through a more traditional weblogs comment system.  Anyone who is involved in data management may find this useful and interesting, and is welcome to contribute.

For more information visit http://www.discussdata.com and just follow http://www.twitter.com/ddata

Web 2.0 Data Scalability

Image via Wikipedia

In this previous post I wrote a section on Web 2.0 data platform scalability, in it I mentioned the importance of scale out.  I also mentioned that scale out was important but currently it is hard.  There really aren’t a lot of “elegant” or “natural” ways to scale out a database if you are using any of the mainstream database vendor products (SQL Server, MySQL, Oracle has grids but isn’t shared nothing scale out so is limited).

A question I was asked was “Surely this has all been sorted by now through their widespread use in the enterprise?”.  Good question but unfortunately the answer is no.  The reason behind this is simple.  There are very few enterprise workloads that cannot be satisfied with a scale up (big server) solution.  Maybe 1 in every 20,000 databases will have a workload so great that it can’t be satisfied by a single server, in fact most enterprise databases will happily share servers with other databases.  The biggest ERP’s and online share trading applications that I am aware of all are provided for by a single big server solution.  Quite simply, none of these applications have the workload requirements of tens of thousands of concurrent users that a Twitter or Facebook has.

So this issue really hasn’t been addressed in any adequate way by the mainstream vendors.  Later on I will post on why the current products are not suited for scale out, and the efforts in progress to rectify this.

Web 2.0 – Ensuring Your Data is the Asset

Image via Wikipedia

For the purpose of this article “A Data Platform” is what I use to describe an implementation of servers, database management software, a data model, and any surrounding classes that separate data from user facing application logic and functionality.

I have noticed that sometimes startups are choosing a data platform based on factors such as purity of the development model, and the familiarly of the coding model to developers which in theory leads to increased speed of development.  While these might seem like important things, this really isn’t a great method of deciding what will be the foundations of the entire offering you build.  I mean would you choose a location, design and construction method for building a house based on what suits your builder best or what will serve your daily domestic needs and be flexible for you and your family now and into the future?

Your counter argument to the above might be that time to market is all important, therefore going with what is the quickest is the best idea and you can always retrofit things later.  Sure I agree time to market is important but probably more important is, once you have hooked people in, the speed at which you respond to user feedback, adapt your application based on user usage and add new and advanced features to stay ahead of your eventual competition. 

You might be building a user focused web application, so you might want to use Java or .NET or some other OO language, so you might think it is a good idea to choose data platform X or Y because that is the best data platform to rapidly build this style of application with your chosen development toolset.  But really that is making decisions based on a subset of requirements, your applications features and functionality.  While no one can predict the future you do need to look a little bit past the end of your nose and think about a few other key requirements that will have to deliver on if people actually start to use your application.

Scalability

If you are actually successful then you will need to be able to scale.  Not over months and years, but over days or weeks.  You will need to be able to rapidly add capacity to your data platform to continue to meet the performance expectations of your rapidly growing user base.

Unlike application layers, scaling the data platform is hard and complex.  Usually the application layer can be easily scaled by just adding more and more application servers into the pool.  Doing so at the data layer is much more difficult as you have all sorts of issues around data availability, consistency, latency, redundancy and so on.  If your cunning plan is to work this all out later you may be unhappily surprised because it could take you months to retrofit your application to a more suitable data platform and while you are doing this Joe the Competitor may scream past you at a hundred scalable miles per hour.

Anyway, while we are on the subject, you don’t care about scaling up (i.e. getting the biggest server you can get).  The type of scalability (and availability) solution you will need isn’t going to be possible with a single mega-server so don’t worry about this approach.  You need to be able to scale out the data platform, to be able rapidly and continually add more and more capacity into your data layer.  This approach also has the plus side of allowing you to start small while you are still paying for this all on your Discover Card, but when you take off and get your first million from Fred Wilson you can start rolling in capacity day and night.

And of course you need to think about availability while doing this.  Most Web 2.0 applications are globally used so there is no such thing as down time, maintenance windows, or quite time when you can make a cup of tea.  It is all on, all day, every day.  And did I mention data loss?  Don’t even think about losing anyone’s blog post, tweet, comment or photo of their kids eating cake.  If you lose data and lose it routinely you will be dead to your users from that point on.

Flexibility

So you have spent a lot of time thinking about what your application will do and which features the user will love.  You thought of them, you love them so of course everyone else will think they are brilliant.  Sure, but what will actually happen is when you will launch you will find that you were almost completely wrong all along, the general concept of what you were trying to do might remain (or might not) but your ability to continue to exist will be driven by your ability to rework your application in response to user criticism, feedback and ideas and completely change what you were doing into something that someone actually wants to use (do I need to mention Facebook and Twitter here?).

The problem here is if you have implemented a data platform specifically based on your initial utopian view of your application.  You see a good data model shouldn’t actually model a specific part of an application but instead it should model the data that it is contains in a more “natural form”.  At the core you should have definable entities (such as person, vehicle, garment), attributes (color, size, taste, hair length), relationships and constraints that represent the data itself not necessarily and applications use of that data.  If you have implemented a data platform that is heavily tied to a misguided V1 view of things then your ability to rapidly adapt for V2, the version people will actually like, is going to be heavily impeded.  This lag will of course be a blessing to all your new competitors who will take all the good ideas that you have sitting in your in-tray and implement them as a more successful rip off of your idea.

Another major downside of tying your data platform heavily into individual parts of the application layer is, well, take a look at where we are heading.  Web 3.0 (we are at 2.5 now!) is coming and as fancy and as pretty as your web site is, it is going to matter less and less.   Integration is the key.  RSS, API’s – data portability and interoperability is the future (man, I feel like I should be saying man there?).  How easy will it be to build integration interfaces to your bulging bag of data if your data model only makes sense to Tom the web site developer?  Don’t get me wrong, Tom is a nice guy but he doesn’t really understand that his site isn’t the only piece of the puzzle.  As we go forward many applications interfaces are going to float around the central data platform,  sure his web site will be there as one of them, but also you will have various API’s and interfaces that allow your users to integrate their data into the 78 other social network applications they use.  They might use your application 10 times a day but never log into your web site at all.

Leverage your Asset

It doesn’t matter what your application does, whether it tweets or trades or pokes or woofs or whatever.  At some point you will reach critical mass and the game will change.  You have to remember the only asset you will have is the data.  Your $1b valuation consists of a very big pile of bits and bytes stored within your data platform, and the associated eyeballs that check every 15 minutes to see if their bits have changed slightly.  To make money you will need to sell those eyeballs stuff.  Whether it is your stuff or someone else’s stuff, you need to your maximum potential ensure every time those eyeballs are checking their bits they are also been tempted with some tantalizing relevant product, service or new feature.

The only way you can do this is to analyze the data you have at hand and work out a trend, profile, pattern or segmentation that will allow you to interact with each user personally like you are an old school chum, despite that fact that they are just one of a trillion or so users who have use your application.  You will of course need to do data analytics.

The data platform you use for running your application day to day isn’t necessarily the same data platform that you use to do the analytics with but you do need to think about how this process is going to work up front.  If you are processing thousands of transactions a second how are you going to feed this data out to your analytical platform and how are you going to feed all the wonderful tidbits back in? 

What will be the timeliness of information doing this?  Is analyzing last month’s data going to be good enough?  Well no, of course it isn’t.  What would your recommendation for your stock application be today if that was the case, hold GOOG?  You will need to be able to do this in real time in response to real time changes of your user base.  Keep in mind all the big successful online players generate their core revenue, or enhance their revenue using advanced analytics (Google, Amazon, EBay).

Analytics will also help you listen through the noise and evolve your application to ensure you are adding things people want, not screwing up things people actually use and that you’re staying ahead in usability shifts as people evolved with your service.  Micro changes in behavior can result in massive changes in usage patterns when you are multiplying everything by a squillion.

Summary

What is interesting after reading this post is that the points I am making aren’t new.  In fact they are old, really old, wind back the clock 10 years and we could have been having the same conversation but instead of Web 2.0 I would have been saying client/server or just Web App.  Basically all I am saying is when building a Web 2.0, 3.0 or whatever, start with the data, focus on the data model and build up from there ensuring you address scalability and availability requirements in your design as you go.   You may not need to action these parts of your design right away but when you get your break on CNN or Oprah you know you will be able to roll in the capacity to see you grow and grow.

And don’t forget everything you were ever taught in development school.  Data models still matter.  Data independence matters.  Platform scalability matters.  Data integration & Interoperability matters. 

So if you get all this data stuff right, you will be able to work out all the minor bits like features and functionality of your web application later on once the users tell you what on earth that want to do with it.

What is Hadoop?

Image via CrunchBase

Ok so you are setting out to build the next Google and are considering using a Map/Reduce based data access strategy over traditional SQL.  Just as you need a database server to process SQL queries you also require the underlying infrastructure to manage your data and to execute your Map/Reduce routines.  Hadoop is one such system that is gaining acceptance, being co-developed and implemented for data analytics purposes at Yahoo and Facebook amongst others.

Hadoop is the system that allows unstructured data to be distributed across hundreds or thousands of machines forming shared nothing clusters, and the execution of Map/Reduce routines to run on the data in that cluster.  Hadoop has its own filesystem which replicates data to multiple nodes to ensure  if one node holding data goes down, there are at least 2 other nodes from which to retrieve that piece of information.  This protects the data availability from node failure, something which is critical when there are many nodes in a cluster (aka RAID at a server level).

So will Hadoop outperform a RDBMS?  Well unless you are dealing with very large volumes of unstructured data (hundreds of GB, TB’s or PB’s) and have large numbers of machines available you will likely find the performance of Hadoop running a Map/Reduce query much slower than a comparable SQL query on a relational database.  Hadoop uses a brute force access method whereas RDBMS’s have optimization methods for accessing data such as indexes and read-ahead.  The benefits really do only come into play when the positive of mass parallelism is achieved, or the data is unstructured to the point where no RDBMS optimizations can be applied to help the performance of queries.  Indeed benchmarks from the Hadoop site show performance significantly slower in straight line query performance when compared to a relational DB on small scale tests.

	MySql 5.0.27	Hadoop-0.15.2
Data	B-tree disk table (MyISAM)	Text files (access_log)
Machine	1	2
Rows	5,914,669	5,914,669
Results	100	100
Time	4.43 sec	172.30 sec

But with all benchmarks everything has to be taken into consideration.  For example, if the data starts life in a text file in the file system (e.g. a log file) the cost associated with extracting that data from the text file and structuring it into a standard schema and loading it into the RDBMS has to be considered.  And if you have to do that for 1000 or 10,000 log files that may take minutes or hours or days to do (with Hadoop you still have to copy the files to its file system).  It may also be practically impossible to load such data into a RDBMS for some environments as data could be generated in such a volume that a load process into a RDBMS cannot keep up.  So while using Hadoop your query time may be slower (speed improves with more nodes in the cluster) but potentially your access time to the data may be improved. 

Also as there aren’t any mainstream RDBMS’s that scale to thousands of nodes, at some point the sheer mass of brute force processing power will outperform the optimized, but restricted on scale, relational access methods.

So while Hadoop and Map/Reduce are gaining more popularity it shouldn’t be considered a like for like alternative to a relational RDBMS for most applications.  It is a specialized tool with a specialized set of criteria that need to be fulfilled to achieve benefit over more traditional approaches.

Related articles by Zemanta

• Yahoo Search Wants to Be More Like Google, Embraces Hadoop
• Yahoo's Supercomputing Initiative Running Hadoop

OT: What does the Crunch mean for Australasian Startups?

Image via Wikipedia

What a week.  The worst week on the Dow ever.  Following that we have seen multiple alerts from VC’s out to their investments to batten down the hatches, cut burn, maximize revenues and prepare for a bumpy ride.  The underlying message that is being put out though by the US VC community, is that things can be ok if you act now and get prepared.  Such examples of these messages are here, here, here, here  .

I have been thinking about how this translates to us down here in Australia and New Zealand.  Obviously the financial crisis is having a global impact, which we saw last week with the nose dive of the ASX.  While the entire economy is slumping, how does this flow through to the Tech sector, and in particular the new tech or startup scene?

An obvious key down side is that investors locally mimic investors globally, so overall available funding sources will decrease.  But getting funding here has always been tough and not too many revenueless business models have been given life downunder.  But sound ideas with proven innovation, and sound business models continue to attract investment.

A positive side of the situation is that globally the landscape has become more competitive for Australasian based startups.  “Be small enough not to fail” is the mantra being recited over in the valley, meaning that the startups on high burn rates should cut costs and staff and downsize to a more sustainable version of themselves.  If we look at the characteristics of many Australasian startups we see that for most this has always been the normal way of life.  Funding has always been much tighter, and proportionally more ventures have been self funded.  I don’t have stats but I would suspect that a large proportion of IT startups would remain in the 3-5 people range in their first 12 months.  During a boom this low flow of funds is prohibitive, not necessarily because the core innovation isn’t competitive but rather they are restrained on everything else that goes with launching a new technology.  This includes things like global marketing, building global alliances, doing big deals with major players, acquiring complimentary technologies etc. 

But in the current climate you have a good head start as you know how to operate in such conditions.  This is what you do; launch successful ventures on a shoestring and you have been doing so for a long time.  So while there is a lot of doom and gloom about globally, remember that the landscape has just become more competitive for you and maybe still is a good time to be involved in a technology startup based in Australasia.

RockSolid @ SQL Pass

We will be exhibiting RockSolid, Red Rock’s SQL Server Automated Management product, at SQL Pass this year.  The summit is November 19-21 in Seattle.  If you want to book a time for a one on one demo of RockSolid please drop me a line at sqlpass@tonybain.com

when clouds evaporate your data

Image via Wikipedia

My response to the following post from Billy Bosworth:

The surge to the cloud feels like a certainty (as much as anything can be a certainty in technology).  It reminds me of the momentum shifts felt when things were moving to client/server computing, and more recently around virtualization.  I think the media is largely in a frenzy precisely *because* it’s the newest thing we’ve had to talk about of “mass” interest since virtualization.  That frenzy feeds on itself (as evidenced even  by my own posts on the topic).

This highlights why I think it will be difficult for a startup or even a well established but small time player to launch a successful cloud data service.  I have posted my concerns about this in the past in relation to 10gen.  Building a key part of your business on top of a shaky or unproven foundation is a never a great idea, and this holds true regardless if you are a Web 2.0 application startup or a more traditional business.  How will you suffer if part of your stack disappears due to their infrastructure or process failures or they just run out of funding and turn everything off?  If this also takes all your data with it then you will be doubly screwed.  An off cloud, locally controlled backup in a non-proprietary format will go a long way to reduce such concerns.  But still if you have your data in hand it is going to take you at least a week or two to port to another provider.

So while I think Cloud data services will have a broad appeal and their acceptance will continue to grow and grow, I think we will see a natural attraction to the big and robust vendors, meaning overtime the vendors owning the market share of data services in the cloud will be largely the same as those owning the market share of data services in the enterprise (of course with a few key exceptions).

 

Related articles by Zemanta

• IBM Launches Cloud Services Initiative

What is Microsoft Gemini?

Image via CrunchBase

Microsoft has announced its Roadmap for integrating the various DW acquisitions it has been doing recently, including Datallegro.  Part of that roadmap includes their “Gemini Project”.

Gemini is the codename for the next major update to Analysis Services.  It includes an in memory, column based storage mode which allows analysis to be performed predefined models, but without the requirement to pre-aggregate or build cubes prior to querying.  Gemini is expected to support the XLMA and MDX query languages so applications that work with existing analysis services cubes could in theory be made to work with Gemini.

Microsoft is again pushing Excel as the query tool of choice for Analysis Services.  This was the approach a few years back also, but the interface was far too limited and troublesome to be used for anything but the simplest requirement.  The “grab and go” ad-hoc reporting ideal was nice in theory but wasn’t really been successful to date due to the complexity in business logic required to roll figures up into neat little KPIs.

More details on Gemini aren’t yet available but will begin to surface now the announcement has been made at their BI conference.

Amazon to provide SQL Server "Cloud" Offering

Image via CrunchBase

A week after all the hype of Oracles/Amazon’s EC2 cloud offering, Amazon has also announced that it will be providing an EC2 SQL Server Cloud offering shortly.  As previously discussed these aren’t really what most people think of as “cloud databases” instead they are just self provisioned hosted solutions. 

Microsoft is probably saving the fanfare until the service is actually available.