Over the next few posts I will look at what Endeca could do for Enterprise BI and how to start evaluating what it can do:
- The possibilities (this post)
- Evaluating on Amazon Web Services
- Finishing touches to the AWS install
So before diving into the detail and looking at what Endeca can do I'd like to look at what we are trying to achieve with BI in the enterprise. In any enterprise, old or new they are trying to be better than their competitors in one or more areas, being more efficient and competing on price, or perhaps differentiating themselves by having better products. As BI professionals we are trying to enable the business to be better at their targeted competitive strategy. So we can focus on improving the performance sales organisations, better product research, better marketing, operational efficiencies, etc. So how do we go about this and where do we start? The path from this points well trodden, and quite well understood.
If your lucky enough to be working in a green field site you have a wealth of options available, the traditional enterprise data warehouse is a realistic target. Much has been written about the likes of the transformational BI in NetFlix and Capital One, both essentially green field sites. If you are using commercial off the shelf business packages then the off the shelf BI packages such as OBIA are a reasonable starting point and offer a low cost entry point. Beyond this the warehouse models such as the oracle warehouse reference model are available and tried and tested methods for best practice custom BI implementations. But the world is changing, where does Big Data fit with the EDW model, how do you analyse unstructured data? Various schemes have been proposed but so far I'm not sure any of them really "feel right".
Sadly though the reality is that the majority of businesses are distinctly brown field. Hopefully your enterprise architects have read the excellent Enterprise Architecture as Strategy, and are banging on the door of the C-level executives with a copy. So while you might be looking at a brighter future the reality is we have to help the business as it exists today, with a large estate of legacy systems and planned system retirements over the next 5 years or so. Even in this harsh environment it is possible to deliver using traditional EDW techniques, but as many have found to their cost without the highest level executive support and the associated funding and resources you face an uphill battle. To get that executive sponsorship you have to have a proven track record of success, and to do that you need to start somewhere, possibly with little or no budget, resource or support. The business probably also balk at the costs that you start to discuss when talking about EDW solutions, they are used to their landscape of spread marts, Access databases and VB Heath Robinson solutions. While being cheap these are hopelessly inflexible after a very short period of time, not scalable, inaccurate silos of data. But when you roll up and say it will cost them $250k or more to replace their spreadsheet they are not unreasonably a little shocked.
So lets look at where the costs go in EDW. If we take the Kimball lifecycle as being a typical model we not unsurprisingly start with the project governance and requirements gathering. This needs to be done to a reasonably detailed level before you can even contemplate the next level of data model design. I've also known some companies get this so wrong it was laughable. Because they treated all "development" as being the same a poor developer was not allowed to start any design work or coding before he had got a requirements document for "each" report signed off. Naturally the business did not really know what they wanted and kept changing their minds, so after 50 days of effort it was still not signed off, even taking a low estimate of $250 a day, this requirements document cost $12500.
Even if you have a project team who are are effective at BI development this is still not a small undertaking, it will be several weeks of information discovery, requirements gathering, design, development, deployment and testing before the end point business partner gets to see anything. This is not even factoring in the problems around resource planning, budget approvals, business case sign off etc. But also factor in the constantly shifting sands of the IT estate, this route looks less and less attractive.
So is this an opportunity for Endeca to ride to the rescue? Well lets look at what we could do differently to enable us to trim some fat from the development process. Why do we need project governance? Put simply it is required to coordinate multiple work threads and resources to collectively achieve their objectives. As the size and complexity of the planned work increases, along with the size of the delivery team, so must the quantity of co-ordination work. So it follows that if we can reduce the size of the delivery team the less project management we need. So lets look at the next step, business requirements definition. Why are we doing this stage? Stating the obvious, its so that the design and build team know what requirements of the system they have been tasked with delivering has to do. Based on the premise that change and rework is expensive, you want to build something once and get it right. You are probably at this point thinking that a few problems are cropping up here for our brown field site. But lets carry on. We are now designing our dimensional models and BI applications, because we have to design them right? How else will we know what to build. But we are at this point "fixing" aspects of what and how we can report on our data. Now we go away and design our ETL to take our source data, cleanse and load it and put it into some data structures we can query. What if we could carry the flexibility of not having to fix our dimensional model so early? What if we did not even have to do it until we were exploring how we wanted to query the data.
There looks to be something promising here. So if we could get away from rework and change actually being costly and something we just expect and factor in we can really back off on the depth of requirements gathering. What if we did not have to define a dimensional model? What if we had a tool so simple to use that someone with a broad skill set of business analyst and developer could use it? That tool is Endeca. We are now looking at genuinely agile BI development. It is possible to use a one or two person team to deliver on the entire project. Even if you use Endeca for nothing more than a proof of concept to enable the business to crystallise the requirements and work out exactly what they need to know it would be beneficial.
So is it practical to embed a single highly skilled individual who is architect, analyst and developer in the business units as required to solve business problems and boost business performance? I believe it is and over the next few blog posts I'm going to look in some detail about how you can use Endeca in an agile way to deliver useful BI enabling business performance improvement.
No comments:
Post a Comment