Directions Hadoop is Moving In

Hadoop is a data system so big it is like a virtual jumbo where your PC is a flea. One of the developers named it after his kid?s toy elephant so there is no complicated acronym to stumble over. The system is actually conceptually simple. It has loads of storage capacity and an unusual way of processing data. It does not wait for big files to report in to its software. Instead, it takes the processing system to the data.

The next question is what to do with Hadoop. Perhaps the question would be better expressed as, what can we do with a wonderful opportunity that we could not do before. Certainly, Hadoop is not for storing videos when your laptop starts complaining. The interfaces are clumsy and Hadoop belongs in the realm of large organisations that have the money. Here are two examples to illustrate the point.

Hadoop in Healthcare

In the U.S., healthcare generates more than 150 gigabytes of data annually. Within this data there are important clues that online training provider DeZyre believes could lead to these solutions:

  • Personalised cancer treatments that relate to how individual genomes cause the disease to mutate uniquely
  • Intelligent online analysis of life signs (blood pressure, heart beat, breathing) in remote children?s hospitals treating multiple victims of catastrophes
  • Mining of patient information from health records, financial status and payroll data to understand how these variables impact on patient health
  • Understanding trends in healthcare claims to empower hospitals and health insurers to increase their competitive advantages.
  • New ways to prevent health insurance fraud by correlating it with claims histories, attorney costs and call centre notes.

Hadoop in Retail

The retail industry also generates a vast amount of data, due to consumer volumes and multiple touch points in the delivery funnel. Skillspeed business trainers report the following emerging trends:

  • Tracing individual consumers along the marketing trail to determine individual patterns for different demographics and understand consumers better.
  • Obtaining access to aggregated consumer feedback regarding advertising campaigns, product launches, competitor tactics and so on.
  • Staying with individual consumers as they move through retail outlets and personalising their experience by delivering contextual messages.
  • Understanding the routes that virtual shoppers follow, and adding handy popups with useful hints and tips to encourage them on.
  • Detecting trends in consumer preferences in order to forecast next season sales and stock up or down accordingly.

Where to From Here?

Big data mining is akin to deep space research in that we are exploring fresh frontiers and discovering new worlds of information. The future is as broad as our imagination.?

Contact Us

  • (+353)(0)1-443-3807 – IRL
  • (+44)(0)20-7193-9751 – UK

Check our similar posts

The Future of Cloud Backup and Recovery

We came across a post on Docurated that pulled together thirty-seven suggestions for the top cloud storage mistakes user companies make. Given that cloud storage seems to be the best backup solution for now at least, we decided to turn these ideas around to sense the direction cloud backup and recovery needs to take, if it is still to be relevant in say ten years? time.

Has Cloud Storage Largely Saturated the West?
It probably has. Outside of major corporates who make their own arrangements ? and SME?s that use free services by email providers ? the middle band of companies in Europe and America have found their service providers, although they may have never tested the recovery process, to see if it works.

The new gold rush in the cloud backup and recovery business is, or should be emerging markets in Asia, Africa, South America, and the Middle East. There, connectivity is brittler than over here. To be relevant in these fragile, more populous areas our cloud backup and recovery industry need to be more agile and nimble.

? It must provide a simpler service emerging commerce can afford, refresh its user interfaces in third world languages, have more accessible help, and be patient to explain how cloud storage works to newbies. In other words, it must source its call centre operators in the areas it serves.

? It must adapt to local connectivity standards, and stop expecting someone with ADSL broadband to keep up with cloud server networks running at up to 1GBPS compared to their 10MBPS at best. For user sourcing and retention purposes, these new cloud backup and recovery services must be the ones who adapt.

? It must facilitate disaster recovery simulations among its clients in calmer moments when things are going well. Are they backing up the right files, are they updating these, and are their brittle ADSL networks able to cope with their cloud service providers? upload and download speeds?

? It must develop lean and agile systems slim enough to accommodate a micro client starting out, but sufficiently elastic to transfer them seamlessly to big data performance. The Asian, African, South American, and Middle Eastern regions are volume driven, and individual economies of scale are still rare.

? It must not expect its users to know automatically what they need, and be honest to admit that Western solutions may be wrong-sized. Conversion funnels in the new gold rush are bound to be longer. Engagements there depend on trust, not elevator sales letters. Our competition in these countries already works this way.

? It must be honest and admit cloud storage is only part of the solution. To recruit and retain users it must step back to 1983, when Compuserve offered its customers 128k of disc space, and spent an amount of effort explaining how to filter what to put there.

Cloud Storage of Data is Only One Part of the Solution
Governance reports and stock certificates burn just as easily as do servers in a fire. We must not transfer bad habits to exciting new markets. We close this article with the thoughts of John Howie, COO of Cloud Security Alliance, as reported in the Docurated post we mentioned, and these apply across the globe, we believe.
There is no single most important thing to carry forward into the future of cloud backup and recovery. We must be mindful when moving data that this can be fragile too. We must also create layers of backup the way insurance companies re-insure, that make any one cloud backup and recovery business redundant if it happens.
We hold the trust of our customers in our hands but trust is delicate too. We must cease trying to make a pile of money quickly, and become more interested in ensuring that data transferred back and forth is synchronised. The cloud backup and recovery industry needs only one notorious mistake, to become redundant itself in the ten years we mentioned.

Contact Us

  • (+353)(0)1-443-3807 – IRL
  • (+44)(0)20-7193-9751 – UK
Why Spreadsheets can send the Pillars of Solvency II Crashing Down


Solvency II is now fast approaching and while it may provide added protection to policy holders, its impact on the insurance industry is not all a bed of roses. Expect insurance companies to restructure, increase manpower, and raise spending on actuarial operations and risk management initiatives. Those that cannot, will have to go. But what have spreadsheets got to do with all these?

Well, spreadsheets aren’t really the main casts in this blockbuster of a regulatory exercise but they certainly have a significant supporting role to play. Pillar I of Solvency II, which calls for improved supervision on internal control, risk management, and corporate governance, and Pillar II, which tackles supervisory reporting and public disclosure of financial and other relevant information, both affect systems that have high-reliance on spreadsheets.

A little background about spreadsheets might help.

Who needs an IT solution when you can have spreadsheets?

Everyone in any organisation just love spreadsheets; from the office clerk to the CEO. Because they’re so easy to use (not to mention they’re a staple in office computers), people employ them for processing numbers and as an all-around tool for planning, forecasting, reporting, complex modelling, market data analysis, and so on. They make such tasks faster and easier. Really?

You probably haven’t heard of spreadsheet hell

Unfortunately, spreadsheets do have certain shortcomings. Due to their inherent structure and lack of controls, it is so easy to commit simple errors like an accidental copy paste, an omission of a negative sign, an incorrect data input, or an unintentional deletion. Such shortcomings may seem harmless until your shareholders discover a multi-million discrepancy in your financial report.

And because spreadsheet errors can go undetected for a long time, they are constant targets of fraudsters. In other words, spreadsheets are high risk applications.

Solvency II Impact on Spreadsheet-based Financial and IT Systems

Regulations like Solvency II, are aimed at reducing risks to manageable levels. Basically, Solvency II is a risk-based system wherein a company?s capital requirements will depend on its measured riskiness. If companies want to avoid facing onerous capital requirements, they have to comply.

The three pillars of Solvency II have to be in place. Now, since spreadsheets (also known as User Developed Applications or UDAs) are high-risk applications with weak control features and prone to produce inaccurate reports, companies will have a lot of work to do to establish Pillars II and III.

There are at least 8 articles that impact spreadsheets in the directive. Article 82, for example, which requires firms to ensure a high level of data quality and accuracy, strikes at the very core of spreadsheets? weakness.

A whitepaper by Raymond Panko entitled ?Spreadsheets and Sarbanes-Oxley: Regulations, Risks, and Control Frameworks? mentioned that 94% of audited real world operational spreadsheets that were included in his study were found to have errors and that an average of 5.2% of all cells in the audited spreadsheets had errors.

Furthermore, many articles in the directive call for the enforcement of better documentation. This is one thing that’s very tedious and almost unrealistic to do with spreadsheets because just about anyone uses them. Besides, with different ‘versions? of the same data existing in different workstations throughout the organisation, it would be extremely difficult to keep track of them all.

Because of spreadsheets you now need an IT solution

It is clear that, with the growing number of regulations and the mounting complexity of tasks needed for compliance, spreadsheets no longer belong in this era. What you need is a server-based solution that allows for seamless collaboration, data reliability, data consistency, increased security, automatic consolidation, and all the other features that make regulation compliance more doable.

One important ingredient for achieving Solvency II compliance is sound data risk management. Sad to say, the ubiquitous spreadsheet will only expose your data to more risks.

More Spreadsheet Blogs


Spreadsheet Risks in Banks


Top 10 Disadvantages of Spreadsheets


Disadvantages of Spreadsheets – obstacles to compliance in the Healthcare Industry


How Internal Auditors can win the War against Spreadsheet Fraud


Spreadsheet Reporting – No Room in your company in an age of Business Intelligence


Still looking for a Way to Consolidate Excel Spreadsheets?


Disadvantages of Spreadsheets


Spreadsheet woes – ill equipped for an Agile Business Environment


Spreadsheet Fraud


Spreadsheet Woes – Limited features for easy adoption of a control framework


Spreadsheet woes – Burden in SOX Compliance and other Regulations


Spreadsheet Risk Issues


Server Application Solutions – Don’t let Spreadsheets hold your Business back


Why Spreadsheets can send the pillars of Solvency II crashing down

Advert-Book-UK

amazon.co.uk

Advert-Book-USA

amazon.com

Contact Us

  • (+353)(0)1-443-3807 – IRL
  • (+44)(0)20-7193-9751 – UK
Vendor Selection

When shopping for an IT solution for your enterprise, there are two things you should scrutinise: the product (or service) itself and its vendor. Many times, companies overlook the importance of the latter, giving the reason that “it’s only the product we need”.

Wrong.

What about after-sales technical support and training? Ok, so you have an in-house team with the required competency for that IT solution in question… not that I believe it’s reasonable basis to pass up on the expertise that the vendor can provide. How about upgrades, patches, and documentation?

Still unperturbed? Here’s one factor that you may not have started to consider – What happens to your product if the vendor goes bankrupt or gets swallowed by a merger and acquisition? Surely, you no longer believe this is far from possible, do you?

But how are you supposed to know the financial stability of each vendor or whether it is an acquisition target? Well, you can either conduct your own research or you can leave that up to us. Part of our job includes not only establishing linkages in the industry but also being in-the-know on such relevant information.

Evaluation of Business Needs

You can’t separate vendor selection from the process of choosing the desired IT tool. That’s why our vendor selection services starts by defining exactly what your business needs are.

Once we’ve pinned down your needs, we can then narrow down the list of possible IT solutions. Only then can we proceed with the main vendor selection process.

Have you ever been caught in a situation wherein you thought you knew what you wanted, only to end up realising it’s not what you were looking for after all? We’re here to make sure you don’t get caught in that kind of situation when choosing an enterprise-class IT solution.

With the TCO (total cost of ownership) of such solutions typically running up to hundreds of thousands of euros, you can’t afford to arrive at what you really want by way of trial and error.

These are the things you stand to benefit the moment we start working with you:

  • Thorough assessment of your IT needs. We’ll consult the people in your organisation who’ll be affected the most in order to obtain a clear picture of what your specific needs really are. Most IT solution purchases are made with very little consultation that, after installation, many of the end users don’t benefit at all.
  • Minimal interruption during assessment. As with all our other services, we see to it that the interruptions we make are absolutely necessary. So the moment we start with our work, you can still continue with yours.
  • Insightful suggestions of the required IT solution. You still know your business better. So even after we’ve gone through the assessment and given our recommendations, the decision as to what IT tool should be pursued will still be up to you. The difference now is, you’ll be making a decision based on expertly gathered information put forward in an insightful proposal.

Request and Evaluation of Vendor Proposals

With so many IT solutions companies mushrooming, it is becoming more difficult to keep track of them, their specialities, strengths, and weaknesses.

Companies selling best-of-breed products may be relatively easy to spot. But there are also other attributes that are equally important but not as well publicised. For instance, which companies offer better quality management philosophies? Which companies have strategic visions running parallel to yours? Which of them possess implementation capabilities that can cater to your rapidly growing IT requirements?

Vendors who answer positively to these queries need to be given the appropriate importance in the selection process. We see to it that these and other relevant attributes are factored into our scorecards and evaluation processes.

These are the things you can look forward to when you grant us the opportunity to serve you.

  • Experience is a vital item in our vendor selection criteria. Our vast knowledge of the reliable players in the industry will lead you to experienced vendors who can hit the ground running from day one and continue with the same vigour onward.
  • We can help you draw positive response for each of your Request For Proposals (RFPs) or Request For Information (RFIs). Did you expect these vendors to be enthusiastic in sending out proposals each time you asked them to? Think again. You’ll have to persuade them first of your sincerity to become a potential customer. With our help, your RFPs will make preferred vendors see “opportunity” written all over.
  • No need to go “Eany, meeny, miny, moe”. Deciding which vendors should move up in the selection process can take up a lot of time if you don’t know which criterion should be given more weight. Our scorecards are designed to collect the most relevant information and to generate results that will help you decide on these matters at a glance.

Interview, Negotiation, and Monitoring

As soon as you start getting positive response to your Request For Proposals, the interview process should be next. It’s at this point that vendors can present and highlight their strengths while we try to glean as much information of their true capabilities as well as their dedication to the project.

Some companies can provide proof-of-concepts and we may require them as part of the interview process. This will not only give us a better idea as with regards to their product’s capabilities, but also to their level of expertise on the solution in question.

  • We’ll help you set up the interview process and organise the evaluation committee. Members of the committee will typically include representatives from each department that will be affected by the new technology, which we would have already identified during our Evaluation of Business Needs.
  • Since our scorecards are designed to expedite the filtering and selection process, you may eventually be able to choose the finalists yourself. However, in the event that two or more vendors turn out evenly matched, we’ll help you identify the better company.
  • We’re very familiar with the price ranges of various IT solutions, including the effects on price of certain variables. As such, we can tell you whether a product’s price tag is justified or not.
  • Our exceptional familiarity on both the IT industry and the entire negotiation processes itself will give you the edge when it’s time for us to haggle for the best bang for the buck.
  • After the contract is awarded, we’ll even be on hand to monitor whether deliverables are handed over and milestones are achieved as promised.

Ready to work with Denizon?