Directions Hadoop is Moving In

Hadoop is a data system so big it is like a virtual jumbo where your PC is a flea. One of the developers named it after his kid?s toy elephant so there is no complicated acronym to stumble over. The system is actually conceptually simple. It has loads of storage capacity and an unusual way of processing data. It does not wait for big files to report in to its software. Instead, it takes the processing system to the data.

The next question is what to do with Hadoop. Perhaps the question would be better expressed as, what can we do with a wonderful opportunity that we could not do before. Certainly, Hadoop is not for storing videos when your laptop starts complaining. The interfaces are clumsy and Hadoop belongs in the realm of large organisations that have the money. Here are two examples to illustrate the point.

Hadoop in Healthcare

In the U.S., healthcare generates more than 150 gigabytes of data annually. Within this data there are important clues that online training provider DeZyre believes could lead to these solutions:

  • Personalised cancer treatments that relate to how individual genomes cause the disease to mutate uniquely
  • Intelligent online analysis of life signs (blood pressure, heart beat, breathing) in remote children?s hospitals treating multiple victims of catastrophes
  • Mining of patient information from health records, financial status and payroll data to understand how these variables impact on patient health
  • Understanding trends in healthcare claims to empower hospitals and health insurers to increase their competitive advantages.
  • New ways to prevent health insurance fraud by correlating it with claims histories, attorney costs and call centre notes.

Hadoop in Retail

The retail industry also generates a vast amount of data, due to consumer volumes and multiple touch points in the delivery funnel. Skillspeed business trainers report the following emerging trends:

  • Tracing individual consumers along the marketing trail to determine individual patterns for different demographics and understand consumers better.
  • Obtaining access to aggregated consumer feedback regarding advertising campaigns, product launches, competitor tactics and so on.
  • Staying with individual consumers as they move through retail outlets and personalising their experience by delivering contextual messages.
  • Understanding the routes that virtual shoppers follow, and adding handy popups with useful hints and tips to encourage them on.
  • Detecting trends in consumer preferences in order to forecast next season sales and stock up or down accordingly.

Where to From Here?

Big data mining is akin to deep space research in that we are exploring fresh frontiers and discovering new worlds of information. The future is as broad as our imagination.?

Contact Us

  • (+353)(0)1-443-3807 – IRL
  • (+44)(0)20-7193-9751 – UK

Check our similar posts

Renewable energy – Is it a common man’s cup of tea?
I came across an article on a young graduate in renewable energy engineering. The fellow was doing technical sales and marketing jobs for renewable energy products though he felt that as a graduate, he ought to be doing more than just sales. His, sentiments, I can relate with but again thinking about the field of renewable energy, how many people understand what it is, its importance/ benefits, how to acquire it, its installation, costs etc.? Renewable energy is energy generated from natural resources. The renewable energy sources include sunlight, wind, rain, tides, geothermal heat and various forms of biomass. These sources are renewable naturally and continuously replenished, therefore this energy cannot be exhausted. Renewable energy technologies range from solar power, wind power, hydroelectricity/micro hydro, biomass and bio-fuels for transportation. Back to the aspiring young professional who felt that his place in the renewable energy sector lies in doing strategies and coming up with new products-the advice fronted to him was that doing technical sales is the best job for engineers, as it helps them impact on users of their products. Sales entail interacting with customers and knowing their needs so that the product features can be enhanced to suit the customer?s needs. Now, that is brilliant and accurate advice. It is however important to take into consideration that renewable energy is not a common man?s cup of tea and right now the focus all over the world is to build green economies. To me the need for more and more people to understand the benefits, savings and cost of renewable energy cannot be overemphasised. Effort should be made to keep marketing of renewable energy products/ services simple and conversational by avoiding use of acronyms or jargon explaining about operational details. More impact can be made if a marketing rather than technical sales approach is used. Technical sales have been described as boring (can be used as a sleeping aid), tends to use extensive vocabulary, jargon and acronyms that product users cannot relate with and tends to discuss the products technical aspects as opposed to the benefits to the customer. Fun should be created out of all this by making things simple and demonstrating cost savings and benefits of renewable energy.
Symbion Pharmacy Services? Definition of Responsibility

A ?symbion? is an organism in a symbiotic (i.e. mutually beneficial) relationship with another one. In the case of Australia?s giant Symbion Pharmacy Services, this means supplying and delivering over-counter Chemmart medicines to more than 3,000 hospital and retail pharmacies, while remaining mindful of its carbon footprint.

In 1999, the company with the tagline ?life matters? and a desire to be seen as ?a good corporate citizen? decided it was time to measure exactly what it was pumping out from 12 facilities and over 200 vehicles. This was a voluntary decision as even now there is still no carbon emissions law in Australia (although no doubt being a ?first mover? will put the company in a competitive position when this inevitably comes).

Symbion decided to install emission detection devices and connect these to a central monitoring system with the intention of managing what these measured. There were two stages to this process. First, Symbion determined its reporting requirements based on one of its larger warehouses. Following that, it established a carbon footprint for each of its wholly owned and managed facilities. This put it in a position to:

  • Analyse total emissions down to a level of detail where it understood the contribution of each source
  • Use big data management tools to identify carbon hotspots for priority remedial action
  • Inform the affected workforce, explain the monitoring system and keep them in the loop
  • Separately manage energy abatement programs such as lighting and delivery routes

The program also had productivity spin-offs in that it focused management attention on the processes behind the emissions that were ripe for material and system improvements. It also provided marketing leverage. Symbion?s customers are in the wellness business, ahead of the curve when it comes to how emissions contribute to chronic illness, and aware of the cost of this in terms of human capital.

EcoVaro could help you manage your throughputs by analysing your data on our cloud-based system. This includes trending your metrics, comparing them to your industry seasonal average, and providing you with a business-like view of how well you are doing.

Our service reduces your reliance on (and the cost of) third party audits, and simplifies the reporting process to your controlling authority. It simply makes more sense to contract your software out this way, and only pay for it when you need it.

How to Reduce Costs when Complying with SOX 404

Section 404 contains the most onerous and most costly requirements you’ll ever encounter in the Sarbanes-Oxley Act (SOX). In this article, we?ll take a closer look at the salient points of this contentious piece of legislation as it relates to IT. We?ll also explain why companies are encountering difficulties in complying with it.

Then as soon as we’ve tackled the main issues of this section and identify the pitfalls of compliance, we can then proceed with a discussion of what successful CIOs have done to eliminate those difficulties and consequently bring down their organisation’s IT compliance costs. From this post, you can glean insights that can help you plan a cost-effective way of achieving IT compliance with SOX.

SOX 404 in a nutshell

Section 404 of the Sarbanes-Oxley Act, entitled Management Assessment of Internal Controls, requires public companies covered by the Act to submit an annual report featuring an assessment of their company?s internal controls.

This ?internal control report? should state management’s responsibility in establishing/maintaining an adequate structure and a set of procedures for internal control over your company?s financial reporting processes. It should also contain an assessment of the effectiveness of those controls as of the end of your most recent fiscal year.

Because SOX also requires the public accounting firm that conducts your audit reports to attest to and report on your assessments, you can’t just make baseless claims regarding the effectiveness of your internal controls. As a matter of fact, you are mandated by both SEC and PCAOB to follow widely accepted control frameworks like COSO and COBIT. This framework will serve as a uniform guide for the internal controls you set up, the assessments you arrive at, and the attestation your external auditor reports on.

Why compliance of Section 404 is costly

Regardless which of the widely acceptable control frameworks you end up using, you will always be asked to document and test your controls. These activities can consume a considerable amount of man-hours and bring about additional expenses. Even the mere act of studying the control framework and figuring out how to align your current practices with it can be very tricky and can consume precious time; time that can be used for more productive endeavours.

Of course, there are exceptions. An organisation with highly centralised operations can experience relative ease and low costs while implementing SOX 404. But if your organisation follows a largely decentralised operation model, e.g. if you still make extensive use of spreadsheets in all your offices, then you’ll surely encounter many obstacles.

According to one survey conducted by FEI (Financial Executives International), an organisation that carried out a series of SOX-compliance-related surveys since the first year of SOX adoption, respondents with centralised operations enjoyed lower costs of compliance compared to those with decentralised operations. For example, in 2007, those with decentralised operations spent 30.1 % more for compliance than those with centralised operations.

The main reason for this disparity lies in the disorganised and complicated nature of spreadsheet systems.

Read why spreadsheets post a burden when complying with SOX and other regulations.

Unfortunately, a large number of companies still rely heavily on spreadsheets. Even those with expensive BI (Business Intelligence) systems still use spreadsheets as an ad-hoc tool for data processing and reporting.

Because compliance with Section 404 involves a significant amount of fixed costs, smaller companies tend to feel the impact more. This has been highlighted in the ?Final Report of the Advisory Committee on Smaller Public Companies? published on April 23, 2006. In that report, which can be downloaded from the official website of the US Securities and Exchange Commission, it was shown that:

  • Companies with over $5 Billion revenues spent only about 0.06% of revenues on Section 404 implementation
  • Companies with revenues between $1B – $4.9B spent about 0.16%
  • Companies with revenues between $500M – $999M spent about 0.27%
  • Companies with revenues between $100M – $499M spent about 0.53%
  • Companies with revenues less than $100M spent a whopping 2.55% on Section 404

Therefore, not only can you discern a relationship between the size of a company and the amount that the company ends up spending for SOX 404 relative to its revenues, but you can also clearly see that the unfavourable impact of Section 404 spending is considerably more pronounced in the smallest companies. Hence, the smaller the company is, the more crucial it is for that company to find ways that can bring down the costs of Section 404 implementation.

How to alleviate costs of section 404

If you recall the FEI survey mentioned earlier, it was shown that organisations with decentralised operations usually ended up spending more for SOX 404 implementation than those that had a more centralized model. Then in the ?Final Report of the Advisory Committee on Smaller Public Companies?, it was also shown that public companies with the smallest revenues suffered a similar fate.

Can we draw a line connecting those two? Does it simply mean that large spending on SOX affects two sets of companies, i.e., those that have decentralised operations and those that are small? Or can there be an even deeper implication? Might it not be possible that these two sets are actually one and the same?

From our experience, small companies are less inclined to spend on server based solutions compared to the big ones. As a result, it is within this group of small companies where you can find a proliferation of spreadsheet systems. In other words, small companies are more likely to follow a decentralised model. Spreadsheets were not designed to implement strict control features, so if you want to apply a control framework on a spreadsheet-based system, it won’t be easy.

For example, how are you going to conduct testing on every single spreadsheet cell that plays a role in financial reporting when the spreadsheets involved in the financial reporting process are distributed across different workstations in different offices in an organisation with a countrywide operation?

It’s really not a trivial problem.

Based on the FEI survey however, the big companies have already found a solution – employing a server-based system.

Typical server based systems, which of course espouse a centralised model, already come with built-in controls. If you need to modify or add more controls, then you can do so with relative ease because practically everything you need to do can be carried out in just one place.

For instance, if you need to implement high availability or perform backups, you can easily apply redundancy in a cost-effective way – e.g. through virtualisation – if you already have a server-based system. Aside from cost-savings in SOX 404 implementation, server-based systems also offer a host of other benefits. Click that link to learn more.

Not sure how to get started on a cost-effective IT compliance initiative for SOX? You might want to read our post How To Get Started With Your IT Compliance Efforts for SOX.?

Ready to work with Denizon?