Directions Hadoop is Moving In

Hadoop is a data system so big it is like a virtual jumbo where your PC is a flea. One of the developers named it after his kid?s toy elephant so there is no complicated acronym to stumble over. The system is actually conceptually simple. It has loads of storage capacity and an unusual way of processing data. It does not wait for big files to report in to its software. Instead, it takes the processing system to the data.

The next question is what to do with Hadoop. Perhaps the question would be better expressed as, what can we do with a wonderful opportunity that we could not do before. Certainly, Hadoop is not for storing videos when your laptop starts complaining. The interfaces are clumsy and Hadoop belongs in the realm of large organisations that have the money. Here are two examples to illustrate the point.

Hadoop in Healthcare

In the U.S., healthcare generates more than 150 gigabytes of data annually. Within this data there are important clues that online training provider DeZyre believes could lead to these solutions:

  • Personalised cancer treatments that relate to how individual genomes cause the disease to mutate uniquely
  • Intelligent online analysis of life signs (blood pressure, heart beat, breathing) in remote children?s hospitals treating multiple victims of catastrophes
  • Mining of patient information from health records, financial status and payroll data to understand how these variables impact on patient health
  • Understanding trends in healthcare claims to empower hospitals and health insurers to increase their competitive advantages.
  • New ways to prevent health insurance fraud by correlating it with claims histories, attorney costs and call centre notes.

Hadoop in Retail

The retail industry also generates a vast amount of data, due to consumer volumes and multiple touch points in the delivery funnel. Skillspeed business trainers report the following emerging trends:

  • Tracing individual consumers along the marketing trail to determine individual patterns for different demographics and understand consumers better.
  • Obtaining access to aggregated consumer feedback regarding advertising campaigns, product launches, competitor tactics and so on.
  • Staying with individual consumers as they move through retail outlets and personalising their experience by delivering contextual messages.
  • Understanding the routes that virtual shoppers follow, and adding handy popups with useful hints and tips to encourage them on.
  • Detecting trends in consumer preferences in order to forecast next season sales and stock up or down accordingly.

Where to From Here?

Big data mining is akin to deep space research in that we are exploring fresh frontiers and discovering new worlds of information. The future is as broad as our imagination.?

Contact Us

  • (+353)(0)1-443-3807 – IRL
  • (+44)(0)20-7193-9751 – UK

Check our similar posts

What is Business Intelligence?

How well do you know your customers? That is, can you actually pinpoint which among them are you most profitable with and which are making you spend more? Are you content with the accuracy of your forecasts and market predictions? Do you feel you’re spending more on legal costs and regulatory compliance than you should?

Your IT department may be handling these concerns pretty well but perhaps you’d like to know how you can further improve things.

What we’ve got is an IT solution wrapped in a fancy name called ‘Business Intelligence’ or BI. If you think that’s too strong a term, we invite you to read more below, then you be the judge.

Dashboards – Determine the health of your business at a glance

Most drivers rarely make use of their car’s dashboard. After all, you can still reach your destination by just using the steering wheel, pedals, gear stick and so on. But that’s not exactly the most efficient way to drive, right?

If you want to save on fuel, you’ll want to glance on the RPM and speedometer from time to time. You might also want to utilise the trip meter to determine which route is the shortest to a given destination. Other dashboard components like the fuel gauge, tire pressure gauge, engine temperature indicator, and volt meter can likewise provide information about your car’s health.

The same concept applies to business management. If you want to run your business intelligently, you can make use of BI dashboards. These are tools in a typical business intelligence package that will allow you to determine the health of your business via a set of smartly configured gauges and other intuitive graphical representations.

So that, literally, at a mere glance, you’ll already know whether various units in your company are working efficiently. A dashboard will also give you instant feedback of the strategies you’ve recently implemented; to let you know if things are working as planned.

If you want more information than a dashboard can provide, our BI packages also include highly customised reports.

Reports that help you decide faster

Dashboards are great for getting valuable information at a glance but they won’t tell you everything. For more details, you’ll need to view highly customised reports. Our reports are tailor made for each user. We see to it that, by default, each person gets the information he needs the most.

If you belong to the sales department, you normally won’t need a presentation of the data that is appropriate for people in accounting. That way, you don’t spend time filtering. Instead, you and your people can move on to making well-informed decisions.

Our BI systems make use of your vast collection of data to provide reports that will organise your regulatory requirements and call your attention to approaching deadlines. The same system will provide the right information for your people on the field. If your team members are equipped with smart phones and Pocket PCs, they can retrieve whatever it is they need to know to close deals, make sales, and serve clients faster than the competition.

Generating logical information from disparate sources of data scattered over an enterprise-wide organisation is no easy task. But we’ll make it look simple. That’s because we’ve got the expertise to bring it all together into a robust data warehouse and to extract them in the form of reports and dashboards through OLAP.

OLAP and Data Warehousing – Powering the generation of actionable information

Want to know how to generate reports with the highest degree of accuracy and reliability? In theory, what you need is a single repository or a data warehouse. That is, order receipts, sales invoices, as well as customer & supplier data is integrated with regulatory details, personnel data, and others. These are all specially organised for future reporting and analysis.

However, data, no matter how all-embracing, is useless until it is processed into actionable information. Through OLAP or Online Analytical Processing, you can seamlessly collect all relevant data from your vast repository to answer queries like “What is our company’s profitability for the 2nd quarter in all identified key cities for our top-of-the-line products?”.

The strength of OLAP lies in its inherent ability to perform data analysis and very complex calculations, thus enabling it to return complex queries much faster than other database technologies. It is therefore suitable for very large data sources, i.e., data warehouses.

Dashboards and reports will only give your organisation the edge if the information retrieved is reliable, fast, and accurate – exactly the kind OLAP is so good at.

Mobile BI – Step back and see the big picture anytime, anywhere

Spreadsheets are great for displaying detailed information. However, in today’s highly competitive market, retrieving information that matters the most in the shortest possible time is vital in maintaining a sizeable lead over the competition. To step back and see the big picture, you’ll need insightful tools like dashboards and automatically generated reports.

Reports can be beamed to mobile devices such as smart phones and Pocket PCs. They can also be viewed on eBook readers as well. You can also do the same tasks with spreadsheets. But imagine how you’d need to scroll over a large spreadsheet on any of these mobile devices just to know which customer in your current location has performed well over the last month.

If you really want to make quick, well-informed decisions, BI dashboards for mobile devices is the way to go. You can make use of various business objects such as drill-able charts, performance metrics, and metric trend graphs to make crucial decisions even when on you’re in the field.

2015 ESOS Guidelines Chapter 2 – Deadlines and Status Changes

The ESOS process is deadline driven and meeting key dates is a non-negotiable. The penalties for not complying / providing false or misleading information are ?50,000 each. Simply not maintaining adequate records could cost you ?5,000. The carrot on the end of the stick is the financial benefits you stand to gain.

Qualifying for inclusion under the ESOS umbrella depends on the status of your company in terms of employee numbers, turnover and balance sheet on 31 December 2014. Regardless of whether you meet the 2014 threshold or not, you must reconsider your situation on 31 December 2018, 2022 and 2026.

Compliance Period Qualification Date Compliance Period Compliance Date
1 31 December 2014 From 17 July 2014* to 5 December 2015 5 December 2015
2 31 December 2018 From 6 December 2015 to 5 December 2019 5 December 2019
3 31 December 2022 From 6 December 2019 to 5 December 2023 5 December 2023
4 31 December 2026 From 6 December 2023 to 5 December 2027 5 December 2027

Notes:

1. The first compliance period begins on the date the regulations became effective

2. Energy audits from 6 December 2011 onward may go towards the first compliance report

Changes in Organisation Status

If your organisation status changes after a qualification date when you met compliance thresholds, you are still bound to complete your ESOS assessment for that compliance period. This is regardless of any change in size or structure. Your qualification status then remains in force until the next qualification date when you must reconsider it.

How To Get Started with your IT Compliance Efforts for SOX

There’s no question about it. For many of you top executives in the corporate world, all roads leading to a brighter future have to go through SOX compliance. And because the business processes that contribute to financial reporting (the crux of the Sarbanes-Oxley Act) are now highly reliant on IT systems, it is important to focus a good part of your attention there.

It is a long and arduous path to IT compliance, so if you don’t want your company to fall by the wayside due to inefficient utilisation of resources, it is important to set out with a plan on hand. What we have here are some vital information that will guide you in putting together a sound plan for SOX compliance of your company?s IT systems.

Why focus on IT systems for SOX compliance?

We’ll get to that. But first, let’s take up the specific portions of the Sarbanes-Oxley Act that affect information technology. These portions can be found in Section 302 and Section 404 of the act.

In simplified form, Section 302 grants the SEC (Securities and Exchange Commission) authority to come up with rules requiring you, CEOs and CFOs, to certify in each annual or quarterly financial report the following:

  • that you have reviewed the report;
  • that based on your knowledge, the report does not contain anything or leave out anything that would render it misleading;
  • that based on your knowledge, all financial information in the report fairly represent the financial conditions of the company;
  • that you are responsible for establishing internal controls over financial reporting; and
  • that you have assessed the effectiveness of the internal controls.

Similarly, Section 404, stated in simplified form, allows the SEC to come up with rules requiring you, CEOs and CFOs, to add an internal control report to each annual financial report stating that you are responsible for establishing internal controls over financial reporting.

You are also required to assess the effectiveness of those controls and to have a public accounting firm to attest to your assessment based upon standards adopted by the Public Company Accounting Oversight Board (PCAOB).

While there is no mention of IT systems, IT systems now play a significant role in financial reporting. Practically all of the data you need for your financial reports are stored, retrieved and processed on IT systems, so you really have to include them in your SOX compliance initiatives and establish controls on them.

Now that that’s settled, your next question could very well be: How do you know what controls to install and whether those controls are already sufficient to achieve compliance?

Finding a suitable guide for IT compliance

The two bodies responsible for setting rules and standards dealing with SOX, SEC and PCAOB, point to a well-established control framework for guidance – COSO. This framework was drafted by the Committee of Sponsoring Organisations of the Treadway Commission (COSO) and is the most widely accepted control framework in the business world.

However, while COSO is a tested and proven framework, it is more suitable for general controls. What we recommend is a widely-used control framework that aligns well with COSO but also caters to the more technical features and issues that come with IT systems.

Taking into consideration those qualifiers, we recommend COBIT. COBIT features a well thought out collection of IT-related control objectives grouped into four domains: Plan and Organise (PO), Acquire and Implement (AI), Deliver and Support (DS), and Monitor and Evaluate (ME). The document also includes maturity models, performance goals and metrics, and activity goals.

A few examples of COBIt’s detailed control objectives are:

DS4.2 – IT Continuity Plans
DS4.9 – Offsite Backup Storage
DS5.4 – User Account Management
DS5.8 – Cryptographic Key Management
DS5.10 – Network Security
DS5.11 – Exchange of Sensitive Data

By those titles alone, you can see that the framework is specifically designed for IT. But the document is quite extensive and, chances are, you won’t need all of the items detailed there. Furthermore, don’t expect COBIT to specify a control solution controls for every control objective. For example, throughout the control objective DS4 (Ensure Continuous Service), you won’t find any mention of virtualisation, which is common in any modern business continuity solution.

Basically, COBIT will tell you what you need to attain in order to achieve effective governance, management and control, but you’ll have to pick the solution best suited to reach that level of attainment.

Articles highly relevant to the one you just read:

Month End Accounting The Way It Should Be Today
Spreadsheet Woes ? Burden in SOX Compliance and Other Regulations
Spreadsheet Woes ? Limited Features For Easy Adoption of a Control Framework
How Internal Auditors Can Win The War Against Spreadsheet Fraud

Ready to work with Denizon?