Directions Hadoop is Moving In

Hadoop is a data system so big it is like a virtual jumbo where your PC is a flea. One of the developers named it after his kid?s toy elephant so there is no complicated acronym to stumble over. The system is actually conceptually simple. It has loads of storage capacity and an unusual way of processing data. It does not wait for big files to report in to its software. Instead, it takes the processing system to the data.

The next question is what to do with Hadoop. Perhaps the question would be better expressed as, what can we do with a wonderful opportunity that we could not do before. Certainly, Hadoop is not for storing videos when your laptop starts complaining. The interfaces are clumsy and Hadoop belongs in the realm of large organisations that have the money. Here are two examples to illustrate the point.

Hadoop in Healthcare

In the U.S., healthcare generates more than 150 gigabytes of data annually. Within this data there are important clues that online training provider DeZyre believes could lead to these solutions:

  • Personalised cancer treatments that relate to how individual genomes cause the disease to mutate uniquely
  • Intelligent online analysis of life signs (blood pressure, heart beat, breathing) in remote children?s hospitals treating multiple victims of catastrophes
  • Mining of patient information from health records, financial status and payroll data to understand how these variables impact on patient health
  • Understanding trends in healthcare claims to empower hospitals and health insurers to increase their competitive advantages.
  • New ways to prevent health insurance fraud by correlating it with claims histories, attorney costs and call centre notes.

Hadoop in Retail

The retail industry also generates a vast amount of data, due to consumer volumes and multiple touch points in the delivery funnel. Skillspeed business trainers report the following emerging trends:

  • Tracing individual consumers along the marketing trail to determine individual patterns for different demographics and understand consumers better.
  • Obtaining access to aggregated consumer feedback regarding advertising campaigns, product launches, competitor tactics and so on.
  • Staying with individual consumers as they move through retail outlets and personalising their experience by delivering contextual messages.
  • Understanding the routes that virtual shoppers follow, and adding handy popups with useful hints and tips to encourage them on.
  • Detecting trends in consumer preferences in order to forecast next season sales and stock up or down accordingly.

Where to From Here?

Big data mining is akin to deep space research in that we are exploring fresh frontiers and discovering new worlds of information. The future is as broad as our imagination.?

Contact Us

  • (+353)(0)1-443-3807 – IRL
  • (+44)(0)20-7193-9751 – UK

Check our similar posts

Authentication and Access Control

Threats to your data can come from external or internal sources.

  1. There are individuals who don’t have the authorisation but are driven by malicious intentions to gain access to certain information. This may refer to individuals who already belong to your organisation (but don’t have the necessary access rights) as well as those who don’t.
  2. There are individuals who have both the authorisation and, unfortunately, the malicious intentions over certain information.
  3. Finally, there are individuals who have the authorisation, no malicious intentions, but have accidentally exposed the information in question to those without the proper authority.

While curbing threats 2 and 3 would require other methods, threat #1 can be countered if the right authentication and access control systems are in place.

Here’s what we can do for you:

  • Work with your key personnel to determine who gets access to what.
  • Help you decide whether a single factor or a two-factor authentication (2FA) is appropriate for your organisation and recommend which factors are most suitable. Login methods may include but are not limited to the following:
    • biometric devices
    • Kerberos tickets
    • mobile phones
    • passwords
    • PKI certificates
    • proximity cards
    • smart cards
    • tokens
  • Install the necessary infrastructure needed for the factors chosen. For instance, if you opt to use biometrics, then biometric scanners will be installed. We’ll make sure that the authentication terminals are situated in places where achieving optimal traffic and work flow has been taken into consideration.

Other defences we’re capable of putting up include:

2015 ESOS Guidelines Chapter 7, 8 & 9 – Sign-Off, Compliance & Appeals

This is the final chapter in our series of short posts summarising the quite complex ESOS guidelines (click on ?Comply with ESOS? to see the details). This one addresses the legalities to follow to complete your report – and how to appeal if you are not happy with any of the Environment Agency?s decisions.

  1. Director Sign-Off

This is by no means an easy ride. Confirmation of the work at individual or lead assessor level locks the company into the penalty cycle in the event there are significant irregularities. By signing off the assessment, the board level director(s) # agree that they have

  • Reviewed the enterprise?s ESOS recommendations
  • Believe the enterprise is within the scope of the scheme
  • Believe the enterprise is compliant with the scheme
  • Believe the information provided is correct

Having an internal assessor requires a second board-level signature.

  1. Compliance

You report compliance on the internet. This is free and you can do it at any time within the deadline. You can dip in and out of the process as many times as you wish, but must use the link in the receipting email. While this is something a board member must do, there is no reason why the lead assessor should not complete the basics. The online compliance notification addresses the following topics:

  • The ESOS contact person in the enterprise
  • Any aggregation / dis-aggregation during the period
  • The names and contact details of the lead assessor
  • The proportion of energy consumption per compliance route

The Environment Agency will acknowledge receipt. This does not constitute acceptance. You should keep the ESOS evidence pack in a safe place with at least one backup elsewhere.

  1. Compliance & Enforcement Issues

In the event the Environment Agency decides your enterprise has not met ESOS requirements, it may either (a) issue a compliance notice with instructions, or (b) apply one of the following civil penalties:

  • A fine of up to ?5,000 for failure to maintain records
  • A fine of up to ?50,000 for failure to undertake an energy audit
  • A fine of up to ?50,000 for a false or misleading statement

Any enterprise has the right of appeal against government decisions. In the case of ESOS, this is via:

  • The First-Tier Tribunal if your enterprise is England, Wales or off-shore based
  • The Scottish Minister if your enterprise is based in Scotland
  • The Planning Commission if your enterprise is Northern Ireland-based

The notice you appeal against will supply details of the appeal steps to take.

This blog and its companion chapters concerning the ESOS Guidelines as amended 2015 are with compliments of ecoVaro. We are the people who break ESOS data into manageable chunks of information, so that board-level directors have greater confidence in what they sign.

The Better Way of Applying Benford’s Law for Fraud Detection

Applying Benford’s Law on large collections of data is an effective way of detecting fraud. In this article, we?ll introduce you to Benford’s Law, talk about how auditors are employing it in fraud detection, and introduce you to a more effective way of integrating it into an IT solution.

Benford’s Law in a nutshell

Benford’s Law states that certain data sets – including certain accounting numbers – exhibit a non-uniform distribution of first digits. Simply put, if you gather all the first digits (e.g. 8 is the first digit of ?814 and 1 is the first digit of ?1768) of all the numbers that make up one of these data sets, the smallest digits will appear more frequently than the larger ones.

That is, according to Benford’s Law,

1 should comprise roughly 30.1% of all first digits;
2 should be 17.6%;
3 should be 12.5%;
4 should be 9.7%, and so on.

Notice that the 1s (ones) occur far more frequently than the rest. Those who are not familiar with Benford’s Law tend to assume that all digits should be distributed uniformly. So when fraudulent individuals tinker with accounting data, they may end up putting in more 9s or 8s than there actually should be.

Once an accounting data set is found to show a large deviation from this distribution, then auditors move in to make a closer inspection.

Benford’s Law spreadsheets and templates

Because Benford’s Law has been proven to be effective in discovering unnaturally-behaving data sets (such as those manipulated by fraudsters), many auditors have created simple software solutions that apply this law. Most of these solutions, owing to the fact that a large majority of accounting departments use spreadsheets, come in the form of spreadsheet templates.

You can easily find free downloadable spreadsheet templates that apply Benford’s Law as well as simple How-To articles that can help you to implement the law on your own existing spreadsheets. Just Google “Benford’s law template” or “Benford’s law spreadsheet”.

I suggest you try out some of them yourself to get a feel on how they work.

The problem with Benford’s Law when used on spreadsheets

There’s actually another reason why I wanted you to try those spreadsheet templates and How-To’s yourself. I wanted you to see how susceptible these solutions are to trivial errors. Whenever you work on these spreadsheet templates – or your own spreadsheets for that matter – when implementing Benford’s Law, you can commit mistakes when copy-pasting values, specifying ranges, entering formulas, and so on.

Furthermore, some of the data might be located in different spreadsheets, which can likewise by found in different departments and have to be emailed for consolidation. The departments who own this data will have to extract the needed data from their own spreadsheets, transfer them to another spreadsheet, and send them to the person in-charge of consolidation.

These activities can introduce errors as well. That’s why we think that, while Benford’s Law can be an effective tool for detecting fraud, spreadsheet-based working environments can taint the entire fraud detection process.

There?s actually a better IT solution where you can use Benford’s Law.

Why a server-based solution works better

In order to apply Benford’s Law more effectively, you need to use it in an environment that implements better controls than what spreadsheets can offer. What we propose is a server-based system.

In a server-based system, your data is placed in a secure database. People who want to input data or access existing data will have to go through access controls such as login procedures. These systems also have features that log access history so that you can trace who accessed which and when.

If Benford’s Law is integrated into such a system, there would be no need for any error-prone copy-pasting activities because all the data is stored in one place. Thus, fraud detection initiatives can be much faster and more reliable.

You can get more information on this site regarding the disadvantages of spreadsheets. We can also tell you more about the advantages of server application solutions.

Ready to work with Denizon?

Contact Us Today