Directions Hadoop is Moving In

Hadoop is a data system so big it is like a virtual jumbo where your PC is a flea. One of the developers named it after his kid?s toy elephant so there is no complicated acronym to stumble over. The system is actually conceptually simple. It has loads of storage capacity and an unusual way of processing data. It does not wait for big files to report in to its software. Instead, it takes the processing system to the data.

The next question is what to do with Hadoop. Perhaps the question would be better expressed as, what can we do with a wonderful opportunity that we could not do before. Certainly, Hadoop is not for storing videos when your laptop starts complaining. The interfaces are clumsy and Hadoop belongs in the realm of large organisations that have the money. Here are two examples to illustrate the point.

Hadoop in Healthcare

In the U.S., healthcare generates more than 150 gigabytes of data annually. Within this data there are important clues that online training provider DeZyre believes could lead to these solutions:

  • Personalised cancer treatments that relate to how individual genomes cause the disease to mutate uniquely
  • Intelligent online analysis of life signs (blood pressure, heart beat, breathing) in remote children?s hospitals treating multiple victims of catastrophes
  • Mining of patient information from health records, financial status and payroll data to understand how these variables impact on patient health
  • Understanding trends in healthcare claims to empower hospitals and health insurers to increase their competitive advantages.
  • New ways to prevent health insurance fraud by correlating it with claims histories, attorney costs and call centre notes.

Hadoop in Retail

The retail industry also generates a vast amount of data, due to consumer volumes and multiple touch points in the delivery funnel. Skillspeed business trainers report the following emerging trends:

  • Tracing individual consumers along the marketing trail to determine individual patterns for different demographics and understand consumers better.
  • Obtaining access to aggregated consumer feedback regarding advertising campaigns, product launches, competitor tactics and so on.
  • Staying with individual consumers as they move through retail outlets and personalising their experience by delivering contextual messages.
  • Understanding the routes that virtual shoppers follow, and adding handy popups with useful hints and tips to encourage them on.
  • Detecting trends in consumer preferences in order to forecast next season sales and stock up or down accordingly.

Where to From Here?

Big data mining is akin to deep space research in that we are exploring fresh frontiers and discovering new worlds of information. The future is as broad as our imagination.?

Contact Us

  • (+353)(0)1-443-3807 – IRL
  • (+44)(0)20-7193-9751 – UK

Check our similar posts

The General Data Protection Regulation & The Duty to use Encryption

The General Data Protection Regulation, abbreviated to GDPR, raised a storm when it arrived. In reality, it merely tightened up on existing good practice according to digital security specialists Gemalto. The right to withhold consent and to be forgotten has always been there, for example. However, the GDPR brings a free enforcement service for consumers, thus avoiding the need for third party, paid assistance.

The GDPR Bottom Lines for Data Security
Moreover, the GDPR has penalties it can apply, of the order that might have a judge choking on his wig. Under it, data security measures such as pseudonymisation (substitution of identifying fields) and encryption (encoding including password protection) have become mandatory. Businesses must further respect their client data by:

a) Storing it in a secure environment supported by robust services and systems

b) Having proven measures to restore availability and access after a breach

c) Being able to prove frequent effectiveness testing of these measures.

The General Data Protection Regulation places an onus on businesses to report any data breaches. This places us in a difficult situation. We must either face at least a wrist slap upon reporting failures. Alternatively, pay a fine of up to ?10 million, or 2% of total worldwide annual turnover.

The Engineered Weak Link in the System
Our greatest threat of breach is probably when the data leaves our secure environment, and travels across cyberspace to an employee, stakeholder, collaborator, or the client themselves. Since email became open to attack, businesses and individuals have turned to sharing platforms like Dropbox, Google Drive, Skydrive, and so on. While these do allow an additional layer of password protection, none of these has proved foolproof. The GDPR may still fine us heavily, whether or not we are to blame for the actual breach.

How Hacking is Approaching Being a Science
We may make a mistake we may regret, if we do not take hacking seriously. The 10 worst data hacks Identity Force lists are proof positive that spending lots of money does not guarantee security (any more than having the biggest stock of nuclear weapons). We have to be smart, and start thinking the way that hackers do.

Hacker heaven is finding an Experian or a Dun & Bradstreet that may have shielded 143 million, and 33 million consumer records respectively, behind a single, flimsy cyber-security door. Ignorance is no excuse for them. They should simply have known better. They should have rendered consumer data unreadable at individual record level. The hackers could have found this too demanding to unpick, and have looked elsewhere.

How Data Encryption Can Help Prevent Hackers Succeeding
Encrypting data is dashboard driven, and businesses need not concern themselves about it works. There are, however, a few basic decisions they must take:

a) Purge the database of all information held without explicit permission

b) Challenge the need for the remaining data and purge the nice-to-haves

c) Adopt a policy of encrypting access at business and customer interfaces

d) Register with three freemium encryption services that seem acceptable

e) After experimenting, sign up for a premium service and be prepared to pay

Factors to Consider When Reaching a Decision
Life Hacker?suggests the following criteria although the list is a one-size-fits-all

a) Is the system fast, simple, and easy to operate

b) Can you encrypt hidden volumes within volumes

c) Can you mass-encrypt a batch of files easily

d) Do all other files remain encrypted when you open one

e) Do files automatically re-encrypt when you close them

f) How confident are you with the vendor, on a scale of 1 to 10

It may be wise to encrypt all the files on your system, and not just your customer data. We are always open to a hack by the competition after our strategic planning. If we leave the decision up to IT, then IT, being human may take the easy way out, and encrypt as little as possible.

Contact Us

  • (+353)(0)1-443-3807 – IRL
  • (+44)(0)20-7193-9751 – UK
What Energy Management Software did for CDC

Chrome Deposit Corporation ? that’s CDC for short ? reconditions giant rollers used to finish steel and aluminium sheets in Portage, Indiana by applying grinding, texturing and plating methods. While management was initially surprised when the University of Delaware singled their plant out for energy assessment, this took them on a journey to bring energy consumption down despite being in an expansion phase.

Metal finishing and refinishing is an energy-intensive business where machines mainly do the work while workforces as small as 50 individuals tend them. Environmental impacts also need countering within a challenging environment of burgeoning natural gas and electricity prices.

The Consultant’s Recommendations

The University of Delaware was fortunate that Chrome Deposit Corporation had consistently measured its energy consumption since inception in 1986. This enabled it to pinpoint six strategies as having potential for technological and process improvements.

  • Insulate condensate tanks and pipes
  • Analyse flue gas air-fuel ratios
  • Lower compressed air pressures
  • Install stack dampers on boilers
  • Replace belts with pulleys and cogs
  • Fit covers on plant exhaust fans

CDC implemented only four of the six recommendations. This was because the boiler manufacturer did not recommend stack dampers, and the company was unable to afford certain process automation and controls.

Natural Gas Savings

The project team began by analysing stack gases from boilers used to heat chrome tanks and evaporate wastewater. They found the boilers were burning rich and that several joints in gas lines were leaking. Correcting these issues achieved an instant gas saving of 12% despite increased production.

Reduced Water Consumption

The team established that city water was used to cool the rectifiers. It reduced this by an astonishing 85% by implementing a closed-loop system and adding two chillers. This also helped the water company spend less on chemicals, and energy to drive pumps, purifiers and fans.

Summary of Benefits

Electricity consumption reduced by 18% in real terms, and natural gas by 35%. When these two savings are merged they represent an overall 25% energy saving. These benefits were implemented across the company?s six other plants, resulting in benefits CDC management never dreamed of when the University of Delaware approached them.

ecoVaro offers a similar data analytics service that is available online worldwide. We have helped other companies slash their energy bills with similarly exciting results. We?ll be delighted to share ideas that only data analytics can reveal.

Contact Us

  • (+353)(0)1-443-3807 – IRL
  • (+44)(0)20-7193-9751 – UK
Symbion Pharmacy Services? Definition of Responsibility

A ?symbion? is an organism in a symbiotic (i.e. mutually beneficial) relationship with another one. In the case of Australia?s giant Symbion Pharmacy Services, this means supplying and delivering over-counter Chemmart medicines to more than 3,000 hospital and retail pharmacies, while remaining mindful of its carbon footprint.

In 1999, the company with the tagline ?life matters? and a desire to be seen as ?a good corporate citizen? decided it was time to measure exactly what it was pumping out from 12 facilities and over 200 vehicles. This was a voluntary decision as even now there is still no carbon emissions law in Australia (although no doubt being a ?first mover? will put the company in a competitive position when this inevitably comes).

Symbion decided to install emission detection devices and connect these to a central monitoring system with the intention of managing what these measured. There were two stages to this process. First, Symbion determined its reporting requirements based on one of its larger warehouses. Following that, it established a carbon footprint for each of its wholly owned and managed facilities. This put it in a position to:

  • Analyse total emissions down to a level of detail where it understood the contribution of each source
  • Use big data management tools to identify carbon hotspots for priority remedial action
  • Inform the affected workforce, explain the monitoring system and keep them in the loop
  • Separately manage energy abatement programs such as lighting and delivery routes

The program also had productivity spin-offs in that it focused management attention on the processes behind the emissions that were ripe for material and system improvements. It also provided marketing leverage. Symbion?s customers are in the wellness business, ahead of the curve when it comes to how emissions contribute to chronic illness, and aware of the cost of this in terms of human capital.

EcoVaro could help you manage your throughputs by analysing your data on our cloud-based system. This includes trending your metrics, comparing them to your industry seasonal average, and providing you with a business-like view of how well you are doing.

Our service reduces your reliance on (and the cost of) third party audits, and simplifies the reporting process to your controlling authority. It simply makes more sense to contract your software out this way, and only pay for it when you need it.

Ready to work with Denizon?

Contact Us Today