Are Master Data Management and Hadoop a Good Match?

Master Data is the critical electronic information about the company we cannot afford to lose. Accordingly, we should sanitise it, look after it, and store it safely in several separate places that are independent of each other. The advent of Big Data introduced the current era of huge repositories ?in the clouds?. They are not, of course but at least they are remote. This short article includes a discussion about Hadoop, and whether this is a good platform to back up your Master Data.

About Hadoop

Hadoop is an open-source Apache software framework built on the assumption that hardware failure is so common that backups are unavoidable. It comprises a storage area and a management part that distributes the data to smaller nodes where it processes faster and more efficiently. Prominent users include Yahoo! and Facebook. In fact more than half Fortune 50 companies were using Hadoop in 2013.

Hadoop – initially launched in December 2011 ? has survived its baptism of fire and became a respected, reliable option. But is this something the average business owner can tackle on their own? Bear in mind that open source software generally comes with little implementation support from the vendor.

The Hadoop Strong Suite

  • Free to download, use and contribute to
  • Everything you need ?in the box? to get started
  • Distributed across multiple fire-walled computers
  • Fast processing of data held in efficient cluster nodes
  • Massive scaleable storage you are unlikely to run out of

Practical Constraints

There is more to Hadoop than writing to WordPress. The most straightforward solutions are uploading using Java commands, obtaining an interface mechanism, or using third party vendor connectors such as ACCESS or SAS. The system does not replace the need for IT support, although it is cheap and exceptionally powerful.

The Not-Free Safer Option

Smaller companies without in-depth in-house support are wise to engage with a technical intermediary. There are companies providing commercial implementations followed by support. Microsoft, Amazon and Google among others all have commercial versions in their catalogues, and support teams at the end of the line.

Check our similar posts

Why Predictive Maintenance is More Profitable than Reactive Maintenance

Regular maintenance is needed to keep the equipment in your facility operating normally. All machinery has a design lifespan, and your goal is to extend this as long as possible, while maintaining optimal production levels. How you go about the maintenance matters, from routine checks to repairing the damaged component parts?all before the whole unit needs to be tossed away and a new one purchased and installed. Here, we will break down the different approaches used, and show you why more industries and businesses are turning to proactive maintenance modes as opposed to the traditional reactive approaches for their?field service operations.?

Reactive Maintenance: A wait and see game

Here, you basically wait for a problem to occur, then fix it. It’s also commonly referred to as a “Run-to-Failure” approach, where you operate the machines and systems until they break. Repairs are then carried out, restoring it to operational condition.?

At face value, it appears cost-effective, but the reality on the ground is far much different. Sure, when the equipment is new, you can expect minimal cases of maintenance. During this time, there?ll be money saved. However, as time progresses there?ll be increased wear, making reliance on a reactive maintenance approach a costly endeavour. The breakdowns are more frequent, and inconsistent as well. Unplanned expenses increase operational costs, and there will be lost productivity during the periods in which the affected machinery won’t be in operation.?

While reactive maintenance makes sense when you’re changing a faulty light bulb at home, things are more complicated when it comes to dealing with machinery in industries, or for those managing multiple residential and commercial properties. For the light bulb, it’s easier to replace it, and failure doesn’t have a ripple effect on the rest of the structures in the household. For industries, each time there is equipment failure, you end up with downtime, production can grind to a halt, and there will be increased environmental risks during equipment start-up and shutdown. If spare parts are not readily available, there will be logistical hurdles as you rush the shipping to get the component parts to the facility. Add this to overworked clients in a bit to complete the repair and to make up for lost hours and delayed customer orders.

For field service companies, more time ends up being spent. After all, there?s the need of knowing which parts needed to be attended to, where they are, and when the servicing is required. Even when you have a planned-out schedule, emergency repairs that are required will force you to immediately make changes. These ramps up the cots, affecting your operations and leading to higher bills for your client. These inconveniences have contributed to the increased reliance on?field service management platforms that leverage on data analytics and IoT to reduce the repair costs, optimise maintenance schedules, and?reduce unnecessary downtimes?for the clients.

Waiting for the machinery to break down actually shortens the lifespan of the unit, leading to more replacements being required. Since the machinery is expected to get damaged much sooner, you also need to have a large inventory of spare parts. What’s more, the damages that result will be likely to necessitate more extensive repairs that would have been needed if the machinery had not been run to failure.?

Pros of reactive maintenance

  1. Less staff required.
  2. Less time is spent on preparation.

Cons of reactive maintenance

  1. Increased downtime during machine failure.
  2. More overtime is taken up when conducting repairs.
  3. Increased expenses for purchasing and storing spare parts.?
  4. Frequent equipment replacement, driving up costs.?

This ?If it ain’t broke, don’t fix it? approach leads to hefty repair and replacement bills. A different maintenance strategy is required to minimise costs. Proactive models come into focus. Before we delve into predictive maintenance, let’s look at the preventive approach.?

Preventive Maintenance: Sticking to a timetable

Here, maintenance tasks are carried out on a planned routine?like how you change your vehicle?s engine oil after hitting a specific number of kilometres. These tasks are planned in intervals, based on specific triggers?like a period of time, or when certain thresholds are recorded by the meters. Lubrication, carrying out filter changes, and the like will result in the equipment operating more efficiently for a longer duration of time. While it doesn’t completely stop catastrophic failures from occurring, it does reduce the number of failures that occur. This translates to capital savings.??

The Middle Ground? Merits And Demerits Of Preventive Maintenance

This periodic checking is a step above the reactive maintenance, given that it increases the lifespan of the asset, and makes it more reliable. It also leads to a reduced downtime, thus positively affecting your company?s productivity. Usually, an 80/20 approach is adopted,?drawing from Pareto’s Principle. This means that by spending 80% of time and effort on planned and preventive maintenance, then reactive maintenance for those unexpected failures that pop up will only occur 20% of the time. Sure, it doesn’t always come to an exact 80/20 ratio, but it does help in directing the maintenance efforts of a company, and reducing the expenses that go into it.?

Note that there will need to be a significant investment?especially of time, in order to plan a preventive maintenance strategy, plus the preparation and delegation of tasks. However, the efforts are more cost effective than waiting for your systems and machinery to fail in order to conduct repairs. In fact, according to the US Dept. of Energy, a company can save between 12-18 % when using a preventive maintenance approach compared to reactive maintenance.

While it is better than the purely reactive approach, there are still drawbacks to this process. For instance, asset failure will still be likely to occur, and there will be the aspect of time and resource wastage when performing unneeded maintenance, especially when technicians have to travel to different sites out in the field. There is also the risk of incidental damage to machine components when the unneeded checks and repairs are being carried out, leading to extra costs being incurred.

We can now up the ante with predictive maintenance. Let’s look at what it has to offer:

Predictive Maintenance: See it before it happens

This builds on preventive maintenance, using data analytics to smooth the process, reduce wastage, and make it more cost effective. Here, the maintenance is conducted by relying on trends observed using data collected from the equipment in question, such as through vibration analysis, energy consumption, oil analysis and thermal imaging. This data is then taken through predictive algorithms that show trends and point out when the equipment will need maintenance. You get to see unhealthy trends like excessive vibration of the equipment, decreasing fuel efficiency, lubrication degradation, and their impact on your production capacities. Before the conditions breach the predetermined parameters of the equipment’s normal operating standards, the affected equipment is repaired or the damaged components replaced.??

Basically, maintenance is scheduled before operational or mechanical conditions demand it. Damage to equipment can be prevented by attending to the affected parts after observing a decrease in performance at the onset?instead of waiting for the damage to be extensive?which would have resulted in system failure. Using?data-driven?field service job management software will help you to automate your work and optimise schedules, informing you about possible future failures.

Sensors used record the condition of the equipment in real time. This information is then analysed, showing the current and future operational capabilities of the equipment. System degradation is detected quickly, and steps can be taken to rectify it before further deterioration occurs. This approach optimises operational efficiency. Firstly, it drastically reduces total equipment failure?coming close to eliminating it, extending the lifespan of the machinery and slashing replacement costs. You can have an orderly timetable for your maintenance sessions, and buy the equipment needed for the repairs. Speaking of which, this approach minimises inventory especially with regards to the spare parts, as you will be able to note the specific units needed beforehand and plan for them, instead of casting a wide net and stockpiling spare parts for repairs that may or may not be required. Repair tasks can be more accurately scheduled, minimising time wasted on unneeded maintenance.??

Preventive vs Predictive Maintenance?

How is predictive different from preventive maintenance? For starters, it bases the need for maintenance on the actual condition of the equipment, instead of a predetermined schedule. Take the oil-change on cars for instance. With the preventive model, the oil may be changed after every 5000?7500 km. Here, this change is necessitated because of the runtime. One doesn’t look at the performance capability and actual condition of the oil. It is simply changed because “it is now time to change it“. However, with the predictive maintenance approach, the car owner would ideally analyse the condition of the oil at regular intervals- looking at aspects like its lubrication properties. They would then determine if they can continue using the same oil, and extend the duration required before the next oil change, like by another 3000 kilometres. Perhaps due to the conditions in which the car had been driven, or environmental concerns, the oil may be required to be changed much sooner in order to protect the component parts with fresh new lubricant. In the long run, the car owner will make savings. The US Dept. of Energy report also shows that you get 8-12% more cost savings with the predictive approach compared to relying on preventive maintenance programs. Certainly, it is already far much more effective compared to the reactive model.?

Pros of Predictive Maintenance

  1. Increases the asset lifespan.
  2. Decreases equipment downtime.
  3. Decreases costs on spare parts and labour.
  4. Improves worker safety, which has the welcome benefit of increasing employee morale.
  5. Optimising the operation of the equipment used leads to energy savings.
  6. Increased plant reliability.

Cons of Predictive Maintenance

  1. Initial capital costs included in acquiring and setting up diagnostic equipment.
  2. Investment required in training the employees to effectively use the predictive maintenance technology adopted by the company.

The pros of this approach outweigh the cons.?Independent surveys on industrial average savings?after implementing a predictive maintenance program showed that firms eliminated asset breakdown by 70-75%, boosted production by 20-25%, and reduced maintenance costs by 25-30%. Its ROI was an average of 10 times, making it a worthy investment.

Benefits of Integrating IoT and Field Service

Owing to the complexity of its definition, many people loosely use the phrase Internet of Things (IoT) without having a solid grasp of its true meaning. A majority in this category take IoT to be nothing more than the automation of home gadgets, where the internet is used to interconnect computing components embedded in everyday devices.

Granted, the whole idea of IoT got its roots from the home setting. Nevertheless, IoT has outgrown that spectrum and has since penetrated into almost every area of business and industry. By employing IoT, you can literally take full control of everything in your business using a single device. From assigning tasks to monitoring security, managing bills to tracking time, IoT has revolutionized the way business is done.

Interestingly, not so long ago, most technology experts limited their forecasts to machine-to-machine (M2M) integration and Augmented Reality (AR), which also, admittedly, hit the technology industry with an admirable suave. Back then, it could have been laughable for anyone to have suggested that IoT would be so commanding in almost every industry, including real estate, medicine, automobile, and more.

It’s not for nothing, therefore, that the field service industry has also embraced IoT, integrating it in the daily running of business activities, including tracking machine diagnostics, detecting breakdowns, and assigning field engineers to attend to customer needs.

How the Field Service Industry is Benefiting from IoT

Machine uptime has remained an ongoing concern for many customers. In the traditional approach, whenever a machine breaks down, the customer alerts the service provider and then the field service manager checks to see if there is any field engineer available for a new task. Once an engineer has been identified, he?s then dispatched to the site. This worked, but it resulted in an extended machine downtime, a terrible experience for customers.

Thanks to IoT, things are now happening differently.

IoT is now integrating machines to a central communications centre, where all alerts and status updates are sent. The notifications are instant. The field service manager, therefore, gets to learn of the status of machines at the exact time of status change. An engineer who?s not engaged would then be immediately assigned to undertake any needed servicing or repair.

By employing IoT, the service provider receives timely reports relating to diagnostics, machine uptime, part failures, and more. The field manager can, as a result, foretell and forestall any possible downtime.

How has this been helpful?

Before giving a definite answer to that question, it’s crucial to note that more than half of all field service organizations now employ IoT in their Asset Management Systems and Field Service Management. And to answer the question, all the organizations that have the two systems integrated using IoT experience twice as much efficiency as those that don’t, states an Aberdeen Group report. As you already know, improved efficiency results in a corresponding upshot in customer satisfaction.

Apps Making a Difference in IoT-Field Service

The integration of IoT into almost every aspect of business prompted the design and development of different applications to link computing devices. Since the advent of IoT, the software development for the technology has come of age. Powerful and lightweight apps that don simple yet beautiful user interfaces are now readily available at affordable price tags.

A good example of such an App is ecoVaro by Denizon.

ecoVaro not only helps businesses to monitor energy and other relevant environmental data such as Electricity, Gas, Water, Oil, Carbon, Temperature, Humidity, Solar Power, and more, but also provides analytics and comprehensive yet easy to understand reports. The data received from devices such as meters is converted into useful information that’s then presented in figures and graphs, thus allowing you to make decisions based on laid down controls.

The focus of the app is to instantly alert service engineers to go on site to fix issues.

With ecoVaro, field service engineers no longer have to return to the office to get new instructions. Also, customers don’t have to manually fire alerts to the service provider whenever something isn’t working correctly. By employing the latest in IoT, ecoVaro sends notifications to field service managers and engineers about respective customers that need support.

How ecoVaro Helps

Best-in-class companies aren’t ready to compromise on customer satisfaction. Therefore, every available avenue is used to address customer concerns with the deserved agility. By using IoT, ecoVaro makes it possible for field service providers to foresee and foreclose any possible breakdowns.

The inter-connectivity among the devices and the central communications centre results in increased revenue and improved interactivity between the system and the field engineers. This results in greater efficiency and lower downtime, which translates into improved productivity, accountability, and customer satisfaction, as well as creating a platform for a possible expansion of your customer base.

ecoVaro isn’t just about failed machines and fixes. It also provides diagnostics about connected systems and devices. With this, the diagnostics centre receives system reports in a timely manner, allowing for ease of planning and despatch of field officers where necessary.

Clearly, but using the right application, IoT can transform your business into an excellently performing field service company.

How DevOps oils the Value Chain

DevOps ? a clipped compound of development and operations – is a way of working whereby software developers are in a team with project beneficiaries. A client centred approach extends the project plan to include the life cycle of the product or service, for which the software is developed.

We can then no longer speak of a software project for say Joe?s Accounting App. The software has no intrinsic value of its own. It follows that the software engineers are building an accounting app product. This is a small, crucially important distinction, because they are no longer in a silo with different business interests.

To take the analogy further, the developers are no longer contractors possibly trying to stretch out the process. They are members of Joe?s accounting company, and they are just as keen to get to market fast as Joe is to start earning income. DevOps uses this synergy to achieve the overarching business goal.

A Brief Introduction to OpsDev

You can skip this section if you already read this article. If not then you need to know that DevOps is a culture, not a working method. The three ?members? are the software developers, the beneficiaries, and a quality control mechanism. The developers break their task into smaller chunks instead of releasing the code to quality control as a single batch. As a result, the review process happens contiguously along these simplified lines.

Code QC Test ? ? ?
? Code QC Test ? ?
? ? Code QC Test ?
? ? ? Code QC Test
Colour Key Developers Quality Control Beneficiary

This is a marked improvement over the previously cumbersome method below.

Write the Code ? Test the Code ? Use the Code
? Evaluate, Schedule for Next Review ?

Working quickly and releasing smaller amounts of code means the OpsDev team learns quickly from mistakes, and should come to product release ahead of any competitor using the older, more linear method. The shared method of working releases huge resources in terms of user experience and in-line QC practices. Instead of being in a silo working on its own, development finds it has a richer brief and more support from being ?on the same side of the organisation?.

The Key Role that Application Program Interfaces Play

Application Program Interfaces, or API?s for short, are building blocks for software applications. Using proprietary software-bridges speeds this process up. A good example would be the PayPal applications that we find on so many websites today. API?s are not just for commercial sites, and they can reduce costs and improve efficiency considerably.

The following diagram courtesy of TIBCO illustrates how second-party applications integrate with PayPal architecture via an API fa?ade.

Working quickly and releasing smaller amounts of code means the OpsDev team learns quickly from mistakes, and should come to product release ahead of any competitor using the older, more linear method. The shared method of working releases huge resources in terms of user experience and in-line QC practices. Instead of being in a silo working on its own, development finds it has a richer brief and more support from being ?on the same side of the organisation?.

imgd2.jpg

The DevOps Revolution Continues ?

We close with some important insights from an interview with Jim Stoneham. He was general manager of the Yahoo Communities business unit, at the time Flickr became a part. ?Flickr was a codebase,? Jim recalls, ?that evolved to operate at high scale over 7 years – and continuing to scale while adding and refining features was no small challenge. During this transition, it was a huge advantage that there was such an integrated dev and ops team?

The ?maturity model? as engineers refer to DevOps status currently, enables developers to learn faster, and deploy upgrades ahead of their competitors. This means the client reaches and exceeds break-even sooner. DevOps lubricates the value chain so companies add value to a product faster. One reason it worked so well with Flickr, was the immense trust between Dev and Ops, and that is a lesson we should learn.

?We transformed from a team of employees to a team of owners. When you move at that speed, and are looking at the numbers and the results daily, your investment level radically changes. This just can’t happen in teams that release quarterly, and it’s difficult even with monthly cycles.? (Jim Stoneham)

Contact Us

  • (+353)(0)1-443-3807 – IRL
  • (+44)(0)20-7193-9751 – UK

Ready to work with Denizon?