Are Master Data Management and Hadoop a Good Match?

Master Data is the critical electronic information about the company we cannot afford to lose. Accordingly, we should sanitise it, look after it, and store it safely in several separate places that are independent of each other. The advent of Big Data introduced the current era of huge repositories ?in the clouds?. They are not, of course but at least they are remote. This short article includes a discussion about Hadoop, and whether this is a good platform to back up your Master Data.

About Hadoop

Hadoop is an open-source Apache software framework built on the assumption that hardware failure is so common that backups are unavoidable. It comprises a storage area and a management part that distributes the data to smaller nodes where it processes faster and more efficiently. Prominent users include Yahoo! and Facebook. In fact more than half Fortune 50 companies were using Hadoop in 2013.

Hadoop – initially launched in December 2011 ? has survived its baptism of fire and became a respected, reliable option. But is this something the average business owner can tackle on their own? Bear in mind that open source software generally comes with little implementation support from the vendor.

The Hadoop Strong Suite

  • Free to download, use and contribute to
  • Everything you need ?in the box? to get started
  • Distributed across multiple fire-walled computers
  • Fast processing of data held in efficient cluster nodes
  • Massive scaleable storage you are unlikely to run out of

Practical Constraints

There is more to Hadoop than writing to WordPress. The most straightforward solutions are uploading using Java commands, obtaining an interface mechanism, or using third party vendor connectors such as ACCESS or SAS. The system does not replace the need for IT support, although it is cheap and exceptionally powerful.

The Not-Free Safer Option

Smaller companies without in-depth in-house support are wise to engage with a technical intermediary. There are companies providing commercial implementations followed by support. Microsoft, Amazon and Google among others all have commercial versions in their catalogues, and support teams at the end of the line.

Check our similar posts

User-Friendly RASCI Accountability Matrices

Right now, you’re probably thinking that’s a statement of opposites. Something dreamed up by a consultant to impress, or just to fill a blog page. But wait. What if I taught you to create order in procedural chaos in five minutes flat? ?Would you be interested then?

The first step is to create a story line ?

Let’s imagine five friends decide to row a boat across a river to an island. Mary is in charge and responsible for steering in the right direction. John on the other hand is going to do the rowing, while Sue who once watched a rowing competition will be on hand to give advice. James will sit up front so he can tell Mary when they have arrived. Finally Kevin is going to have a snooze but wants James to wake him up just before they reach the island.

That’s kind of hard to follow, isn’t it ?

Let’s see if we can make some sense of it with a basic RASCI diagram ?

Responsibility Matrix: Rowing to the Island
Activity Responsible Accountable Supportive Consulted Informed
Person John Mary Sue James Kevin
Role Oarsman Captain Consultant Navigator Sleeper

?

Now let’s add a simple timeline ?

Responsibility Matrix: Rowing to the Island
? Sue John Mary James Kevin
Gives Direction ? ? A ? ?
Rows the Boat ? R ? ? ?
Provides Advice S ? ? ? ?
Announces Arrival ? ? A C ?
Surfaces From Sleep ? ? ? C I
Ties Boat to Tree ? ? A ? ?

?

Things are more complicated in reality ?

Quite correct. Although if I had jumped in at the detail end I might have lost you. Here?s a more serious example.

rasci

?

There?s absolutely no necessity for you so examine the diagram in any detail, other to note the method is even more valuable in large, corporate environments. This one is actually a RACI diagram because there are no supportive roles (which is the way the system was originally configured).

Other varieties you may come across include PACSI (perform, accountable, control, suggest, inform), and RACI-VS that adds verifier and signatory to the original mix. There are several more you can look at Wikipedia if you like.

Becoming Nimble the Agile Project Management Way

In dictionary terms, ?agile? means ?able to move quickly and easily?. In project management terms, the definition is ?project management characterized by division of tasks into short work phases called ?sprints?, with frequent reassessments and adaptation of plans?. This technique is popular in software development but is also useful when rolling out other projects.

Managing the Seven Agile Development Phases

  • Stage 1: Vision. Define the software product in terms of how it will support the company vision and strategy, and what value it will provide the user. Customer satisfaction is of paramount value including accommodating user requirement changes.
  • Stage 2: Product Roadmap. Appoint a product owner responsible for liaising with the customer, business stakeholders and the development team. Task the owner with writing a high-level product description, creating a loose time frame and estimating effort for each phase.
  • Stage 3: Release Plan. Agile always looks ahead towards the benefits that will flow. Once agreed, the Product Road-map becomes the target deadline for delivery. With Vision, Road Map and Release Plan in place the next stage is to divide the project into manageable chunks, which may be parallel or serial.
  • Stage 4: Sprint Plans. Manage each of these phases as individual ?sprints?, with emphasis on speed and meeting targets. Before the development team starts working, make sure it agrees a common goal, identifies requirements and lists the tasks it will perform.
  • Stage 5: Daily Meetings. Meet with the development team each morning for a 15-minute review. Discuss what happened yesterday, identify and celebrate progress, and find a way to resolve or work around roadblocks. The goal is to get to alpha phase quickly. Nice-to-haves can be part of subsequent upgrades.
  • Stage 6: Sprint Review. When the phase of the project is complete, facilitate a sprint review with the team to confirm this. Invite the customer, business stakeholders and development team to a presentation where you demonstrate the project/ project phase that is implemented.
  • Stage 7: Sprint Retrospective. Call the team together again (the next day if possible) for a project review to discuss lessons learned. Focus on achievements and how to do even better next time. Document and implement process changes.

The Seven Agile Development Phases ? Conclusions and Thoughts

The Agile method is an excellent way of motivating project teams, achieving goals and building result-based communities. It is however, not a static system. The product owner must conduct regular, separate reviews with the customer too.

Systems Integration as a means to cost reduction

System integration in an organisation refers to a process whereby two or more separate systems are brought together for the purpose of pooling the value in the separate systems into one main system. A key component of process consolidation within any organisation is the utilisation of IT as a means to achieve this end. As such, system integration as a means to cost reduction offers organisations the opportunity to adopt and implement lean principles with the attendant benefits. The implementation of lean techniques requires an adherence to stated methods to facilitate the elimination of wastage in the production of goods and services. In summary, the lean philosophy seeks to optimise the speed of good and service production, through the elimination of waste.

While analysing some of the traditional sources of waste in organisational activities, things like overproduction, inventory, underutilised ideas, transmission of information and ideas, transportation of people and material, time wastage and over-processing stand out. The fact is that companies can eliminate a significant portion of waste through the utilisation of IT to consolidate processes within their organisation.

Adopting lean principles calls for the identification of all of the steps in the company value stream for each product family for the purpose of the eliminating the steps that do not create any value. In other words, this step calls for the elimination of redundant steps in the process flow. This is exactly what the utilisation of IT to consolidate processes offers a company. For instance, the adoption of a central cloud system across a large organisation with several facilities could increase efficiencies in that company. Such a company would drastically reduce the redundancies that used to exist in the different facilities, eliminate the instances of hardware and software purchase, maintenance and upgrade, modernise quality assurances processes and identify further opportunities for improvement.

Perhaps, from the company’s point of view, and from the perspective of lean process implementation, the most important factor is?the effect it has?on the bottom line.’reducing the number of hardware, eliminating the need for maintaining and upgrading hardware, removing the necessity for software purchase and upgrade across facilities also contributes to a significant reduction in operational costs.?This reduction in the cost of operations leads to a corresponding increase in the profit margin of the company.

Applying system integration as a means to cost reduction can also lead to the reduction in the number of people needed to operate the previous systems that have been integrated into one primary unit. Usually, companies must hire people with specialised knowledge to operate and maintain the various systems. Such employees must also receive special training and frequent ongoing education to constantly stay informed of the latest trends in process management. With the integration of the system, the number of people needed to maintain the central system will be significantly reduced, also improving the security of information and other company trade secrets.

Based on an analysis of the specific needs that exist in a particular company environment, a system integration method that is peculiar to the needs of that organisation will be worked out. Some companies may find it more cost-effective to use the services of independent cloud service providers. Others with more resources and facilities may decide to set up their own cloud service systems. Often, private cloud service system capabilities far exceed the requirements of the initiating company, meaning that they could decide to “sell” the extra “space” on their cloud network to other interested parties.

A company that fully applies the lean principles towards the integration of its systems will be able to take on additional tasks as a result of the system consolidation. This leads to an increase in performance, and more efficiency due to the seamless syncing of information in a timely and uniform manner.

Companies have to combine a top-down and a bottom-up approach towards their system integration methods. A top-down approach simply utilises the overall system structure that is already in place as a starting point, or as a foundation. The bottom-up approach seeks to design new systems for integration into the system. Other methods of system integration include the vertical, star and horizontal integration methods. In the horizontal method, a specified subsystem is used as an interface for communication between other subsystems. For the star system integration method, the subsystems are connected to the system in a manner that resembles the depiction of a star; hence, the name. Vertical integration refers to the method of the integration of subsystems based on an analysis of their functionality.

The key to successful system integration for the purpose of cost reduction is to take a manual approach towards identifying the various applicable lean principles, with respect to the system integration process. For instance, when value has been specified, it becomes easier to identify value streams. The other process of removing unnecessary or redundant steps will be easier to follow when the whole project is viewed from the whole, rather than’the part. Creating an integrated system needs some?patience?in order to work out kinks and achieve the desired perfect value that creates no waste.

Contact Us

  • (+353)(0)1-443-3807 – IRL
  • (+44)(0)20-7193-9751 – UK

Ready to work with Denizon?