Best Practices For BI/DW Project Planning

BI Application & Data Warehouse Project Planning should be done perfectly from all aspects: technically, resource-wise and from a business perspective. It is key to a project’s successful execution and completion.

So here is what you should do.

1. For Data Sourcing

Ensure you have all the data that is needed to compile measurements and to answer the driving business questions. For that matter, focus on these 4 things:

1. Identify data sources in the early planning stages so that all the data you might need, is readily available.
2. Develop an understanding of how you will integrate the different data sources from, let’s say, multiple internal systems and external repositories.

iii. Ensure timeliness and consider all the cost-related constraints. You should consider how frequently you can collect data from the aforementioned sources. Also, consider the cost of refreshing your data from external data sources. In case you are using social media as a data source, more frequent updates may be required.

1. If cost becomes a limiting factor, as would be the case with social media data, you can consider sampling smaller data sets more frequently rather than collecting large amounts of data. Larger data sets may give you more accurate social media measures; however, well designed sampling techniques can still give useful information.

2. For BI/Data Warehouse Resources

You need a team with the capability to handle cross-organizational interactions and a deep understanding of the business issues for strengthening relationships with business users. Moreover, they should know the best ways to work with others and communicate more effectively both in writing and in business meetings and presentations. The challenges become even more pronounced as the DW/BI system evolves into a standard component of the IT environment, thus weakening the connection to the business.

3. When Dealing With Big Data

Historically, Business Intelligence (BI) was practiced using established practices and tools, including dimensional modeling, extraction transformation and load (ETL), ad hoc reporting and dashboards. These techniques required that a data warehouse or at least a data mart is able to support management reporting that was not generally available from transaction processing systems.

Today, we have to tap into a broader array of information captured in big data. Here are some essential characteristics to keep in mind as you develop a strategy for big data analysis.

Big data does not completely come from sales, inventory or human resources systems. It is generally a varied mixture of data sourced from application log files, machine sensor data, and social media.

1. Descriptive statistics. It is especially useful with big data sets when you partition the data and compare different groups. Most BI practitioners are familiar with this technique wherein calculations such as mean, median, and standard deviation are commonly used to describe a population. So when you get the data from Application log, you can find the average time spent in your Web application prior to the sale and not just calculate the average dollar value of a sales transaction from a sales data

2. Grouping your customers. Now normally you would do that by sales region but that is not the most informative way to organize and analyze your data. In lieu, you should think of clustering. This machine learning technique is used to identify subsets of data with similar characteristics. For example, clustering might help you identify different groups of customers based on the time spent in your Web application and the amount of their sales transaction.

3. Last you can explore big data sets and testing hypotheses to complement the types of analysis you do with your traditional BI systems.

Once you have identified a group of customers, you can also analyze their navigation patterns on the site.  Since they spend a significant amount of time on your site, you can reasonably hypothesize whether they are interested in making a purchase or not.

4. Dealing with Big Data Types

• For large volumes, deploy enough commodity servers and storage along with a distributed file system, like the Hadoop File System (HDFS), and you can collect petabytes of data.
• If you do not need to store the data for too long, you can take advantage of public clouds like Amazon AWS, Microsoft Azure and Rackspace, but watch out for long term storage costs.
• If you have rapidly changing data consider using a real time big data analytics tool like Storm.

Traditional BI is not dead yet and will not be anytime soon. It is here to stay as long as the business fundamentals are the same.  So the management reporting based on data from transaction processing systems will remain useful. However, big data shall introduce new ways of understanding business operations that complement existing management reporting systems.

 

Big Data Technology Trends in 2015

Until 2013, it was a topic of debate whether “Big Data Technology” was a niche technology best suited for Internet companies or had the potential to go mainstream.  Things began to change and just in a span of 2 years, industry arrived at a consensus – a clear one. So in 2015, we will see the impact of big data across almost every industry sector. Here’re big data technologies trends for the year 2015:

Big data analytic shall address all security concerns

While earlier it was possible, now corporations collect vast quantities of data that is getting increasingly tedious to handle. This data is so highly complex and voluminous that it has quickly outgrown the capabilities of traditional security software. Not just its managing is difficult, but also it is nearly impossible to analyze it, given the fact that most security platforms used currently don’t talk to each other. Moreover, the processing huge volumes and varieties of data is both cost and time-consuming.

In such times, CISOs have started to look at big data analytics as a powerful complementary solution to traditional security software. It offers some clearly distinct advantages as it enables the security infrastructure to identify and fix “low and slow” advanced-persistent-threat outbreaks that might otherwise be indiscernible when protected by security solutions that don’t connect to or relate with each other. This phenomenon is sure to catch up quickly and expand to other sectors.

Fake technologies will no longer be able to sustain

In 2015, buyers are sure to come to a clear understanding that even when these come from experienced vendors, not everything they try to sell in the name of big data technology could help them. For their own good and that of enterprises, they have to choose solutions that are distinct from traditional business intelligence and data warehouse technologies so that it handles (effectively) the volume or variety of data they have to manage. 2015 shall be the year of technology replacement wherein big data analytics will be emerging as a fundamentally powerful next-generation analytics and the preferred choice of buyers.

Big data shall help IoT reach its full potential

There is a huge amount of data i.e. generated by the household and industrial devices, and wearable et al. and it comprise today’s Internet of things (IoT). As yet, hundreds of device makers building the IoT have not come together. So this data, despite of growing exponentially, is lying unused with the device makers.

However, if big data technology is put to use, it can bring this data together so that its true value is realized. It will also involve understanding how devices and services are working, communication flow that could constitute security threats, and intertwining interdependent & dependent failure modes.

This is the first vital first step that should be followed by a variety of analytical approaches. In 2015, device makers will have to turn to big data analytics to help the emerging Internet of things market reach its full potential— or they will be left out if they don’t.

Companies, both big & small, will be moving towards big data analytics

A few years ago, big data tools were available only to corporate biggies that could invest in proprietary and largely inflexible infrastructure. From 2015, companies of all sizes will increasingly adopt it to identify and quickly adapt to opportunities and challenges, if they want to remain competitive, as they rethink the way their employees collaborate and use data. It is expected as companies will be increasingly adopting a data centric approach to set strategy, encourage collaboration among colleagues, and interact with customers.

Big Data vendor consolidation will happen
In 2015, some of these small players in the area of big data, with their niche products, will not be able to meet their customers’ overall needs. So it is expected that many startups will either fail or become acquisition targets as larger vendors who could offer end-to-end solutions would eye them.

Conclusion

These are a number of proof points that go beyond elusive claims to prove that in 2015 the full potential of big data analytics will be unleashed empowering business analysts with new capabilities. So this year shall mark the end of testing and early-stage roll outs and the rise of a ripe big data landscape where we’ll see enterprises making bigger strategic bets and asking for end-to-end solutions built to handle humongous amounts of incongruent data without calling for data scientists. It’s shakeout time.