Intelligent Data Mining

Sponsored Links

Data mining has had a checkered history mainly due to technical constrains placed by limitations of software design and architecture. Most of the algorithms used in data mining are mature and have been around for over twenty years. The next challenges in data mining are not algorithmic but software design methodologies. Commonly used data mining algorithms are freely available and processes that optimize data mining computing speed are well documented.

Most early data mining software were spun off from academia and were built around an algorithm. The inability of early data mining software to integrate to external data sources and usability issues resulted in data mining being marginalized.

The cost associated with data mining is still unnecessarily high and often not cost effective. New standards in data extraction and better software platforms holds promise that the threshold barrier to entry will be reduced.
Data access standards such as OLE-DB, XML for Analysis and JSR will minimize the challenges for data access. Building a user friendly software interfaces for the end-user are the next steps in the evolution of data mining. A comparable analogy can be made with the increasing ease of use of OLAP client tools.
The J2EE and .NET software platforms offer a large spectrum of built-in APIs that enable smarter software applications.

DAT-A : Open Source Data Mining and OLAP on MySQL

DAT-A is an open source application that is built to allow intelligent data mining. By intelligent data mining, DAT-A's software architects are creating a highly decouple application that focuses the user's attention on the data mining results and not the data extraction or data modeling process. All data exchanges are in XML and SOAP to ensure interoperability.

An enterprise version is also being planned that is built on a BEA WebLogic Server that writes to a Web Services interface.
Presently MySQL does not have built-in data mining modules. DAT-A applies a data mining abstraction layer on MySQL. The business logic for controlling the data mining model and model training is written in the J2EE framework.

For the personal edition of DAT-A, the MySQL data mining application server is contained within the business logic developed on the J2EE framework layer. In the upcoming enterprise version, the business logic and data extraction controls will be hosted on BEA's WebLogic application server.

Article Source:
[1]http://www.dwreview.com/Data_mining/Intelligent_DataMining.html

Financial Data Warehouse

Sponsored Links

Oracle launched a new data warehouse built specifically for financial services businesses, a release that one IDC analyst says could be part of a coming wave of industry-specific releases from technology vendors.

Unveiled Thursday, Oracle Financial Services Data Warehouse features tools and integration capabilities for companies in the business of money management. Capabilities included in the warehouse are based on more than a decade of domain and data model experience with top financial companies, Oracle stated in a news release.

Oracle Financial Services Data Warehouse comes with a pre-built data model for simplified ETL, unified infrastructure to run analytics, contextual data quality checks to spot inconsistencies across ledgers and books, and high-volume, cross-functional computations often required for financial regulations and stress tests. The warehouse also leverages Oracle’s Exadata Database for analytical scenarios.

S. Ramakrishnan, Oracle financial services analytical applications group manager, said in a statement on the release that the speed and context of financial business requires a niche warehouse to avoid users from being hampered by “indiscriminate, tedious capture and stewardship” of data.

Henry Morris, analyst with consultant group IDC, says Oracle’s new warehouse is on cue with analyst predictions of an increase in rollouts of data access products and appliances for different areas of business. Morris says that this approach saves time and effort in getting data straight in the analytics area, and that there might be more of these releases coming.

“By providing an integrated data model … Oracle addresses this need in a manner that is industry-specific. This approach would add value to Exadata and increase its attractiveness to buyers in this industry,” says Morris.

Article Source:
Justin Kern,http://www.information-management.com/news/data_warehouse_business_intelligence_analytics_Oracle-10019601-1.html

real time data warehousing

Sponsored Links

Also known as active data warehousing, real time data warehousing is the process of storing and analyzing data across multiple types of storage systems. Companies tend to use this approach in an ongoing effort to maximize the benefits from various forms of business intelligence, especially in terms of positioning the company for growth through sales. By capturing the information that becomes available and the assimilation of data with historical information, it is possible to predict shifts in customer demand, and develop new marketing strategies that will attract new customers.

Basic process real time data warehousing requires transactional data added to the database, such as placing orders or invoicing system, immediately analyzed, classified, and linked to information that has been shed from the previous transaction. Ideally, the additional information will yield additional information that helps to show the trend in purchases of goods or services offered by the company, the generation of profits or losses.

With assimilation and the rate of transactions and other data occurs, companies can move more quickly to take advantage of the trends that have the potential to recover significant business. Because real-time data warehousing process automatically, no need for anyone to turn this trickle of data from various transactional databases to a central database in real time. So, it is possible to access bank information is updated every time, and using data for planning future projects or actions that will be in accordance with corporate interests.

The process of data mining in real time can also include automatic generation of reports tailored to the needs of end users. Design a data warehouse often allows users to select from a series of pre-programmed report formats, or use a tool built into the software package to create a special report that manage data in many ways. This versatile data warehouse architecture makes it easy for companies of various sizes and related to different areas to use the same basic software, but adjusts the use software in accordance with their respective needs.

Most real-time data warehousing package also allows on-demand report generation and also on the schedule you set. This can be very helpful, as they may submit a request for a sudden moment, and the answer in seconds or minutes. For example, if a sales manager presented with a question about the sales day today is along the lines of a particular product, he can only frame the query, has a software pull-up-to-minute data, and reports that provide information with ease. other conventional methods would require up to half an hour or more to manage what real time warehouses can be set in less than three minutes.

Article Source:
http://www.wisegeek.com/what-is-real-time-data-warehousing.htm

data warehouse appliance

Sponsored Links

Usually people think that the data warehouse appliance is all the burden of storing old data, but this is not the full facts. In addition to the archive, also revealed large amounts of data from different sources into a single comprehensive database, so users can examine and manipulate the content as needed. Users will only have one database, making it important for him to access and data feeds from various sources, for information, which requires. Data warehousing tools are tried and tested way of checking information to make a survey of the cluster statistics and other content, so the company can plan the future of intelligence and can know the outcome of the process. the correct use of content can make a big difference with the way in which trade can evolve and be developed, because it allows the production of projections and estimates. This data can be used to create a business module, which helps users to focus on business. In addition, help in cutting costs and making better use of resources that can be accessed.

There is a big difference between data warehousing and business intelligence tools. Business intelligence uses various methods to compare and interpret the information and content, the motive to increase efficiency. Advanced content management software allows to provide useful information for the smooth business. Data warehouse appliance can be referred to as pot, which consists of various kinds of information. With help from the dash board, reports, analysis, and various content can be assimilated and can be used. Furthermore, both the data warehouse appliance as well as cooperation in the intelligence business to provide end users the right information needed. The employees, who are required to use this application, must be well trained in building a database and management. They must be proficient in statistics, to take full advantage of the given application.

By Moon Saud, Article Source:
http://EzineArticles.com/?expert=Moon_Saud

data warehouse best practices

Sponsored Links

Data Warehousing is the innovation of the 90s who promised to change the landscape for good data. How far have we come? Many vendors have entered the market because it makes sense to bring together data from throughout the organization, and this will continue to make sense of the future.

How large a data warehouse market will grow no one knows yet. But it must still growing rapidly, and currently estimated at 4.5 billion dollars per year (IDC).

1. Why Run Into Data Warehouse Project Scope Creep?

To quote Bill Inmon (teacher and author of several books on Data Warehousing) "Traditional project begins with requirements and ends with data Data Warehousing. The project begins with data and end with the requirements." Once this project will take place, users will find new applications, and with it will come new demands for data. Interestingly, these projects are often justified by moving the T & R work away from "the data. What we have seen is that the first thing that happens immediately after the project is that it gives more demand for special requests submitted to this data is the same person''. This may appear to undermine the initial business case but actually signal the beginning of the creation of value from DWH project.

2. Star Schema Entity Relationship Model Versus?

There has been a major debate in society about the benefits of different data models. At the risk of over simplifying: ER models tend to have better performance (processing time) to end users, and is often regarded as "easier" to be understood by end users. Drawbacks is that the ER model requires more disk space, and, because of intrinsic redundancy in the data, have a consistency problem from the perspective of maintenance. Having said this, it seems that the practice is often some combination of the two can not be avoided in practical settings, although the preference (ER or Star) of the chief architects. Overall, the Star model seems to have gained the most ground.

3. Importance of Data Warehouse Business Case

Much has been written about the business case for Data Warehouse. What happened to good business case? IT savings everywhere in the business case DWH. The important thing is to not restrict to 'genuine savings', but to connect to the main business processes as much as possible. For example, more rapid cycle changes to the selection list fine (if the calculated charge per hour), but it's better if the revenue from the acquisition of more customers that follow from this choice can be tied into a relationship not only will revenue growth rather than savings for make a business case that is more balanced, more important is the intrinsic business buy-in results from a direct connection to the company's bottom line. These days, changes in legislation (especially the Sarbanes-Oxley) plays a major role in justifying the business case. This may be either through a higher company valuation to collect information that is transparent, or, lack of sleep the night for the CEO, which of course is priceless ...

4. Why Data Warehouse Project 'Do not' Go Wrong?

Actually, the Data Warehouse projects sometimes fail. But, they fail so rarely, that it is actually very hard to believe ... Especially after talking to so many end-user satisfaction. And there are many ways the project could be one of the Data Warehouse. Delivery on time, data administration problems, and data can not be avoided the issue of quality for food systems. Corporate politics (see Tip 7) may be the best explanation for this phenomenon at close to 100% success rate DWH project. In my experience, the reason why failures or 'semi-fail' can go unnoticed is the senior management either because they do not realize, or, say, "motivated" to talk about misspending company funds. As a result, not enough studied. Maybe we as consultants have a stake in this as well, because it ensures much business the industry is in progress ... J

5. What is different about Data Warehousing Web?

Kimball & Merz (2000): "Although this clickstream data in many cases is raw and real, have the potential to provide unprecedented detail about every move made by every human being by using Web media." Subatomic nature of clickstream data raises unique challenges. There is little built in feedback mechanisms to ensure data quality, compared with other data streams. The relationship between user and server logs record the mouse clicks are not so tight as in "traditional processing" transaction because of technical issues such as proxy servers and caching. Because of these differences, IT people need to adjust to the web process flow, rather than the process of adapting to the needs of IT as an interface common to most other DWH.

6. Should the data contained in the Data Warehouse?

Incoming data DWH ultimately determine its place in the organization. A "let's load all the data, to be" safe attitude is a sure way to derail your DWH project. Options for what should and should not be included needs to be created since the beginning, so that projects are managed. After the proven success of the delivered, deployed, and profitably exploited DWH, there will always be a place to put funds previously neglected interface. Given the anticipated life cycle of the DWH, it makes sense to consciously exclude certain sources. Options such as what data to include the need is driven by business considerations, and in particular reference to the company's bottom line. If you can not show how the data will be used profitably, they stay out! See also tip # 3.

7. Data Warehousing & Company Politics

Data warehouses have an impact on corporate bottom lines. Therefore, they may be candidates for turf battles, and also at risk of being "small changes" in negotiating budget allocations. None of the consideration of the benefits of long-term corporate goals. Managing projects is quite difficult because the DWH, and budget issues should not be made more difficult than it already is. Because DWH investment in current income and is located in the future, even more important to secure funds through sound business case and buy-ins from the appropriate (high) level of management. See also Tip # 3. Access to data means power, and talking about power management is one of the greatest taboos still exist. Sensitive as they are, even more easily discussed the budget ...

8. Trap Data Warehouse Project

Some 'frequently repeated barrier' on-time delivery route data warehouse project:

    * ETL process has eaten so much time (and still need to be "babysitters"), that little if any time left to develop applications that are required to exploit DWH
    * Some of the data needed, but were not not available, or not timely
    * Maintenance required for tuning, indexing, and backup and recovery is very underrated
    * Various ways of calculating the same phenomenon lead to different results, and no one could convince explain the difference (s)
    * Data is loaded (and recombination) which turned out to contain previously unknown inconsistencies in the source system, the 'classical' data quality problems that travel DWH projects
    * Metadata is less, and developer of the amount spent so much time figuring out what the field really means''


9. DWH Hardware and Software Go Hand in Hand

In Data Warehousing, not about the hardware, and not about the software: it is about the perfect integration of a second. Those who start their projects from both ends, will pay dearly for this mistake. The reasons are:

• In terms of price / performance, new, pre-integrated hardware-software combination takes the lead

° of the project management perspective, you do not want to caught between vendors when a proposed solution does not work as expected

· Database tuning and indexing is very important and very complex work, it should be left to specialists (in-house trained)

10. Performance is key

Although I do not often find this technology has become an important factor in the acceptance of the Data Warehouse, there is no other factor will be as important as performance. As size increased from time to time, this factor becomes more important. There are three reasons for this:

   1. performance has a major impact on the development of the velocity (initial load is always very time consuming), and therefore the overall maturity at the time of delivery DWH
   2. performance can make or break the end-user acceptance, particularly the predictability of performance
   3. performance has a tremendous impact on end-user productivity, the main driver of business pay-off

By Tom Breur, Article Source:
http://EzineArticles.com/?expert=Tom_Breur