ETL Process Further........

Market Definition

The extract / transform / load (ETL) middleware market comprises vendors that offer tools used in designing and populating analytic databases such as data warehouses, data marts, and operational data stores. Specifically, these tools provide for the periodic extraction, transformation, and integration of data from any number of heterogeneous data sources (frequently transaction databases) into time-variant databases used predominantly for query and reporting (i.e., business intelligence). ETL tools provide developers with an interface for designing source-to-target mappings, transformation, and job control parameters.

Market Forecast

The ETL adoption rate will expand 10%-20% annually during the next three to five years, with the services component (not addressed in this evaluation) growing at a commensurate pace. However, due to price pressure from DBMS vendors offering increasingly competitive offerings at bundled prices, the financial growth of this market will be in the 5%-15% range, annually. ETL market growth will resume, albeit modestly, in 2004, fueled by an aggressive uptake in customer data integration (CDI) efforts and consolidation of departmental data warehouse efforts into enterprise initiatives. Consolidation within the ETL market will not occur during the next few years. Instead, second-tier ETL purveyors will continue to be forced to consider joining forces with business intelligence (BI) vendors. Top-tier ETL vendors, on the other hand, will either absorb or build out enterprise application integration technology along with other emerging forms of data integration (e.g., demand-based enterprise information integration). New entrants (even those from the past two years) will find the market too crowded, noisy, and mature to

compete.

Key Findings

With continued frequency in 2004, ETL buyers are demonstrating a preference for market leaders. Fearing market consolidation/atrophy, shaky services/support, and DBMS/OS version lag, enterprises are willing to make architectural concessions to embrace a market leader. Indeed, price has entered the selection equation more so than in 2002/03, but significant elasticity remains for most Global 2000 enterprises. Nonetheless, the predominant gating factors remain support for the direct extract of an enterprise’s range of data sources

(increasingly including commercial business applications) and target environment support.

Therefore, market leaders will continue to be those with the strongest presence-oriented strategies (e.g., boisterous marketing, exclusive partnering) combined with the broadest array of source/target platform support. That said, there is still room for new market leaders to emerge — especially those that take serious innovative risks while offering the requisite technology stack and generate forceful messaging. Essential criteria for evaluating and selecting ETL vendors include the following: 

• Platform support: Support for an enterprise’s existing sources, targets, and execution environments should be paramount. Increasingly, support for non-DBMS sources (e.g., Web services, message queues, log files) and commercial business application schemas are critical concerns. 

• Transformations: Developers require both a broad palette of selectable data transformations and flexibility in developing and incorporating custom business logic. Built-in data quality functions are the latest differentiator.

Market Summary

• Data management utilities: Having to go outside the tool for high-performance sorting, job scheduling, and data transport can be a nuisance and maintenance headache.

• Performance characteristics: As more data is integrated faster while batch windows are shrinking, ETL tools often must be able to execute job streams/steps in parallel.

• Developer environment features: GUI features and flexibility, multideveloper capabilities, code debugging, application versioning, and star-schema helpers are some of the developer niceties to consider.

• Metadata: Support for a range of metadata sources and interchange standards (e.g., CWM/XMI), metadata browsing/reporting, and repository extensibility can be significant differentiators. 

• Viability: Even near-term metadata standards do not provide for the porting of ETL applications; therefore, enterprises building out data plumbing should be concerned with long-term support.