Optimizing Database Warehousing Operations

The biggest challenge or bottleneck across date warehousing operations is the speed at which the database engine can read from and write to disk. This is commonly known as disk I/O bottleneck.

Most efficient analytics platforms minimize data movement, and are able to process streaming data from disk to memory in parallel, and on a massive scale. In the case of IBM's Netezza, this is accomplished by using innovative hardware acceleration. It uses Field Programmable Gate Arrays (FPGA) to filter extraneous data as early in the stream and as fast as the data can be streamed off disk.

By eliminating data that is not required close to the data source, downstream components like CPU, memory and network do not have to deal superfluous data, significantly reducing I/O bottlenecks and improving system performance.