Part I of this blog series introduced the concept of complex event processing (CEP) and possible needs for CEP software applications. One such broad CEP platform, Progress Apama, has been offered by Progress Software Coporation after acquiring the formerly independent Apama LTD in 2005. It is worth analyzing what has happened with the Apama product since being acquired by Progress Software.
Progress Apama’s Current State of Affairs
For one, Apama revenues have increased multiple times (e.g., 70 percent growth in fiscal 2008). This growth probably makes Progress the market share leader in the CEP market, together with the fact that Progress Apama currently has approximately 110 customer deployments.
A sampling of Apama customers, those that Progress has publicly announced, can be found at the company’s website here. The product’s “sweet spot” and leading presence to date has been in the Capital Markets and Financial Services segments for high frequency trading applications. Apama’s customer profiles range from the largest sell-side firms in the world to smaller, boutique buy-side firms.
Furthermore, Progress Apama now has a worldwide footprint, with deployments in North and South America, throughout Europe, the Middle East, Australia, Japan, and Korea. The product now logically supports some internationalization capabilities. Last but not least (and to be detailed later) Apama has expanded in capability with more CEP functionality, sophisticated development tools, flexible event capture and replay, visual dashboards, and open integration framework, while emphasizing less on just promoting the product’s high performance and scalability (e.g., sub-millisecond latency for thousands of scenarios) traits.
Progress Apama customers are generally distinguished by a desire to leverage the product’s rapid application development (RAD) tools to build unique trading or trading-related applications that allow them to incorporate their own business logic, rather than packaged, off-the-shelf applications. While the company’s early market adoption has been in capital markets, Progress Apama has also established penetration outside financial sectors in other event-driven environments.
Other sectors are telecommunications, supply chain/logistics, energy grid monitoring, manufacturing process monitoring, retail banking fraud detection, entertainment (i.e., gaming surveillance), and other areas. For example, when it comes to telecommunication providers’ revenue assurance, a CEP platform could monitor the billing of several million subscribers across multimedia channels (i.e., voice/video, data, content, and unlimited multimedia messaging service [UMMS]), to prevent revenue loss in real-time.
Real-life and Prospective CEP Deployments Outside Capital Markets
At Progress Software’s Analyst Summit 2009, there was a case study presentation about Apama empowering advanced international logistics in terms of shipping and ports management. The customer is Royal Dirkzwager, which since its founding in 1872 has developed into the maritime information and service provider for Northwest Europe, with a strong focus on the Port of Rotterdam.
Dirkzwager deals with vast information on vessels’ characteristics, ship’s position reports, and ever-varying estimated times of arrival (ETAs) and actual times of arrival (ATAs). From about 200 position reports per second 10 years ago or so, today the company has to deal with over 1,000 position reports per second. Dirkzwager’s public sector customers are the related port authorities, port state control, customs, seaport police, and coast guard. Private sector customers are ship owners and agents, terminals, and service providers (pilots, tugs, maintenance crews, etc.).
One of the business issues for the company has been to integrate berth planning for terminals and employee planning for authorities and service providers into its customers’ business processes. Another issue for Dirkzwager has been the globalization of customers and its geographical coverage expansion.
Namely, from customarily focusing on Rotterdam, the company intends to focus on northwest Europe and, albeit to a much lesser extent, on a worldwide coverage. Last but not least, harnessing electronic position information tools such as AIS (Automatic Identification System) and LRIT (Long Range Identification & Tracking) has become much more important.
Dirkzwager reportedly implemented Apama due to its capability to handle different position report types and to handle large amounts of position reports (i.e., scalability). Also, the company’s employees can now create and modify business rules for the port’s operation. As for future directions, the port operator company expects to enable customers to create own business rules, and to also be able to process port related messages such as route advice and monitoring. The latter capability should result in a reduced overall fuel consumption and improved port arrival planning.
Monitoring Manufacturing Processes, Catching Crooks and Other Bad Guys…
Given Progress’ traditional approach to leverage partners to embed and sell its products, we should note that to this point Apama sales have been based mostly on a direct sales model. To date, sales have been primarily to banks and other larger companies in financial services, where CEP is deemed as a “bet the farm” solution. Sales into governments, telecommunications, utilities, transportation enterprises and so on will also likely target major enterprises. These operations will require a continued expansion of a direct sales channel as well as partner-based channels for smaller organizations.
One good example of a partner embedding Apama would be Manuvis within its FactoryMRI manufacturing execution system (MES). Manuvis’ system continuously monitors production equipment and other key production statistics in a discrete manufacturing environment (e.g., production of auto parts). Should the software detect a continuing anomaly in a machine, it will notify workers to tend to the machine. The software will also dynamically re-direct production and re-optimize the production schedule.
Generally speaking, CEP tools can link directly into data collection and automation systems sending signals from the production floor, as well as into packaging, warehousing management systems (WMS), and other related business systems to guide problem resolution and improvement. Another example of Apama’s use in the production environment is to detect bottle-filling trending low or high in a large high-speed bottling plant.
By defining threshold and time window limits, the system provides alarms and dashboard visibility on the fly, as well as comparisons over any period in history. Data input can be sensor outputs, control instructions, transactions and so forth, while connection taps can be made into programmable logic controller (PLC) and control system data streams, as necessary.
Progress has been working with several international regulators, including the UK’s Financial Services Authority (FSA, which is the counterpart of the US Securities & Exchange Commission [SEC]), to incorporate real-time fraud detection technologies into its market monitoring endeavors to help detect fraud. One of the drivers for FSA’s SABRE II (Surveillance and Automated Business Reporting Engine) fraud-detecting initiative was for the regulatory authority to become more dynamic and proactive in detecting market abuse. This proactive approach would be achieved by investigating potential offenders more quickly with relevant evidence, and by identifying trading rings and links between individuals generating illicit market impressions.
The idea was to also instantly detect and prosecute “extremely lucky” individuals (e.g., traders who constantly take a best offer in a market to drive up a share price) and monitor for price/volume movements where companies may need to make a disclosure of price sensitive information. The other drive was to implement all requirements for the Markets in Financial Instruments Directive (MiFID) in terms of transactions reports, policy calculations, and inter-regulatory reporting.
There are hundreds of known and possible illegal trading patterns (tricks), but traditional algorithmic techniques have not been able to detect their use in real time. Thus, Progress Apama is used to promptly detect insider trading, breaches of short-selling rules, wash trading, the spreading of illegal rumors followed by suspicious buying patterns, “painting the tape” to drive a stock’s price up, front-running of orders, and trader collusion (with insider knowledge and “agendas” to buy across different trading venues), as well as many other common market abuses.
The point here is to detect fraud while it is occurring (not after the fact), so that regulators can detect market manipulation that is in breach of regulations in a timely manner. Hence, before a trade is placed, real-time rules should detect both illicit doings and honest mistakes (like so-called “fat-finger” errors or decimal points in the wrong places), apply real-time compliance rules, or make sure the firm is not over a certain percentile of an actively traded market.
Given the need for in-depth know-how and domain expertise, Progress has teamed up with specialist consultancy Detica Group PLC (now part of BAE Systems) to encode market knowledge into algorithms that detect illegal trading patterns. Other possible examples of Apama deployments could be: the detection and prevention of credit card fraud (e.g., detecting several same-number credit card transactions at physically distant retail stores in an atypical time bracket); analyzing patterns in passenger movements (e.g., to detect potential terrorists); aviation control; and predicting the best route for vehicles such as tanks and long-haul transport.
More on Apama’s Life Under Progress Software
Beyond the original acquisition of Apama by Progress Software, there have been no further CEP products acquisitions. As part of Progress Software, however, Apama has expanded its product capabilities with enhanced visual dashboard technology, which involves the RTView technology that is provided via an original equipment manufacturer (OEM) relationship with SL.
Apama has also meanwhile gained backtesting analysis capabilities, by leveraging integration of other technology from Progress–ObjectStore. This object-oriented database is used in the Apama product as Apama EventStore, a time series database that can capture event streams (such as market data and trade execution calculations/decisions), making that data available for replay. This “TIVo”-like effect of a sort uses an application within Apama called Research Studio.
For example, as the radio frequency identification (RFID) system continuously reads each tag, Boekhandels Groep Nederland’s (BGN) Selexyz retail bookstores use Progress Apama to filter out duplicate reads from incoming streams and ensure that every book is counted just once. Duplication is also prevented by reconciling the advanced shipping notice (ASN) data streams coming from the distributor.
The third and final part of this blog series will conclude with Progress Apama’s differentiating traits, future roadmap, and competitive landscape. The post will also discuss what more CEP is capable of and what it is not capable of.
Progress Apama’s Current State of Affairs
For one, Apama revenues have increased multiple times (e.g., 70 percent growth in fiscal 2008). This growth probably makes Progress the market share leader in the CEP market, together with the fact that Progress Apama currently has approximately 110 customer deployments.
A sampling of Apama customers, those that Progress has publicly announced, can be found at the company’s website here. The product’s “sweet spot” and leading presence to date has been in the Capital Markets and Financial Services segments for high frequency trading applications. Apama’s customer profiles range from the largest sell-side firms in the world to smaller, boutique buy-side firms.
Furthermore, Progress Apama now has a worldwide footprint, with deployments in North and South America, throughout Europe, the Middle East, Australia, Japan, and Korea. The product now logically supports some internationalization capabilities. Last but not least (and to be detailed later) Apama has expanded in capability with more CEP functionality, sophisticated development tools, flexible event capture and replay, visual dashboards, and open integration framework, while emphasizing less on just promoting the product’s high performance and scalability (e.g., sub-millisecond latency for thousands of scenarios) traits.
Progress Apama customers are generally distinguished by a desire to leverage the product’s rapid application development (RAD) tools to build unique trading or trading-related applications that allow them to incorporate their own business logic, rather than packaged, off-the-shelf applications. While the company’s early market adoption has been in capital markets, Progress Apama has also established penetration outside financial sectors in other event-driven environments.
Other sectors are telecommunications, supply chain/logistics, energy grid monitoring, manufacturing process monitoring, retail banking fraud detection, entertainment (i.e., gaming surveillance), and other areas. For example, when it comes to telecommunication providers’ revenue assurance, a CEP platform could monitor the billing of several million subscribers across multimedia channels (i.e., voice/video, data, content, and unlimited multimedia messaging service [UMMS]), to prevent revenue loss in real-time.
Real-life and Prospective CEP Deployments Outside Capital Markets
At Progress Software’s Analyst Summit 2009, there was a case study presentation about Apama empowering advanced international logistics in terms of shipping and ports management. The customer is Royal Dirkzwager, which since its founding in 1872 has developed into the maritime information and service provider for Northwest Europe, with a strong focus on the Port of Rotterdam.
Dirkzwager deals with vast information on vessels’ characteristics, ship’s position reports, and ever-varying estimated times of arrival (ETAs) and actual times of arrival (ATAs). From about 200 position reports per second 10 years ago or so, today the company has to deal with over 1,000 position reports per second. Dirkzwager’s public sector customers are the related port authorities, port state control, customs, seaport police, and coast guard. Private sector customers are ship owners and agents, terminals, and service providers (pilots, tugs, maintenance crews, etc.).
One of the business issues for the company has been to integrate berth planning for terminals and employee planning for authorities and service providers into its customers’ business processes. Another issue for Dirkzwager has been the globalization of customers and its geographical coverage expansion.
Namely, from customarily focusing on Rotterdam, the company intends to focus on northwest Europe and, albeit to a much lesser extent, on a worldwide coverage. Last but not least, harnessing electronic position information tools such as AIS (Automatic Identification System) and LRIT (Long Range Identification & Tracking) has become much more important.
Dirkzwager reportedly implemented Apama due to its capability to handle different position report types and to handle large amounts of position reports (i.e., scalability). Also, the company’s employees can now create and modify business rules for the port’s operation. As for future directions, the port operator company expects to enable customers to create own business rules, and to also be able to process port related messages such as route advice and monitoring. The latter capability should result in a reduced overall fuel consumption and improved port arrival planning.
Monitoring Manufacturing Processes, Catching Crooks and Other Bad Guys…
Given Progress’ traditional approach to leverage partners to embed and sell its products, we should note that to this point Apama sales have been based mostly on a direct sales model. To date, sales have been primarily to banks and other larger companies in financial services, where CEP is deemed as a “bet the farm” solution. Sales into governments, telecommunications, utilities, transportation enterprises and so on will also likely target major enterprises. These operations will require a continued expansion of a direct sales channel as well as partner-based channels for smaller organizations.
One good example of a partner embedding Apama would be Manuvis within its FactoryMRI manufacturing execution system (MES). Manuvis’ system continuously monitors production equipment and other key production statistics in a discrete manufacturing environment (e.g., production of auto parts). Should the software detect a continuing anomaly in a machine, it will notify workers to tend to the machine. The software will also dynamically re-direct production and re-optimize the production schedule.
Generally speaking, CEP tools can link directly into data collection and automation systems sending signals from the production floor, as well as into packaging, warehousing management systems (WMS), and other related business systems to guide problem resolution and improvement. Another example of Apama’s use in the production environment is to detect bottle-filling trending low or high in a large high-speed bottling plant.
By defining threshold and time window limits, the system provides alarms and dashboard visibility on the fly, as well as comparisons over any period in history. Data input can be sensor outputs, control instructions, transactions and so forth, while connection taps can be made into programmable logic controller (PLC) and control system data streams, as necessary.
Progress has been working with several international regulators, including the UK’s Financial Services Authority (FSA, which is the counterpart of the US Securities & Exchange Commission [SEC]), to incorporate real-time fraud detection technologies into its market monitoring endeavors to help detect fraud. One of the drivers for FSA’s SABRE II (Surveillance and Automated Business Reporting Engine) fraud-detecting initiative was for the regulatory authority to become more dynamic and proactive in detecting market abuse. This proactive approach would be achieved by investigating potential offenders more quickly with relevant evidence, and by identifying trading rings and links between individuals generating illicit market impressions.
The idea was to also instantly detect and prosecute “extremely lucky” individuals (e.g., traders who constantly take a best offer in a market to drive up a share price) and monitor for price/volume movements where companies may need to make a disclosure of price sensitive information. The other drive was to implement all requirements for the Markets in Financial Instruments Directive (MiFID) in terms of transactions reports, policy calculations, and inter-regulatory reporting.
There are hundreds of known and possible illegal trading patterns (tricks), but traditional algorithmic techniques have not been able to detect their use in real time. Thus, Progress Apama is used to promptly detect insider trading, breaches of short-selling rules, wash trading, the spreading of illegal rumors followed by suspicious buying patterns, “painting the tape” to drive a stock’s price up, front-running of orders, and trader collusion (with insider knowledge and “agendas” to buy across different trading venues), as well as many other common market abuses.
The point here is to detect fraud while it is occurring (not after the fact), so that regulators can detect market manipulation that is in breach of regulations in a timely manner. Hence, before a trade is placed, real-time rules should detect both illicit doings and honest mistakes (like so-called “fat-finger” errors or decimal points in the wrong places), apply real-time compliance rules, or make sure the firm is not over a certain percentile of an actively traded market.
Given the need for in-depth know-how and domain expertise, Progress has teamed up with specialist consultancy Detica Group PLC (now part of BAE Systems) to encode market knowledge into algorithms that detect illegal trading patterns. Other possible examples of Apama deployments could be: the detection and prevention of credit card fraud (e.g., detecting several same-number credit card transactions at physically distant retail stores in an atypical time bracket); analyzing patterns in passenger movements (e.g., to detect potential terrorists); aviation control; and predicting the best route for vehicles such as tanks and long-haul transport.
More on Apama’s Life Under Progress Software
Beyond the original acquisition of Apama by Progress Software, there have been no further CEP products acquisitions. As part of Progress Software, however, Apama has expanded its product capabilities with enhanced visual dashboard technology, which involves the RTView technology that is provided via an original equipment manufacturer (OEM) relationship with SL.
Apama has also meanwhile gained backtesting analysis capabilities, by leveraging integration of other technology from Progress–ObjectStore. This object-oriented database is used in the Apama product as Apama EventStore, a time series database that can capture event streams (such as market data and trade execution calculations/decisions), making that data available for replay. This “TIVo”-like effect of a sort uses an application within Apama called Research Studio.
For example, as the radio frequency identification (RFID) system continuously reads each tag, Boekhandels Groep Nederland’s (BGN) Selexyz retail bookstores use Progress Apama to filter out duplicate reads from incoming streams and ensure that every book is counted just once. Duplication is also prevented by reconciling the advanced shipping notice (ASN) data streams coming from the distributor.
The third and final part of this blog series will conclude with Progress Apama’s differentiating traits, future roadmap, and competitive landscape. The post will also discuss what more CEP is capable of and what it is not capable of.
