Streaming System Benchmarks

Streaming systems are complex; apart from correct functionality (which might differ between implementations and vendors) many non-functional aspects can be benchmarked such as memory comsumption, latency, and throughput. For RDF Stream Processing several benchmarks exist, shown as follows. From data stream management, older benchmarks exist which are not specific to RDF data but might be adapted. Some are listed below.

RDF Stream Benchmarks

Other Stream or CEP Benchmarks

  • BEAST 4
  • NEXMark 5
  • Linear Road 6
  • BiCEP 7 – a benchmarking framework
  • Fast Flower Delivery (FFD) 8 – a functional benchmarking scenario

  1. Zhang, Y.; Duc, P.; Corcho, O. & Calbimonte, J.-P. SRBench: A Streaming RDF/SPARQL Benchmark The Semantic Web –- ISWC 2012, Springer Berlin Heidelberg, 2012, 7649, 641-657 

  2. Dell’Aglio, D.; Calbimonte, J.-P.; Balduini, M.; Corcho, O. & Della Valle, E. On Correctness in RDF Stream Processor Benchmarking. The Semantic Web – ISWC 2013, Springer Berlin Heidelberg, 2013, 8219, 326-342 

  3. Le-Phuoc, D.; Dao-Tran, M.; Pham, M.-D.; Boncz, P.; Eiter, T. & Fink, M. Linked Stream Data Processing Engines: Facts and Figures. The Semantic Web – ISWC 2012, Springer Berlin Heidelberg, 2012, 7650, 300-312 

  4. Geppert, A.; Berndtsson, M.; Lieuwen, D. & Roncancio, C. Performance evaluation of object-oriented active database systems using the BEAST benchmark. Theor. Pract. Object Syst., John Wiley & Sons, Inc., 1998, 4, 135-149 

  5. Tucker, P.; Tufte, K.; Papadimos, V. & Maier, D. NEXMark – A benchmark for querying data streams. Oregon Health & Sciences University, 2002 

  6. Arasu, A.; Cherniack, M.; Galvez, E.; Maier, D.; Maskey, A. S.; Ryvkina, E.; Stonebraker, M. & Tibbetts, R. Linear road: a stream data management benchmark. VLDB ’04: Proceedings of the Thirtieth international conference on Very large data bases, VLDB Endowment, 2004, 480-491 

  7. Bizarro, P. BiCEP – Benchmarking Complex Event Processing Systems Event Processing, Internationales Begegnungs- und Forschungszentrum für Informatik (IBFI), Schloss Dagstuhl, Germany, 2007 

  8. Etzion, O. & Niblett, P. Event Processing in Action Manning Publications Co., 2010  


Definition of “Event” for Event Processing

I collected some definitions of event from the view of event processing research and practise. The emphasis is mine:

  • (Etzion & Niblett 2010)1 wrote:

    An event is an occurrence within a particular system or domain; it is something that has happened, or is contemplated as having happened in that domain. The word event is also used to mean a programming entity that represents such an occurrence in a computing system.

  • (Luckham & Schulte 2011)2 wrote:

    Anything that happens, or is contemplated as happening. Or: An object that represents, encodes, or records an event, generally for the purpose of computer processing.

  • (Gupta and Jain 2011)3 wrote:

    Events are first-class objects which means a fundamental information unit which can be stored, queried and merged with other events

  • (Hinze & Voisard 2002)4 wrote:

    An event is the occurrence of a state transition at a certain point in time.

  • (Michelson 2006)5 wrote:

    A notable thing that happens inside or outside your business.

  • (Mühl et al. 2006)6 wrote:

    Any happening of interest that can be observed from within a computer is considered an event. […] A notification is a datum that reifies an event, i.e., it contains data describing the event.

  1. Etzion, O. & Niblett, P. Event Processing in Action Manning Publications Co., 2010 
  2. Luckham, D. C. & Schulte, R. Event Processing Glossary – Version 2.0, 2011 Link 
  3. Gupta, A. & Jain, R. Managing Event Information: Modeling, Retrieval, and Applications, Morgan & Claypool Publishers, 2011 
  4. Hinze, A. & Voisard, A. A parameterized algebra for event notification services Ninth International Symposium on Temporal Representation and Reasoning, 2002. TIME 2002, 2002 
  5. Michelson, B. M. Event-Driven Architecture Overview Patricia Seybold Group, Feb, 2006 Link 
  6. Mühl, G.; Fiege, L. & Pietzuch, P. Distributed Event-Based Systems Springer-Verlag New York, Inc., 2006