Cygnus is a connector in charge of persisting certain sources of data in certain configured third-party storages, creating a historical view of such data.

Internally, Cygnus is based on Apache Flume, a technology addressing the design and execution of data collection and persistence agents. An agent is basically composed of a listener or source in charge of receiving the data, a channel where the source puts the data once it has been transformed into a Flume event, and a sink, which takes Flume events from the channel in order to persist the data within its body into a third-party storage.

Cygnus is designed to run a specific Flume agent per source of data.

Current stable release is able to persist the following sources of data in the following third-party storages:

  • NGSI-like context data in:
    • HDFS, the Hadoop distributed file system.
    • MySQL, the well-known relational database manager.
    • CKAN, an Open Data platform.
    • MongoDB, the NoSQL document-oriented database.
    • STH Comet, a Short-Term Historic database built on top of MongoDB.
    • Kafka, the publish-subscribe messaging broker.
    • DynamoDB, a cloud-based NoSQL database by Amazon Web Services.
    • PostgreSQL, the well-known relational database manager.
    • Carto, the database specialized in geolocated data.
  • Twitter data in:

Documentation and API

FIWARE Cygnus: