An integration component for receiving, transformation, aggregation, enrichment and storing data to the DB. Overall it is the processing of data. Communication (requests, data,…) takes place via Message Queue or RESTful API (not implemented yet), synchronization via Message Queue. The main components are Workers, which perform defined operations with data. They pick up data (or the message to perform a task) from the queue system and process them individually and independently. Performs computationally demanding operations. The application does not just have to receive the data, it can also download it and process it (polling) after an initiative (from Message Broker). Realized as a Node.js application. Written in TypeScript, uses Mongoose and Sequelize ORM for connecting to the DB, amqplib for connecting to RabbitMQ.
Here you can find a git repository with complete codes of the module under the MIT license. This repository also serves as a primary source for the operating environment of the Golemio data platform. What we publish, we use ourselves.