Data Recorders

This document describes Kronometrix Data Recorders.

Introduction

Kronometrix connects to a wide variety of data sources: everything from IoT devices, ICT enterprise to weather and environment sensors. In addition to multifaceted data ingress, the distributed data fabric provides high-speed transport for data consolidation, analysis and visualization in real-time.

All recorded observations are stored as raw data. Raw data is produced by a recorder, which fetches data from a system, device or sensor, data which has not been modified, altered or changed in any way. All collected metrics are variable measured sequentially in time, called time series. All these observations collected over fixed sampling intervals create a historical time series. To easy the access to all this set of data we store the observations on commodity disk drives, compressed, in text format, like CSV format.

Time series let us understand what has happened in past and look in the future, using various statistical models

Data Message

All collected metrics over time are combined as a data message. There can be many types of data messages: metrics regarding computer system utilization cpu or memory utilization, or weather data from a meteorological station, or water cubic meters per hour from an water pump. A data message is in direct relation to a data source.

Data Source

A data source, is described as any system connected to a public or private network with a valid IPv4 or IPv6 address. Example: a server, a logger, a graphic workstation, an iPad or an IoT sensor capable to send and receive data. There can be many types of data sources, each having one or many of data messages:

  • Computer system: overall cpu utilization, disk and network IO, per device metrics (Linux, FreeBSD, Windows)

  • HTTP server: throughput and utilization along with its inventory data (Nginx, Apache, Tomcat)

  • Enterprise service: response time performance and availability (SMTPS, IMAP, HTTP, LDAP, NTP, AD)

  • Automatic weather station: air temperature and pressure, humidity, wind speed and direction

What is a data recorder?

Kronometrix uses data recorders to connect to various data sources and collect data. Some other systems are using software based agents to fetch data and transport it forward for analysis.

Recorder

A light software probe, designed to connect and extract data from different sources:  operating systems, web applications, weather and environment sensors, or industrial equipment, using several technologies and data communication protocols. The recorder does not offer data transport capabilities within.

Agent

A software module that resides on a certain system or device, which does not require user’s interaction, and have internal transport capabilities. Usually, a proprietary software application installed to collect performance data from a computer system, or network equipment.

Features

Light software probe

The recorders are designed as very simple standalone command-line interface utilities, which can run interactively or continnous in the background, to collect various data.

Data-Driven Programming

Recorders are designed based on the data-driven paradigm, implemented by a main loop, and focusing on structured data, pattern matching and resulting processing. This way the recorders can easily handle and process different data, without massive computing capabilities. 

Dynamic Programming Language

All recorders are developed using a high-level, dynamic programming language, Perl to easy and speed-up the development process and reduce the complexity.  

Rapid Prototyping

It is easy to start and develop a new recorder for any technology or industry, using Kronometrix Data Recording. Using a dynamic programming language will offer access to wide level of functions: from data communication protocols to string processing. 

Easy to Customize

Add or remove any metrics you want, in a matter of minutes. The recorders offer a very simple access to add new monitoring metrics.

Raw Data Support

The recorder is collecting all observations and metrics, and stores them as simple text flat files, on commodity disk drives. Data is structured as CSV records, to be universal compatible with any software or 3rd parties applications.

Data Ontology

You don’t need to record all sort of metrics and parameters. To help you, we have carefully selected and analyzed, for each industry, the most needed metrics for different business cases. The metrics have been grouped and classified, to build a very efficient data analysis process.

Timeseries Compatible

Raw data is produced by a recorder, which fetches data from a system, device or sensor, data which has not been modified, altered or changed in any way. All collected metrics are variable measured sequentially in time, called time series. All these observations collected over fixed sampling intervals create a historical time series.

IoT Readiness

Recorders are designed to connect and fetch data from various data sources, using different communication protocols. Being simple, without a monolithic architecture of a software agent, the recorders can easily adapt to any usage, from industrial MODBUS devices to healthcare or other industries. The recorders don’t offer any data transport capabilities, focusing entirely on the data collection.

Updated on March 10, 2021

Was this article helpful?

Related Articles