10. TML Solution Components
Below describes the entire TML technology solution stack.
10.1. 1. TML Components: Three Binaries
TML is comprised of 3 binaries written in Go: found on Github
Viper - source - sink binary for Apache Kafka - runs on MAC/Windows/Linux
HPDE - AutoML binary for real-time data - runs on MAC/Windows/Linux
Viperviz - Visualization binary for real-time dashboards - runs on MAC/Windows/Linux
Important
These 3 binaries perform all functions in every TML solution. These binaries can be seen as microservices that can be instantiated any number of times to scale your solution for unlimited data processing.
These binaries are critical for TML solutions and are the “secret sauce” inside TML.
Binary |
Description |
Viper |
This is the CORE binary that performs all the major TML functions of processing and machine learning. This binary acts like a microservice that can be instantiated any number of times to process large amounts of real-time data. This binary is compatible with REST API. TML solutions (using Python) connect to this binary and instruct it to stream data to Kafka, preprocess data, and develop training data sets for machine learning. |
HPDE |
Hyper-prediction technology performs all machine learning functions. Viper connects to HPDE, using REST, and instructs it to perform machine learning. By off-loading this function to HPDE, TML can perform very fast machine learning (in few seconds) for each entity and sliding time window. Refer to :ref:`TML Performs Entity Level Machine Learning and Processing` and Entity Based Machine Learning By TML |
Viperviz |
This binary performs real-time streaming visualization using websockets. This is a very powerful binary because it uses the underlying network to streaming data to a client browser for fast, and very cost-effective, visualization of real-time TML solution outputs. This means users do NOT need a third-party visualization tool like Tableau or PowerBI. Users can create amazing real-time dashboards quickly and cheaply. Refer to example dashboards |
10.2. 2. TML Component: One Core Python Library
MAADSTML Python Library: https://pypi.org/project/maadstml/
MAADSTML Python library : API to build TML solutions that connect to the Viper binary. Refer to the MAADSTML API here MAADSTML Python Library API
Important
All TML solutions use the the MAADSTML python library. This is a critical library, and controls the 3 binaries.
10.3. 3. TML Component: Apache Kafka
TML integrates with Apache Kafka - on-premise or in the cloud.
Attention
TML binaries are integrated with Apache Kafka. TML solutions can be run On-PREMISE using Open Source Kafka or in the CLOUD using AWS MSK or Confluent Cloud.
10.4. 4. TML Component: Docker Containers
All TML solutions are containerized with docker for production deployments.
10.5. 5. TML Component: Kubernetes
All TML solution containers scale with Kubernetes. This allows companies to build fast, scalable, real-time solutions.
10.6. 6. TML Component: PrivateGPT for Generate AI
TML uses PrivateGPT for fast, real-time, AI. The container is here Docker PrivateGPT
Refer to TML and Generative AI for more details.
10.7. 7. TML Component: TML Solution Studio Container
For convenience, TML solution can be easily built using the TML Solution Studio container. Refer to TML Solution Studio (TSS) Container for further details.
10.8. How The TML Components Are Integrated
TML solutions are developed using the MAADSTML Python library that connects to the TML Binaries, using REST API, for streaming real-time data to Apache Kafka, processing data in Kafka, and performing machine learning. Once the TML solutions are built, they are containerized with Docker and scaled with Kubernetes.
Attention
TML performs in-memory processing of real-time data and does NOT require an external database - ONLY KAFKA is needed. This results in dramatic cost- savings for storage, compute and network data transfers.
TML does NOT perform SQL queries, it performs JSON PROCESSING. This results in much faster, and much cheaper processing of real-time data.