TML Solution Studio (TSS) Container ====================================== Before you can build TML solutions, you will need to run the TSS container for your OS. The TSS builds your TML solutions. .. list-table:: * - **TSS Containers** * - `Windows/Linux Users `_ * - `Mac/Linux Users `_ .. tip:: Watch the TSS `YouTube Video `_ .. important:: You MUST have the pre-requisites met before running this container: :ref:`TSS Pre-Requisites` .. figure:: tss0.png :scale: 50% .. figure:: tss-main-arch.png :scale: 50% This is the main container that you need to use to build TML solutions. Below is the :ref:`TSS Docker Run Command` to run the container. .. important:: **Use this TSS container and start building amazing, advanced and scalable real-time data streaming solutions - with real-time dashboards - auto deployment to Docker - auto solution documentation - integrated with AI - integrated with Github - integrated with Apache Airflow - integrated with Apache Kafka - in just a few minutes.** TML Solution Container Global Deployment ------------------------------ There are TWO ways to deploy TML solution containers: #. **Docker Container:** Anyone around the world can run the container using the Solution's Docker Run Command #. **Kubernetes:** Anyone around the world can run the `TML Solution containers in kubernetes `_ For 1: see figure below .. figure:: tmldeploy.png :scale: 70% TML solution developer creates the TML solution container, any user around the world can run this solution with their own: 1. Github username/repo 2. Readthedocs token 3. Docker username This means that ONE TML solution container is used by others with their OWN data, configurations parameters, and credentials without creating thousands of containers … think of this as FORKING in Github. TSS Pre-Requisites ------------------- .. attention:: **The following prerequisites MUST be met before you run the TML Solution Studio Container** using the :ref:`TSS Docker Run Command` : 1. You MUST Install Docker - in Linux Ubuntu run (if using Windows use `WSL `_): .. code-block:: sudo apt update sudo apt upgrade sudo apt install docker.io sudo docker --version sudo docker run hello-world sudo docker ps 2. You MUST have a `Github Account `_ 3. You MUST Clone Github Repo: https://github.com/smaurice101/raspberrypi.git 4. You MUST Create Github **Personal Access Token**. Refer to :ref:`Set Up Personal Access Tokens in Github` 5. You MUST have sign up for a `Docker Hub account: `_ 6. Create a `Readthedocs `_ account and get an API token: :ref:`Set Up Readthedocs` 7. Create a `HiveMQ account `_ - while this is OPTIONAL - if using MQTT and HiveMQ you will need this. 8. Kafka Cloud API keys from `Confluent `_ or `AWS MSK `_ - while this is OPTONAL - it will be for production, or large scale, deployments. FOLLOW THE :ref:`How To Use the TML Solution Container` SECTION. .. tip:: DEMO CREDENTIALS are available for quick access: :ref:`TSS Demo Github, Docker and Readthedocs Site Credentials` Note: the demo credentials are public so others may be using it. Once you are happy with TSS you will need to enter yur own private credentials. TSS Contains a TML Dev Environment ---------------------------- .. important:: Another powerful feature is the TSS TML Development environment which is contained directly inside the TSS container. TSS comes with all the TML solution components installed like: 1. Apache Kafka, 2. TML binaries, 3. MariaDB config db 4. your TML DAG scripts, 5. Viperviz for visualization Once you are satisfied with your solution - you can use the :ref:`STEP 8: Deploy TML Solution to Docker : tml-system-step-8-deploy-solution-to-docker-dag` to deploy your solution to Docker. .. tip:: TML developers can test each component or their entire TML solution inside the TSS before deploying the solution in the container. This is a very convenient way to make sure all the solution components are working before shipping your TML product. TSS Docker Run Command -------------------- .. note:: **If you are producing data using a local file, you need to add an extra -v volume map to the /rawdata folder in the container:** Refer to :ref:`Producing Data Using a Local File`. For example add **-v /your_localmachine/foldername:/rawdata:z**, where **your_localmachine/foldername** is a path in your local machine, and it is where you save your local file for processing. Your file must contains JSON messages on each line. See `Sample File `_ .. code-block:: docker run -d --net="host" \ --env CHIP="AMD64" \ --env MAINHOST=127.0.0.1 \ --env TSS=1 \ --env SOLUTIONNAME=TSS \ --env NPMSERVERPORT=3000 \ --env AIRFLOWPORT=9000 \ --env VIPERVIZPORT=9005 \ --env EXTERNALPORT=-1 \ -v /var/run/docker.sock:/var/run/docker.sock:z \ -v /:/dagslocalbackup:z \ -v /your_localmachine/foldername:/rawdata:z \ --env READTHEDOCS='' \ --env GITREPOURL= \ --env GITUSERNAME='' \ --env GITPASSWORD='' \ --env DOCKERUSERNAME='' \ --env DOCKERPASSWORD='' \ --env MQTTUSERNAME='' \ --env MQTTPASSWORD='' \ --env KAFKABROKERHOST=127.0.0.1:9092 \ --env KAFKACLOUDUSERNAME='' \ --env KAFKACLOUDPASSWORD='' \ --env SMTP_SERVER='' --env SMTP_PORT= --env SMTP_USERNAME='' --env SMTP_PASSWORD= --env recipient='' --env UPDATE=1 \ maadsdocker/tml-solution-studio-with-airflow-amd64 .. list-table:: * - **Parameter** - **Description** * - CHIP - Specifies the container OS. NOTE: If you are using MAC then change to CHIP=ARM64 * - NPMSERVERPORT - This is the port number for TSS. For example, NPMSERVERPORT=3000, will cause TSS to listen on port 3000 i.e. http://localhost:3000 Users can change this port. * - MAINHOST=127.0.0.1 - This is the IP address for the TML solution container. It will normally listen on 127.0.0.1 * - TSS - Do not modify. * - SOLUTIONNAME - Do not modify. * - AIRFLOWPORT=9000 - This is the AIRFLOWPORT. This port will be needed to access the TML solution studio from your browser. For sample, enter: http://localhost:3000/ You will be asked for a username and password: enter **tml** for both. * - VIPERVIZPORT - Choose a Viperviz port for visualization. For example, 9005 * - -v /:/dagslocalbackup:z - If you like, you can also backsup the dags to your local folder with this volume mapping * - -v /your_localmachine/foldername:/rawdata:z - If you like, you can also map our local folder to the rawdata folder. This is neede if you will be processing local files with TML. * - -v /var/run/docker.sock:/var/run/docker.sock:z - This maps the docker volume to the container - so TML studio can automatically build your solution container. * - READTHEDOCS= - Create, copy and paste the Readthedocs token here. Refer to :ref:`Set Up Readthedocs` * - GITREPOURL= - This is your Git repo you cloned from: https://github.com/smaurice101/raspberrypi.git. * - GITUSERNAME= - This is the username to your repo. * - GITPASSWORD= - This is the **Personal Access Token** for your repo. Look at the image below to find out how to generate this token. * - DOCKERUSERNAME= - This is your Docker Hub username. * - DOCKERPASSWORD= - This is your password to Dockerhub account. * - MQTTUSERNAME= - This is your MQTT username. * - MQTTPASSWORD= - This is your password to MQTT cluster. * - UPDATE=1 - This enables system updates if set to 1: meaning any updates to the system DAGS made by the TSS system maintainer will update all the user DAGS in all of the projects. This is a remote GitHub pull that keeps users ALWAYS with the updated Dags. You can Turn OFF system updates by setting to 0. * - **maadsdocker/tml-solution-studio-with-airflow-amd64** - This is the official TML Solution Studio container for Window/Linux users with AMD64 chip architecture. If using MAC/Linux change: **amd64** to **arm64** **For example:** maadsdocker/tml-solution-studio-with-airflow-**arm64** .. code-block:: docker run -d --net="host" \ --env CHIP="ARM64" \ --env MAINHOST=127.0.0.1 \ --env TSS=1 \ --env SOLUTIONNAME=TSS \ --env NPMSERVERPORT=3000 \ --env AIRFLOWPORT=9000 \ --env VIPERVIZPORT=9005 \ --env EXTERNALPORT=-1 \ -v /var/run/docker.sock:/var/run/docker.sock:z \ -v /:/dagslocalbackup:z \ -v /your_localmachine/foldername:/rawdata:z \ --env READTHEDOCS='' \ --env GITREPOURL='' \ --env GITUSERNAME='' \ --env GITPASSWORD='' \ --env DOCKERUSERNAME='' \ --env DOCKERPASSWORD='' \ --env MQTTUSERNAME='' \ --env MQTTPASSWORD='' \ --env KAFKACLOUDUSERNAME='' \ --env KAFKACLOUDPASSWORD='' \ --env UPDATE=1 \ maadsdocker/tml-solution-studio-with-airflow-arm64 Backing Up TML Solutions Locally ------------------------------ All TML solutions are not only auto-committed to GitHub but also saved locally to disk. .. important:: It is highly recommended you map your local folder to the **dagslocalbackup** folder: **-v /:/dagslocalbackup:z** This ensures that if anything happens to Github you always have a local copy of all of your solution dags. How To Use the TML Solution Container ------------------------- .. tip:: Once you have the TML Solution container running you can go to your favourite browser and type the URL: http://localhost:3000 After you enter the URL you will the following website: .. figure:: ts1.png :scale: 70% .. tip:: The username and password are both **tml** After you have signed in successfully you will see the following screen with example DAGs: .. figure:: ts2.png :scale: 70% If you scroll down you will see the **TML DAGs** - as defined here: :ref:`DAG Table`. These are the DAGs you will use to build your TML Solutions: .. figure:: ts3.png :scale: 70% TSS Code Editor ----------------- .. important:: Next go into the DAG Code Editor: Select Drop-down menu **Admin --> DAGs Code Editor**. Most of your TML Solution building will be done here. Note the DAGs solution process flows defined here: :ref:`Apache Airflow DAGs` .. figure:: ts4.png :scale: 70% TSS Demo Github, Docker and Readthedocs Site Credentials ---------------------- .. important:: To make it easier for users to run a TML solution quickly, a TML Demo Github site is provided here: - **GITHUB USERNAME:** tsstmldemo - **GITHUB REPO URL:** https://github.com/tsstmldemo/tsstmldemo - **GITHUB Personal Access Token:** - **Readthedocs Token:** aefa71df39ad764ac2785b3167b77e8c1d7c553a - **Docker Username:** tsstmldocker - **Docker Password:** TssTml0828!? **Note: The above credentials are for demo purposes ONLY and are shared with anyone.** They ARE NOT meant to be used permanently. If you like TSS/TML and plan to use it seriously, then personal accounts and credentials must be used. Refer to `TSS Pre-requisites `_ on how to set these up. Readthedocs Documentation URL ------------------------- .. important:: The TSS is integrated with `Readthedocs `_ for automated documention via Github. All TML auto-generated documentaton can be found on readthedocs. The format of the URL is: https://.readthedocs.io. For example, if you created a project name **cybersecurityrtms-3f10**, then the url will be: `https://cybersecurityrtms-3f10.readthedocs.io `_ The **solution name** can be found in the Step 1 task under your project name. As shown in the figure below, the TML solution name is **cybersecurityrtms-3f10** .. tip:: You also add **-a stdout** to your solution Docker Run command. This will allow you to see the **READTHEDOCS URL** to your documentation. .. figure:: rtdurlformat.png :scale: 70% Common Docker and TMUX Commands -------------------- This is a list of common commands for Docker and Tmux. .. list-table:: * - **Description** - **Command** * - List Docker containers - Type: **docker image ls** * - Delete Docker containers - From: **docker image ls** and copy the **REPOSITORY** to delete Type: **docker rmi \-\-force** * - List Running Docker containers - Type: **docker ps** * - Stop Running Docker containers - From **docker ps** copy the Container ID Type: **docker stop ** * - Stop ALL Running Docker containers - Type: **docker stop $(docker ps -a -q)** * - Go inside the Docker containers - From **docker ps** copy the Container ID Type: **docker exec -it bash** * - List the TMUX windows once inside the container - Type: **tmux ls** * - Go inside TMUX windows - From **tmux ls** copy the window name you want to enter Type: **tmux a -t ** * - To scroll inside a TMUX window - Press: **CTRL+b, [** * - To UN-scroll inside a TMUX window - Press: **CTRL + [** * - To EXIT a TMUX window - Press: **CTRL + b, d** * - To EXIT docker container - Type: **exit** * - **To Prune Docker System and Recover memory** - System Prune type: **docker system prune** * - To remove ALL containers - Type: **docker rm $(docker ps -a -q)** TSS Logging ----------------- The entire TSS solution build process is logged and committed to Github. This makes it very convenient to check for any errors in the TSS build process, and because errors are commited to the remote branch, the errors become visible to others to help in quickly rectifying any issues. .. figure:: tsslogs.png :scale: 70% .. tip:: The logs are committed to your Github folder: **/tml-airflow/logs/logs.txt** .. figure:: tsslogs2.png :scale: 70%