..
#
# Critical Application Monitoring (CAM)
#
# SPDX-FileCopyrightText: <text>Copyright 2023-2024 Arm Limited
# and/or its affiliates <open-source-office@arm.com></text>
#
# SPDX-License-Identifier: BSD-3-Clause
#
.. _getting_started:
###############
Getting Started
###############
This section of documentation describes how to download, build and execute CAM
project on Linux.
.. note::
All command examples on this page from the HTML document format can be copied
by clicking the copy button.
****************************
Build Host Environment Setup
****************************
System Requirements
===================
* AArch64 or x86_64 host to build the software
* Ubuntu Desktop or Server 20.04 Linux distribution
Install Dependencies
====================
The following dependencies are required to build CAM:
* CMake (3.16 or later)
* GCC (9.4 or later)
* Python (3.8.10 or later)
* Python-venv (3.8.10 or later)
* Pip (20.0 or later)
* pkg-config (0.29.1 or later)
* CUnit (2.1.3 or later)
* Doxygen (1.8.17 or later): Generate API documentation from source code.
* Sphinx (6.1.3 or later): Build the project documentation.
* Git version control system.
* Python virtual environment (3.8.2 or later).
* [optional] tcpdump (4.9.3 or later)
Install essential packages required for the build host:
.. code-block:: console
sudo apt-get install -y --no-install-recommends \
git \
build-essential \
cmake \
doxygen \
pkg-config \
python3 \
python3-venv \
python3-pip \
libcunit1-dev
.. note::
CAM project has Dockerfiles to build docker images which will contain all
dependencies needed by CAM. Refer to :cam-repo:`Dockerfiles` for more
information on the Dockerfiles.
********
Download
********
Download the CAM repository using Git, via:
.. code-block:: console
:substitutions:
mkdir ~/cam
git clone |cam remote| --branch |cam refspec| ~/cam/critical-app-monitoring
.. _getting_started_build_and_install:
*****
Build
*****
Create a Python virtual environment then install the Python dependencies:
.. code-block:: console
cd ~/cam/critical-app-monitoring
python3 -m venv ~/cam/venv
source ~/cam/venv/bin/activate
pip install -r requirements.txt
Create a build directory and run cmake:
.. code-block:: console
mkdir ~/cam/build
cd ~/cam/build
cmake ~/cam/critical-app-monitoring
Build and install cam-app-example, cam-service and documentation:
.. code-block:: console
make
cmake --install . --prefix ~/cam/cam-packages
Install cam-tool:
.. code-block:: console
cd ~/cam/critical-app-monitoring
pip install cam-tool/ --prefix ~/cam/cam-packages
.. note::
When running the command above, a warning message might appear
``"WARNING: The scripts futurize and pasteurize are installed in
'/home/<user>/cam/cam-packages/bin' which is not on PATH."`` where ``<user>``
might vary depending on the user system.
The following components get installed as a result of the above steps:
~/cam/cam-packages/bin
* **cam-app-example**: An application example that introduces how to integrate
the project.
* **cam-service**: The service executable used to monitor critical
applications.
* **cam-tool**: Python based **cam-tool** binary.
~/cam/cam-packages/share/doc/CAM/
* Includes all CAM documentations of the project, it can be rendered and
explored pointing the browser to
``file:///home/<user>/cam/cam-packages/share/doc/CAM/index.html`` where
``<user>`` might vary depending on the user system.
.. _getting_started_setup_terminal_environment:
**************************
Setup terminal environment
**************************
The terminal environment can be setup by running the following commands:
.. code-block:: console
source ~/cam/venv/bin/activate
python_path=$(find ~/cam/cam-packages -maxdepth 3 -name "site-packages")
export PYTHONPATH=${PYTHONPATH}:${python_path}
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:~/cam/cam-packages/lib/
***
Run
***
This demo requires two Linux terminals: ``Terminal 1`` for running **cam-tool**,
and **cam-app-example** and ``Terminal 2`` for running **cam-service**. Make
sure to follow the steps listed in the
:ref:`Build <getting_started_build_and_install>` section, afterwards make sure
to follow the step listed in
:ref:`Setup terminal environment <getting_started_setup_terminal_environment>`
section for both ``Terminal 1`` and ``Terminal 2``.
One of the prerequisites of CAM demo is the creation and deployment of CSD files
to the system where **cam-service** runs.
1. From ``Terminal 1``, start **cam-app-example** in calibration mode.
.. code-block:: console
mkdir ~/cam/cam-run
cd ~/cam/cam-run
~/cam/cam-packages/bin/cam-app-example \
--processing-period 3000 \
--processing-count 2 \
--stream-count 4 \
--enable-calibration-mode \
--uuid-base 94085ddc-bc10-11ed-9a44-7ef9696e \
--service-address 127.0.0.1 \
--service-port 21604
This mode generates one CSEL file for each stream. The CSEL file contains the
stream metadata and event timestamps for further analysis. The output
should look like as below:
.. code-block:: console
Cam application configuration:
Service IP address: 127.0.0.1
Service port: 21604
UUID base: 94085ddc-bc10-11ed-9a44-7ef9696e
Stream count: 4
Processing period (ms): 3000
Processing count: 2
Multiple connection support: false
Calibration mode support: true
Calibration directory: ./[uuid].csel
Fault injection support: false
Event(s) interval time (ms): 0
Using libcam v0.1
Starting activity...
Starting activity...
Starting activity...
Starting activity...
Stream 0 sends event 0
Stream 1 sends event 0
Stream 2 sends event 0
Stream 3 sends event 0
...
List the files generated:
.. code-block:: console
ls -1 *.csel
The CSEL files can be shown as below:
.. code-block:: console
94085ddc-bc10-11ed-9a44-7ef9696e0000.csel
94085ddc-bc10-11ed-9a44-7ef9696e0001.csel
94085ddc-bc10-11ed-9a44-7ef9696e0002.csel
94085ddc-bc10-11ed-9a44-7ef9696e0003.csel
2. From ``Terminal 1``, run **cam-tool** to analyze the CSEL files and convert
them to CSC files.
.. code-block:: console
~/cam/cam-packages/bin/cam-tool analyze --margin 500000 --input 94085ddc-bc10-11ed-9a44-7ef9696e0000.csel --output 94085ddc-bc10-11ed-9a44-7ef9696e0000.csc.yml
~/cam/cam-packages/bin/cam-tool analyze --margin 500000 --input 94085ddc-bc10-11ed-9a44-7ef9696e0001.csel --output 94085ddc-bc10-11ed-9a44-7ef9696e0001.csc.yml
~/cam/cam-packages/bin/cam-tool analyze --margin 500000 --input 94085ddc-bc10-11ed-9a44-7ef9696e0002.csel --output 94085ddc-bc10-11ed-9a44-7ef9696e0002.csc.yml
~/cam/cam-packages/bin/cam-tool analyze --margin 500000 --input 94085ddc-bc10-11ed-9a44-7ef9696e0003.csel --output 94085ddc-bc10-11ed-9a44-7ef9696e0003.csc.yml
The analysis result is reported by **cam-tool** as below:
.. code-block:: shell
CAM event log analyze report:
Input event log file: 94085ddc-bc10-11ed-9a44-7ef9696e0000.csel
Output configuration file: 94085ddc-bc10-11ed-9a44-7ef9696e0000.csc.yml
Stream UUID: 94085ddc-bc10-11ed-9a44-7ef9696e0000
Stream name: CAM STREAM 0
Timeout between init and start: 300000
Timeout between start and event: 450000
Application running times: 1
Processing count in each run: [2]
Event ID timeout
0 3500103
...
.. note::
The CSC files contain human-readable settings corresponding to the stream
metadata and event timestamps of the streams. Users can modify this
configuration, for example to fine tune timeout values depending on the
system capabilities.
List the files generated:
.. code-block:: console
ls -1 *.csc.yml
The CSC files can be shown as below:
.. code-block:: console
94085ddc-bc10-11ed-9a44-7ef9696e0000.csc.yml
94085ddc-bc10-11ed-9a44-7ef9696e0001.csc.yml
94085ddc-bc10-11ed-9a44-7ef9696e0002.csc.yml
94085ddc-bc10-11ed-9a44-7ef9696e0003.csc.yml
Run **cam-tool** along with ``pack`` option for each of the CSC files to
generate the corresponding CSD files:
.. code-block:: console
~/cam/cam-packages/bin/cam-tool pack --input 94085ddc-bc10-11ed-9a44-7ef9696e0000.csc.yml --output 94085ddc-bc10-11ed-9a44-7ef9696e0000.csd
~/cam/cam-packages/bin/cam-tool pack --input 94085ddc-bc10-11ed-9a44-7ef9696e0001.csc.yml --output 94085ddc-bc10-11ed-9a44-7ef9696e0001.csd
~/cam/cam-packages/bin/cam-tool pack --input 94085ddc-bc10-11ed-9a44-7ef9696e0002.csc.yml --output 94085ddc-bc10-11ed-9a44-7ef9696e0002.csd
~/cam/cam-packages/bin/cam-tool pack --input 94085ddc-bc10-11ed-9a44-7ef9696e0003.csc.yml --output 94085ddc-bc10-11ed-9a44-7ef9696e0003.csd
List the files generated:
.. code-block:: console
ls -1 *.csd
The CSD files can be shown as below:
.. code-block:: console
94085ddc-bc10-11ed-9a44-7ef9696e0000.csd
94085ddc-bc10-11ed-9a44-7ef9696e0001.csd
94085ddc-bc10-11ed-9a44-7ef9696e0002.csd
94085ddc-bc10-11ed-9a44-7ef9696e0003.csd
3. From ``Terminal 2``, run **cam-service** to monitor the multiple streams
that will be generated by **cam-app-example**.
.. code-block:: console
cd ~/cam/cam-run/
mkdir -p ~/cam/cam-packages/usr/share/cam-data/
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:~/cam/cam-packages/lib/
~/cam/cam-packages/bin/cam-service --config-dir ~/cam/cam-packages/usr/share/cam-data/ --log-level info --address 127.0.0.1 --port 21604
Users have the flexibility to customize the socket on which **cam-service**
listens by modifying the IP address and port using command-line parameters.
Once critical applications connect to **cam-service** and initiate event
streams, **cam-service** can receive and process stream events through this
socket. The following messages are expected from the terminal:
.. code-block:: shell
Cam service configuration:
IP address: 127.0.0.1
port: 21604
log level: info
configuration directory: ~/cam/cam-packages/usr/share/cam-data/
fault action: none
4. From ``Terminal 1``, run **cam-tool** along with ``deploy`` option to deploy
the CSD files into the system where **cam-service** is running.
.. code-block:: console
~/cam/cam-packages/bin/cam-tool deploy --input 94085ddc-bc10-11ed-9a44-7ef9696e0000.csd --address 127.0.0.1 --port 21604
~/cam/cam-packages/bin/cam-tool deploy --input 94085ddc-bc10-11ed-9a44-7ef9696e0001.csd --address 127.0.0.1 --port 21604
~/cam/cam-packages/bin/cam-tool deploy --input 94085ddc-bc10-11ed-9a44-7ef9696e0002.csd --address 127.0.0.1 --port 21604
~/cam/cam-packages/bin/cam-tool deploy --input 94085ddc-bc10-11ed-9a44-7ef9696e0003.csd --address 127.0.0.1 --port 21604
The output on the ``Terminal 2`` should look like as below, the connection
number might change:
.. code-block:: console
Connection 1 is created.
Deploy Message
Connection 1 is closed.
...
From ``Terminal 1``, list the CSD files deployed:
.. code-block:: console
ls -1 ~/cam/cam-packages/usr/share/cam-data/*.csd
The CSD files can be shown as below, the output might change because the
``~`` is expanded in different ways on different systems, usually to
``/home/<user>``:
.. code-block:: console
~/cam/cam-packages/usr/share/cam-data/94085ddc-bc10-11ed-9a44-7ef9696e0000.csd
~/cam/cam-packages/usr/share/cam-data/94085ddc-bc10-11ed-9a44-7ef9696e0001.csd
~/cam/cam-packages/usr/share/cam-data/94085ddc-bc10-11ed-9a44-7ef9696e0002.csd
~/cam/cam-packages/usr/share/cam-data/94085ddc-bc10-11ed-9a44-7ef9696e0003.csd
5. From ``Terminal 1``, start **cam-app-example** to run the Critical
Application with four streams. Each stream sends an event message 10 times
with a period of 3000ms.
.. code-block:: console
~/cam/cam-packages/bin/cam-app-example \
--processing-period 3000 \
--processing-count 10 \
--stream-count 4 \
--uuid-base 94085ddc-bc10-11ed-9a44-7ef9696e \
--service-address 127.0.0.1 \
--service-port 21604
The following messages are expected on ``Terminal 1``:
.. code-block:: console
Cam application configuration:
Service IP address: 127.0.0.1
Service port: 21604
UUID base: 94085ddc-bc10-11ed-9a44-7ef9696e
Stream count: 4
Processing period (ms): 3000
Processing count: 10
Multiple connection support: false
Calibration mode support: false
Fault injection support: false
Event(s) interval time (ms): 0
Using libcam v0.1
Starting activity...
Starting activity...
Starting activity...
Starting activity...
And the log of event messages send are shown repeatedly:
.. code-block:: console
Stream 0 sends event 0
Stream 1 sends event 0
Stream 2 sends event 0
Stream 3 sends event 0
Stream 0 sends event 0
Stream 1 sends event 0
Stream 2 sends event 0
Stream 3 sends event 0
...
As observed from the ``Terminal 2``, **cam-service** loads four CSD files for
monitoring. In the following log, the stream messages are received and
processed by it:
.. code-block:: console
Connection 4 is created.
Init Message
Stream 94085ddc-bc10-11ed-9a44-7ef9696e0001 configuration is loaded.
Init Message
Stream 94085ddc-bc10-11ed-9a44-7ef9696e0000 configuration is loaded.
Init Message
Stream 94085ddc-bc10-11ed-9a44-7ef9696e0002 configuration is loaded.
Init Message
Stream 94085ddc-bc10-11ed-9a44-7ef9696e0003 configuration is loaded.
Start Message
Start Message
Start Message
Start Message
Event Message
Event Message
Event Message
Event Message
Event Message
# Repeated event messages
...
6. CAM provides a mechanism to generate a fault from within **cam-app-example**
that can be detected by **cam-service**. From ``Terminal 1``, run
**cam-app-example** with fault injection set to event stream 0:
.. code-block:: console
~/cam/cam-packages/bin/cam-app-example \
--processing-period 3000 \
--processing-count 10 \
--stream-count 4 \
--enable-fault-injection \
--fault-injection-stream 0 \
--fault-injection-time 1000 \
--uuid-base 94085ddc-bc10-11ed-9a44-7ef9696e \
--service-address 127.0.0.1 \
--service-port 21604
The fault happens 1000ms after stream initialization. At that time
**cam-service** should detect a stream temporal error with the following
output on ``Terminal 2``.
.. code-block:: console
#Repeated event messages
...
ERROR: Stream temporal error:
ERROR: stream_name: CAM STREAM 0
ERROR: stream_uuid: 94085ddc-bc10-11ed-9a44-7ef9696e0000
ERROR: event_id: 0
ERROR: time_received: 0
ERROR: time_expected: 1701066141314201
...
The following diagram describes the fault injection option:
.. image:: images/fault_injection.svg
:align: center
After fault is injected, **cam-app-example** stops performing new processing
activity. The interval time from stream initialization to the first activity
is 500ms.
.. note::
In addition to sending only one event during each activity,
**cam-app-example** supports sending a specific number of events within one
activity.
.. code-block:: console
~/cam/cam-packages/bin/cam-app-example \
--processing-period 3000 \
--processing-count 10 \
--stream-count 4 \
--uuid-base 94085ddc-bc10-11ed-9a44-7ef9696e \
--service-address 127.0.0.1 \
--service-port 21604 \
--event-interval "<t0,t1,...tn>"
With the command line above, cam-app-example sends n events in each processing
activity. The interval time between events is defined by t\ :sub:`0` -
t\ :sub:`n`. The following diagram shows the definition of the event interval
option.
.. image:: images/event_interval.svg
:align: center
t\ :sub:`n`: The list of event intervals(in ms) in each processing activity.
t\ :sub:`0` is the interval time from activity start to event 0.
8. To stop **cam-service** running on ``Terminal 2``, type ``Ctrl-C``.
.. _getting_started_cam_integration_with_kronos:
***************************
CAM Integration with Kronos
***************************
A complete integration of CAM can be found in Kronos Reference Software Stack.
This integration demonstrates the feasibility of monitoring Primary Compute
applications by **cam-service** running on Safety Island.
Kronos makes use of CAM CMake option **CAM_TARGET=<Linux> | <Zephyr>** to build
**cam-service** for Zephyr.
Refer to `Kronos Reference Software Stack CAM Documentation`_ for more
information on the integration.