Working Group

Written Philipp Ahmann, ELISA Ambassador and TSC member and Business Development Manager at Robert Bosch GmbH

Many projects share architectural elements, including container technologies, RTOS requirements, or virtualization, but safety concerns mainly touch single-point elements of such a system rather than sticking everything together. This construction of full system architectures is created by distributors or within companies towards a product. Due to their complexity, these systems utilize the work results of various related projects, either open source or proprietary. 

As a result of these recent discussions with industry partners and the open source software (OSS) community, the ELISA Technical Steering Committee (TSC) approved the formation of the Systems Work Group in June 2022.

Although the use of these systems heavily differs, their system architecture elements are repeating. As mentioned, the embedded systems world often ends up with a heterogeneous architecture mixed with RTOS and Linux, virtualization and containers. This is true for medical, industrial, automotive and other industries. These systems have to fulfill certain compliance requirements, including a proper Software Bill of Material (SBOM). It is needed to tailor them for various product lines, to enable a quick project start, reduce maintenance and training efforts and ensure faster product ramp ups. Lastly, these systems typically connect with cloud services for features like monitoring, over the air update, or feature extensions, to increase their maintenance time and to architect higher care for safety as well as dependability, reliability and safety. 

Working Groups, Project interaction and Architecture

The Systems WG explicitly encourages the collaboration of companies and OSS projects to develop a reference system as a workbench. It will target a community, which either also works toward enabling safety use cases with open source software or which plans to make use of open source mixed-criticality system elements as a base for their product lines. 

Currently, the ELISA project already interacts with following community projects:

• Automotive Grade Linux (AGL) 
• Yocto Project
• Xen Project
• Zephyr Project

Taking these puzzle pieces from different projects and bringing them together in a reproducible showcase with goals such as proper SBOM generation is not established in any other open source project so far. There is a lack of an umbrella work group within the extended embedded and edge domain, which provides a reference architecture and a reproducible, dependable system to serve as a blueprint to explore new use cases, functions, or modules. It goes well in line with the recent trend toward software defined vehicles and the first steps into software defined industries, but is not limited to this.

Although the reference architecture may include elements like hypervisors or containers, which have explicitly not been part of ELISA so far, it should enable other working groups to showcase their work on process, features, and tooling with the implementation of a reference system. It can also create ideas and show potential risks or hazards due to a better system understanding; a systems-based understanding of an architecture later on used in a similar way in real world products.

As an important remark, the working group starts with a reference system fully based on “as- is” open source technology. Only a few of the elements contained in the reference system have been developed with considerable safety in mind. This means the created system will act as an example and stimulus environment for mixed criticality Linux based use cases, but is not at all safe or certifiable. 

Instead, it should enable users to exchange certain elements like the container technology, RTOS, or hypervisor technology. This will require a modular design, good documentation and a way to reproduce the system with easy to follow steps.

Next steps and Beyond

The starting point for the reference system is based on the work Stefano Stabelini presented during the Open Source Summit North America, hosted in Austin, TX, June 21-24, called “Static Partitioning with Xen, LinuxRT, and Zephyr: A Concrete End-to-end Example”. As a next step, the system architecture presented in this tutorial will be further extended by adding the Yocto Project as build tooling for all system elements and enhancing the base Linux operating system by adding Automotive Grade Linux towards the end of this year. A full SBOM shall be generated as well. 

Through early 2023, it is planned to extend the system to a physical hardware next to the existing QEMU image, and to also to add container technology, along with cloud connections represented. By doing so, it is possible that a Debian-based Linux may be used as a guest VM or within a container to pick up a wider community.

The work will be documented in a way that showcases why and how the system is configured as it is, while the underlying tooling will enable reproducibility by a few steps as checkout, build and deploy.

Summary

Along with proper documentation on tailoring and reproduction of the showcase, the reference system can serve as a workbench to challenge concepts and implementations from ELISA and other open source projects. It can be used as a quick start to test new ideas during proofs of concept and to let others easily experience achievements of the ELISA project working groups dealing with Linux features, tooling, architecture and more. It is there to foster collaboration of a wide range of open source projects and enables learning from real world scenarios close to later product line architectures. 

Join us

The new Systems WG hosts weekly meetings on Mondays  at 15:00 UTC (8 am PT/11 am ET). To receive the meeting invitation (and working group posts), simply subscribe to the mailing list. The meeting invitation will be sent to you after subscription. Also feel free to reach out to one of the ELISA ambassadors to learn about further ELISA activities.

Written by Shuah Khan, Chair of the ELISA Project Technical Steering Committee, and Milan Lakhani, member of the ELISA Medical Devices Working Group

Key Points

• Understanding system resources necessary to build and run a workload is important.
• Linux tracing and strace can be used to discover the system resources in use by a workload.
• Once we discover and understand the workload needs, we can focus on them to avoid regressions and evaluate safety.

OpenAPS is an open source Artificial Pancreas System designed to automatically adjust an insulin pump’s insulin delivery to keep Blood Glucose in a safe range at all times.

It is an open and transparent effort to make safe and effective basic Automatic Pancreas System technology widely available to anyone with compatible medical devices who is willing to build their own system.

Broadly speaking, the OpenAPS system can be thought of performing 3 main functions. Monitoring the environment and operational status of devices with as much data relevant to therapy as possible collected, predicting what should happen to glucose levels next, and enacting changes through issuing commands, emails and even phone calls.

ELISA Medical Devices WG team has set out to discover the Linux kernel subsystems used by OpenAPS. Understanding the kernel footprint necessary to run a workload helps us focus on the  subsystem and modules that make up the footprint for safety. We set out to answer the following questions:

• What happens when an OpenAPS workload runs on Linux?
• What are the subsystems and modules that are in active use when OpenAPS is running?
• What are the interactions between OpenAPS and the kernel when a user checks how much insulin is left in the insulin pump?

So how do we discover the Linux kernel subsystem in use? The Linux kernel has several features and tools that can help discover which modules and functions are being used by an application during run-time. Using these tools, we can gather the system state while the OpenAPS workload is running to determine which parts of the kernel are being used.

Let’s talk a bit about kernel states. The kernel system state can be viewed as a combination of static and dynamic features and modules. Let’s first define what static and dynamic system states are and then explore how we can visualize the static and dynamic system parts of the kernel.

Static System View comprises system calls, features, static modules and dynamic modules enabled in the kernel configuration. Supported system calls and Kernel features are architecture dependent. System call numbering is different on different architectures. We can get the supported system call information using the auditd package tool. ausyscall –dump prints out the supported system calls on a system. You can install the auditd package by running “sudo apt-get install auditd” on Debian systems. Linux kernel script scripts/checksyscalls.sh can be used to check if current architecture is missing any function calls compared to i386. scripts/get_feat.pl <get_feat.pl list> can be used to list the Kernel feature support matrix for a system or get_feat.pl list –arch=arm for example lists the Kernel feature support matrix of the ‘arm’ architecture

Dynamic System View comprises system calls, ioctls invoked, and subsystems used during the runtime. A workload could load and unload modules and also change the dynamic system configuration to suit its needs.

OpenAPS Static View

Let’s first look at the OpenAPS sources to understand the workload from a static view. The OpenAPS workload is a collection of  python libraries, python-dev, software-properties-common, python-software-properties, python-numpy, python-pip, nodejs-legacy and npm. You can have a look at the OpenAPS repositories in https://github.com/openaps with the two main ones being https://github.com/openaps/oref0 and https://github.com/openaps/openaps . The initial dependencies that the user installs can be seen here https://github.com/openaps/docs/blob/master/scripts/quick-packages.sh 

One easier way to understand its runtime characteristics is watching the system state while a workload is running. We determined that the following methodology and tools would work well for us to observe the system activity. 

What is the methodology?

The first step is gathering the default system state such as the dynamic and static modules loaded on the system. lsmod command prints out the dynamically loaded modules on a system. Statically configured modules can be found in the kernel configuration file. Understanding the default system is necessary to determine the changes if any made by the workload.

The next step is discovering the Linux kernel footprint used by OpenAPS by enabling event tracing before starting the workload, gathering dynamic modules while the workload is running. Once the workload is stopped, gather the event logs, kernel messages.

Once we have the necessary information, we can extract the system call numbers from the event trace log and map them to the supported system calls.

Putting our methodology to test

Our initial plan was to use strace to trace system calls and signals used by OpenAPS commands in strace. strace runs the specified command until it  exits. It intercepts  and  records  the  system  calls  which are called by the process and the signals which are received by the process. It gives insight into the syscalls OpenAPS commands use. However, we realized quickly that OpenAPS employs setup scripts to launch its workload. As a result, using strace was not an option.

We modified OpenAPS oref0-setup.sh in https://github.com/OpenAPS/oref0.git to enable event tracing before OpenAPS starts its workload (processes, shell scripts).  This approach gives us an overall information about OpenAPS. We will develop a higher level view and then dive into individual OpenAPS commands..

==================================================================

diff –git a/bin/oref0-setup.sh b/bin/oref0-setup.sh

index 261da95b..5ae666e2 100755

— a/bin/oref0-setup.sh

+++ b/bin/oref0-setup.sh

@@ -1269,6 +1269,11 @@ if prompt_yn “” N; then

 fi # from ‘read -p “Continue? y/[N] ” -r’ after interactive setup is complete

+# ELISA enable event tracing

+echo “ELISA: Enable event tracing on all events”

+echo 1 > /sys/kernel/debug/tracing/events/enable

+cat /sys/kernel/debug/tracing/events/enable

+

 # Start cron back up in case the user doesn’t decide to reboot service cron start

==================================================================

We were able to gather traces with the above modification to oref0-setup.sh. As mentioned earlier, the ausyscall tool dumps out mapping for syscalls and their corresponding syscall table numbers. The mapping is architecture dependent for some system calls. The trace data includes NR followed by a number in the trace file. These are the syscalls that are run during the time the tracing was on. Using the tracing and system call information, we determined the system calls used by the OpenAPS workload. In addition we used scripts/checksyscalls.sh to check for system call support status on RasPi.

What did we do to gather traces and system state?

• Start OpenAPS workload with modified the modification to enable tracing
• Let the workload run for 30 minutes
• Discard last 5 minutes of trace from analysis to account for interference (rsync and plugging devices) with trace file extraction
• Stop OpenAPS.
• Extract trace file from the system – cat /sys/kernel/debug/tracing/trace > trace.out
• Run lsmod after OpenAPS workload starts to gather module information
• Run “ausyscall –dump > syscalls_dump.out”

Analyzing traces:

• Map the NR (syscal) numbers from the trace to syscalls from the syscalls dump.
• Categorize system calls and map them to Linux subsystems.

Findings and observations:

• ioctls in use are all spidev ioctls. OpenAPS python modules use spidev interfaces.
  • https://www.kernel.org/doc/Documentation/spi/spidev
  • https://pypi.org/project/spidev/
• Raspi SPI Dev and Python Spidev use (914 spidev ioctls invoked)
• CONFIG_SPI, SPI_BCM2835 options enabled.
• Heavy use of syscall32
  • Use scripts/checksyscalls.sh to check for support status

Kernel module usage:

Module Name	Usage count  (OpenAPS)	Usage count (default)
cmac	Not loaded	1
ecdh_generic	1 (bluetooth)	2 (bluetooth)
ecc	1 (ecdh_generic)	1 (ecdh_generic)
spidev	2	Not loaded
i2c_bcm2835	1	Not loaded
spi_bcm2835	0	Not loaded
i2c_dev	2	Not loaded
ipv6	26	24

Subsystem usage:

Subsystem	# of calls
kmem_*	200990
mm_*	182471
sched_*	195241
rcu_*	223011
irq_*	1781503
kmalloc	79801
cpu_idle	62623
rss_stat	22130
ipi_*	42514
sys_*	148034
vm_*	60489
task_*	72813
timer_	58572
hrtimer_*	152271
softirq_*	28860
workqueue_*	6129
writeback_*	3933
ext4_*	34461
jbd2_*	2062
block_*	3590
dwc_otg*	27701
arch_timer	13446

Updated System view:

Conclusion

This tracing activity was a good way of identifying which parts of the kernel are used by OpenAPS. This helped to generate the Updated system view, so it is useful for our goal to do an STPA analysis of OpenAPS Operating System activity. We plan to theoretically analyse how these different subsystems can interact unsafely, while using fault injection and mocked components to collect more traces.

As mentioned earlier, the approach we used so far gave us a higher level of information about the OpenAPS usage. It isn’t ideal that this higher information doesn’t tell us the system usage by individual OpenAPS commands. As an example, we won’t be able to clearly identify which system calls are invoked when a user queries insulin pump status.

We are in the process of gathering fine grained information about individual OpenAPS commands and important use-cases. As an example, what subsystems are used when a user queries the insulin pump status. We are  using the strace command to trace the OpenAPS commands. We will share our findings in our next blog on this topic.

SPDX-License-Identifier: CC-BY-4.0

This document is released under the Creative Commons Attribution 4.0 International License, available at https://creativecommons.org/licenses/by/4.0/legalcode. Pursuant to Section 5 of the license, please note that the following disclaimers apply (capitalized terms have the meanings set forth in the license). To the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.

To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.

The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.

Written by Kate Stewart and Jason Smith, ELISA Project Medical Devices Working Group Members

The ELISA Project has several working groups with different focuses including Automotive, Linux Features for Safety-Critical Systems, Medical Devices, Open Source Engineering Process, Safety Architecture and Tool Investigation and Code Improvement. The Medical Devices Working Group consists of experts in Linux, medical, and functional safety applications that work together on activities and deliverables intended to help the safe development of medical devices that include Linux-based software.  These activities include, but are not limited to, white papers describing best practices and safety requirements for medical devices using operating systems such as Linux, and conducting safety analyses of open source medical device projects that use Linux such as OpenAPS.

The safety of medical devices is very important, and can be influenced to a great extent by any software that is contained in the medical device.  Failure of software in a medical device can unfortunately cause harm to persons or worse, as demonstrated in the incidents involving the Therac-25 several decades ago.  Therefore, if a medical device is using an operating system such as Linux, the performance and safety of Linux then comes under scrutiny.

In the context of medical device safety standards such as IEC 62304, when Linux is incorporated into a medical device, it is considered to be something called Software of Unknown Provenance (SOUP).  In this case, the medical device manufacturer incorporating Linux into their device did not develop Linux and therefore does not fully know what level of quality processes were used to develop Linux in the first place.  Standards like IEC 62304 allow the usage of SOUP such as Linux; however, IEC 62304 requires that risks associated with the failure of SOUP have been considered and addressed by the manufacturer.

The Medical Devices Working Group is in the process of developing a white paper summarizing requirements from IEC 62304 pertaining to SOUP to assist medical device manufacturers.  If you have experience in Linux, medical, or functional safety applications, the Medical Devices Working Group welcomes your input on this white paper.

One of the interesting challenges with medical devices is that often most of the source for the system is restricted, and not openly available.  This presents a challenge when trying to do analysis on how Linux is being used in such systems.   

The OpenAPS project is a hobbyist project to create a feedback system between an insulin pump and glucose monitor to aid the Type 1 diabetes users to build systems to help manage their blood glucose levels.  That the project is open source means that we can see the code and have a starting point for analysis. 

The Medical Devices Working Group has been using System Theoretic Process Analysis (STPA) to analyze the system, which they call a “rig”, and the Linux system interactions within it.  A rig consists of the Raspberry PI (running Linux and algorithms), glucose monitor (commercial) , insulin pump (commercial), and some data logging device.  How to set up a rig and use it is documented by the OpenAPS project, which has significantly aided our analysis. 

At this point,  we’ve applied the STPA analysis through a couple of levels and have iterated on the analysis a few times (STPA process helped us identify some factors we’d not considered in diagraming the system initially). The team is now working on collecting traces of the system interacting with the Linux kernel.   Tracing will let us continue to take the STPA analysis into the kernel subsystems.    

We are interested in learning of other open source projects using Linux in the context of a medical device.   If you know of such a project, or are interested in working with our team of volunteers,  please feel free to reach out at medical-devices@lists.elisa.tech.

Written by Eli Gurvitz,  ELISA Project Ambassador and Functional Safety Architect at Intel (Mobileye)

In a functional safety system FFI is required when the system consists of elements of different Safety Integrity Levels (ASIL).This is to ensure that elements allocated with a lower ASIL do not interfere with elements allocated with a higher ASIL; if FFI cannot be demonstrated the lower ASIL elements must be upgraded to the higher ASIL.

The Architecture Working Group has been discussing “Freedom From Interference (FFI)” in the last several meetings and is considering two aspects:

• FFI between user space processes allocated with different ASIL
• FFI between Linux Kernel components/drivers/subsystems allocated with different ASIL

This blog post focuses on the second bullet.

FFI is a key goal of a possible Safety Concept for Linux because Linux is too complex and has too many features, thus considering Linux as a single element of a certain ASIL would result in a very high functional safety qualification effort. If the application runs in a single threaded process and handles interrupts synchronously, then it may be possible to avoid allocating Safety Requirements to the OS and mitigate all failures with application-level safety mechanisms. But this kind of use requires just a simple OS and Linux is an overkill. Using Linux in the way it was meant to be used means it will be the OS of a multi-core SoC that runs many processes with different requirements of different ASILs.

This mode of use is referred to in ISO 26262 part 6 section 7.4.8:

This section refers to ISO 26262 part 9 Clause 6 “Criteria for co-existence of elements”. This clause states:

The Architecture WG investigation considers the Linux kernel partitioned into  sub-elements of mixed criticality, therefore the goal is to show FFI between the sub-elements. The approach to FFI that is currently being discussed in the Architecture WG was developed by ELISA Project members Mobileye and BMW.  

The first step in demonstrating FFI between safety-related and non-safety-related sub-elements is to identify the sub-elments and to allocate them with an ASIL. Since we are analyzing a SW component, the sub-elements are functional areas (or features) of the kernel, e.g. memory management or file systems, and they are made of C language functions. We classify the C functions according to the allocated ASIL by using the Call Tree Tool.

The goal of Call Tree is to statically generate the tree of function calls departing from a specified input one; hence starting for example from a syscall, Call Tree would generate the tree of all invoked functions. Call Tree scans the Linux source code by using the GNU CFlow utility and generates an SQLite database that contains all functions and their calling relations – this provides an almost full call-tree for every C function. 

To classify every Kernel function we allocate Kernel entrypoints (syscalls and interrupt handlers) with safety requirements and associated ASIL; hence every function falling in a certain tree inherits the ASIL associated with the top level entrypoint. If a function is present in multiple trees, it is then assigned with the highest ASIL across those allocated to the different trees.

For example, if there’s a safety requirement for “safe dynamic memory” then we consider the related system calls – mmap, sbrk – as safety related. The union of all functions in the call trees of mmap and sbrk are considered SR and inherit the ASIL allocated to mmap and sbrk.

Once we have partitioned the Kernel the next step is to consider the possible types of interference. These types are defined in Annex D or ISO 26262 part 6. There are three types of interference:

• Temporal – interference related to time or scheduling. The most common case is when one kernel thread prevents other threads from getting CPU cycles, thereby causing delays. Another example is a process crashing.
• Spatial – interference related to space, or memory. For example, a lower ASIL driver  may corrupt a kernel data structure.
• Communication – normally this type of interference relates to transfer of data between two entities over a communication channel. In our analysis we consider static and global variables and pointers as communication channels between sub-elements of the kernel.

The Architecture Working Group plans to deal with all types of interference and currently we are considering the third type – communication interference. We are looking at areas where the internal state of the kernel can be corrupted because of the interaction between NSR and SR C functions (or more generally, C functions of different ASIL ratings). 

The internal state of the kernel consists of many persistent data structures. These data structures, for example linked lists, are pointed to by global and static variables and pointers. Corruption of these data structures can occur in different ways.

Data structures that are accessed via global variables can be corrupted when a lower ASIL function (for example a driver that is rated as ASIL QM) accesses the same data structure that is also used by an ASIL-B function, as depicted in the diagram below. 

Corruption of data structures that are accessed via static variables can occur when a static variable is used by a higher ASIL (or SR) function but this function is used by a lower ASIL (or NSR) function. The NSR function may pass a faulty argument to the SR function and the SR function may use this argument to modify the data structure. The faulty data structure is later used in a safety-related flow. This failure mode is depicted in the diagram below.

This description is only a preliminary formulation of the concept of communication interference within the kernel. The working group is debating the correct use of terms, the concept itself, the correct use of the Call Tree Tool and the selection of ASIL rated system calls for our sample automotive use case – The Tell-Tale signal.

If you are interested in safety engineering, the Linux kernel, or both, then please join us in these discussions. The nice thing about applying the existing Functional Safety standards to the Linux kernel is that there’s plenty of space and freedom for creativity, as these standards were designed for much simpler HW and very much simpler SW. It is as if there’s a written tradition of Safety architecture – the ISO 26262 standard and an Oral interpretation of it which creates a more modern tradition of Safety. You can be a part of creating this tradition. I should also take back the word “creativity” I used four lines above because it will certainly trigger a hot debate around the question of whether Safety likes “creativity” or hates it. So I’ll clarify that we are trying to be creative in a conservative way.

Learn more about the ELISA Architecture Working Group or any of the other groups in this white paper.