Skip to main content
Category

Working Group

ELISA Welcomes 3 Mentees!

By Blog, Mentorship, Working Group

The Linux Foundation has had a robust mentorship program for years that invests in new talent and diversity that helps the open source community – no matter what the focus or project – thrive as a whole. Since its formal launch in 2019, the LFX Mentorship has graduated more than 190 mentees and has hosted almost 100 mentorship programs. 

This Spring, the ELISA Project is hosting two mentorships that will help developers gain real-world knowledge in a hands-on learning experience with Linux and open source. It also provides a more defined path for ELISA to connect with the next generation to inject more talent into their developer base.

The Spring Mentorship session, which kicked off in March, paired mentees with leaders from Codethink, the Linux Foundation and Mobileye. The ELISA Project is excited to welcome  Irenge Jules Bashizi, Shefali Sharma and Wenhui Zhang as the newest mentees in the ELISA community. Please see below for more details about their mentorships and mentors. As they settle into their new roles, we hope to feature their mentorship journey in upcoming blog posts. 

Mentorship: Analysis of eBPF (extended Berkley Packet Filter) Verifier

To make eBPF programs “safe”, the Linux kernel validates all eBPF code before loading. However, the current validator has many known limitations, leading to rejection of working programs. 

Focus in this mentorship will be: 

  • In-depth analysis and review of the eBPF validator, and its use to validate eBPF programs.
  • Code enhancements to the validator to improve usability.
  • Identify use cases for kernel profiling in safety critical applications.
Elana Copperman

Mentor: Elana Copperman, Chair of the Linux Features for Safety-Critical Systems Working Group and System Safety Architect at Mobileye (part of Intel)

Elana provides support for designing safety features in Mobileye products, including system boot; drivers; and Linux infrastructure. Before working at Mobileye, she worked as a Security Architect for Cisco-Il (formerly NDS) and more recently as a security consultant for major European automotive concerns on behalf of various Israeli startups. Research interests focus on software engineering methodologies and security engineering.

In particular, focusing on expanding open source and Linux-based tools to support safety critical and life saving product development.

Irenge Jules Bashizi

Mentee: Irenge Jules Bashizi

Jules is a Computer science student at University of Manchester. He is a certified Linux System administrator.  Jules is interested in improving his skills in Kernel engineering by contributing to the Linux Kernel community by submitting patches. This internship offers him a unique opportunity tailored to improve and contribute.  As a hobby, Jules enjoys jogging..

Mentee: Wenhui Zhang

Mentorship: Discovering Linux kernel subsystems used by OpenAPS

OpenAPS is an open source Artificial Pancreas System designed to automatically adjust an insulin pump’s insulin delivery to keep Blood Glucose in a safe range at all times. It is an open and transparent effort to make safe and effective basic Automatic Pancreas System technology widely available to anyone with compatible medical devices who is willing to build their own system.

What happens when an OpenAPS workload runs on Linux? What are the subsystems and modules that are in active use when OpenAPS is running? What are the interactions between OpenAPS and the kernel when a user checks how much insulin is left in the insulin pump?

The ELISA Medical Devices Working Group set out to answer these questions. Understanding the kernel footprint necessary to run a workload helps us focus on the  subsystem and modules that make up the footprint for safety.

The mentee will:

  • Use Linux kernel tracing and strace tool to discover Linux kernel subsystems used by OpenAPS. 
  • Find Linux system calls supported on various architectures. 
  • Write a blog/whitepaper on the findings which will aid ELISA Medical Devices WG to focus on the  subsystem and modules that make up the footprint for safety.

Shefali Sharma has started working on the project to advance the work Shuah and Milna have shared in their recent blog here

Mentor: Milan Lakhani, Co-Chair of the Medical Devices Working Group and Systems and Software Engineer at Codethink

In open source, Milan’s contributions to Linux kernel are aimed at achieving ELISA project goals. Other than that, he has previously worked in the Trustable and community – mainly STPA analysis on design and writing requirements and tests and also some patches to help with making a webapp and porting. 

There are a lot of aspects and opportunities to really learn through experience and take responsibility to make an impact on a highly approved, tested and growing Closed Loop Open-Source insulin delivery system that is really helping to reduce issues of people with type 1 diabetes. There should also be some variety in the tasks and the approach that the mentee can do. Milan is excited to share his skills and knowledge with STPA (our method of safety analysis for the system), the OpenAPS system and codebase (OpenAPS is the medical device itself) and Linux kernel.

Shuah Khan

Mentor: Shuah Khan, ELISA Project TSC Chair and Linux Foundation Fellow

Shuah is an experienced Linux Kernel developer, maintainer, and contributor. She has extensive experience in open source development, actively working across Linux Kernel sub-systems.

She currently maintains the Kernel Selftest, USB over IP, and cpupower tools. She is an active contributor to the Linux media sub-system.

Shuah has a passion for mentoring and educating the next generation. She loves mentoring and training engineers new to open source and helping them become committers and reviewers.

Shefali Sharma

Mentee: Shefali Sharma

Shefali is a third year Computer Science Engineering student from Meerut Institute of Engineering and Technology, Meerut, India. She likes to explore new technical domains. She is very excited to work on the OpenAPS project as it will give her an opportunity to use her technical skills for the welfare of others and to get involved in the Linux kernel community. Apart from this she is also interested in DevOps and Machine Learning.

Established Working Group Updates

By Blog, Working Group, Workshop

Planning is currently underway for the ELISA Project Spring Workshop, which takes place virtually on April 5-7. If you haven’t yet, you can submit a CFP here (by Friday, March 4) or register to attend here.

As we prepare for the next workshop, we’ll be taking a look at the most popular sessions from the November event. A full recap by Philipp Ahmann, ELISA Project Ambassador and TSC member can be found here.

In this video, Shuah Khan, ELISA Project Chair of the Technical Steering Committee, kicks off the November Workshop with an overview of the TSC and introductions to a few of the Working Group Chairs for updates. Watch the video to learn more about the focused working groups for Safety Architecture, Tool Investigation and Code Improvement, Medical Devices and Automotive.

To join a working group click here: https://elisa.tech/community/working-groups/.

To attend the April 2022 Workshop, register here: https://events.linuxfoundation.org/elisa-workshop-spring/register/.

Discovery Linux Kernel Subsystems used by OpenAPS

By Blog, Working Group

Written by Shuah Khan, Chair of the ELISA Project Technical Steering Committee, and Milan Lakhani, member of the ELISA Medical Devices Working Group

Key Points

  • Understanding system resources necessary to build and run a workload is important.
  • Linux tracing and strace can be used to discover the system resources in use by a workload.
  • Once we discover and understand the workload needs, we can focus on them to avoid regressions and evaluate safety.

OpenAPS is an open source Artificial Pancreas System designed to automatically adjust an insulin pump’s insulin delivery to keep Blood Glucose in a safe range at all times.

It is an open and transparent effort to make safe and effective basic Automatic Pancreas System technology widely available to anyone with compatible medical devices who is willing to build their own system.

Broadly speaking, the OpenAPS system can be thought of performing 3 main functions. Monitoring the environment and operational status of devices with as much data relevant to therapy as possible collected, predicting what should happen to glucose levels next, and enacting changes through issuing commands, emails and even phone calls.

ELISA Medical Devices WG team has set out to discover the Linux kernel subsystems used by OpenAPS. Understanding the kernel footprint necessary to run a workload helps us focus on the  subsystem and modules that make up the footprint for safety. We set out to answer the following questions:

  • What happens when an OpenAPS workload runs on Linux?
  • What are the subsystems and modules that are in active use when OpenAPS is running?
  • What are the interactions between OpenAPS and the kernel when a user checks how much insulin is left in the insulin pump?

So how do we discover the Linux kernel subsystem in use? The Linux kernel has several features and tools that can help discover which modules and functions are being used by an application during run-time. Using these tools, we can gather the system state while the OpenAPS workload is running to determine which parts of the kernel are being used.

Let’s talk a bit about kernel states. The kernel system state can be viewed as a combination of static and dynamic features and modules. Let’s first define what static and dynamic system states are and then explore how we can visualize the static and dynamic system parts of the kernel.

Static System View comprises system calls, features, static modules and dynamic modules enabled in the kernel configuration. Supported system calls and Kernel features are architecture dependent. System call numbering is different on different architectures. We can get the supported system call information using the auditd package tool. ausyscall –dump prints out the supported system calls on a system. You can install the auditd package by running “sudo apt-get install auditd” on Debian systems. Linux kernel script scripts/checksyscalls.sh can be used to check if current architecture is missing any function calls compared to i386. scripts/get_feat.pl <get_feat.pl list> can be used to list the Kernel feature support matrix for a system or get_feat.pl list –arch=arm for example lists the Kernel feature support matrix of the ‘arm’ architecture

Dynamic System View comprises system calls, ioctls invoked, and subsystems used during the runtime. A workload could load and unload modules and also change the dynamic system configuration to suit its needs.

OpenAPS Static View

Let’s first look at the OpenAPS sources to understand the workload from a static view. The OpenAPS workload is a collection of  python libraries, python-dev, software-properties-common, python-software-properties, python-numpy, python-pip, nodejs-legacy and npm. You can have a look at the OpenAPS repositories in https://github.com/openaps with the two main ones being https://github.com/openaps/oref0 and https://github.com/openaps/openaps . The initial dependencies that the user installs can be seen here https://github.com/openaps/docs/blob/master/scripts/quick-packages.sh 

One easier way to understand its runtime characteristics is watching the system state while a workload is running. We determined that the following methodology and tools would work well for us to observe the system activity. 

What is the methodology?

The first step is gathering the default system state such as the dynamic and static modules loaded on the system. lsmod command prints out the dynamically loaded modules on a system. Statically configured modules can be found in the kernel configuration file. Understanding the default system is necessary to determine the changes if any made by the workload.

The next step is discovering the Linux kernel footprint used by OpenAPS by enabling event tracing before starting the workload, gathering dynamic modules while the workload is running. Once the workload is stopped, gather the event logs, kernel messages.

Once we have the necessary information, we can extract the system call numbers from the event trace log and map them to the supported system calls.

Putting our methodology to test

Our initial plan was to use strace to trace system calls and signals used by OpenAPS commands in strace. strace runs the specified command until it  exits. It intercepts  and  records  the  system  calls  which are called by the process and the signals which are received by the process. It gives insight into the syscalls OpenAPS commands use. However, we realized quickly that OpenAPS employs setup scripts to launch its workload. As a result, using strace was not an option.

We modified OpenAPS oref0-setup.sh in https://github.com/OpenAPS/oref0.git to enable event tracing before OpenAPS starts its workload (processes, shell scripts).  This approach gives us an overall information about OpenAPS. We will develop a higher level view and then dive into individual OpenAPS commands..

==================================================================

diff –git a/bin/oref0-setup.sh b/bin/oref0-setup.sh

index 261da95b..5ae666e2 100755

— a/bin/oref0-setup.sh

+++ b/bin/oref0-setup.sh

@@ -1269,6 +1269,11 @@ if prompt_yn “” N; then

 fi # from ‘read -p “Continue? y/[N] ” -r’ after interactive setup is complete

+# ELISA enable event tracing

+echo “ELISA: Enable event tracing on all events”

+echo 1 > /sys/kernel/debug/tracing/events/enable

+cat /sys/kernel/debug/tracing/events/enable

+

 # Start cron back up in case the user doesn’t decide to reboot service cron start

==================================================================

We were able to gather traces with the above modification to oref0-setup.sh. As mentioned earlier, the ausyscall tool dumps out mapping for syscalls and their corresponding syscall table numbers. The mapping is architecture dependent for some system calls. The trace data includes NR followed by a number in the trace file. These are the syscalls that are run during the time the tracing was on. Using the tracing and system call information, we determined the system calls used by the OpenAPS workload. In addition we used scripts/checksyscalls.sh to check for system call support status on RasPi.

What did we do to gather traces and system state?

  • Start OpenAPS workload with modified the modification to enable tracing
  • Let the workload run for 30 minutes
  • Discard last 5 minutes of trace from analysis to account for interference (rsync and plugging devices) with trace file extraction
  • Stop OpenAPS.
  • Extract trace file from the system – cat /sys/kernel/debug/tracing/trace > trace.out
  • Run lsmod after OpenAPS workload starts to gather module information
  • Run “ausyscall –dump > syscalls_dump.out

Analyzing traces:

  • Map the NR (syscal) numbers from the trace to syscalls from the syscalls dump.
  • Categorize system calls and map them to Linux subsystems.

Findings and observations:

Kernel module usage:

Module NameUsage count  (OpenAPS)Usage count (default)
cmacNot loaded1
ecdh_generic1 (bluetooth)2 (bluetooth)
ecc1 (ecdh_generic)1 (ecdh_generic)
spidev2Not loaded
i2c_bcm28351Not loaded
spi_bcm28350Not loaded
i2c_dev2Not loaded
ipv62624

Subsystem usage:

Subsystem# of calls
kmem_*200990
mm_*182471
sched_*195241
rcu_*223011
irq_*1781503
kmalloc79801
cpu_idle62623
rss_stat22130
ipi_*42514
sys_*148034
vm_*60489
task_*72813
timer_58572
hrtimer_*152271
softirq_*28860
workqueue_*6129
writeback_*3933
ext4_*34461
jbd2_*2062
block_*3590
dwc_otg*27701
arch_timer13446

Updated System view:

Conclusion

This tracing activity was a good way of identifying which parts of the kernel are used by OpenAPS. This helped to generate the Updated system view, so it is useful for our goal to do an STPA analysis of OpenAPS Operating System activity. We plan to theoretically analyse how these different subsystems can interact unsafely, while using fault injection and mocked components to collect more traces.

As mentioned earlier, the approach we used so far gave us a higher level of information about the OpenAPS usage. It isn’t ideal that this higher information doesn’t tell us the system usage by individual OpenAPS commands. As an example, we won’t be able to clearly identify which system calls are invoked when a user queries insulin pump status.

We are in the process of gathering fine grained information about individual OpenAPS commands and important use-cases. As an example, what subsystems are used when a user queries the insulin pump status. We are  using the strace command to trace the OpenAPS commands. We will share our findings in our next blog on this topic.

SPDX-License-Identifier: CC-BY-4.0

This document is released under the Creative Commons Attribution 4.0 International License, available at https://creativecommons.org/licenses/by/4.0/legalcode. Pursuant to Section 5 of the license, please note that the following disclaimers apply (capitalized terms have the meanings set forth in the license). To the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.

To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.

The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.

Analyzing Open Source Interactions in Linux Based Medical Devices

By Blog, Working Group

Written by Kate Stewart and Jason Smith, ELISA Project Medical Devices Working Group Members

The ELISA Project has several working groups with different focuses including Automotive, Linux Features for Safety-Critical Systems, Medical Devices, Open Source Engineering Process, Safety Architecture and Tool Investigation and Code Improvement. The Medical Devices Working Group consists of experts in Linux, medical, and functional safety applications that work together on activities and deliverables intended to help the safe development of medical devices that include Linux-based software.  These activities include, but are not limited to, white papers describing best practices and safety requirements for medical devices using operating systems such as Linux, and conducting safety analyses of open source medical device projects that use Linux such as OpenAPS.

The safety of medical devices is very important, and can be influenced to a great extent by any software that is contained in the medical device.  Failure of software in a medical device can unfortunately cause harm to persons or worse, as demonstrated in the incidents involving the Therac-25 several decades ago.  Therefore, if a medical device is using an operating system such as Linux, the performance and safety of Linux then comes under scrutiny.

In the context of medical device safety standards such as IEC 62304, when Linux is incorporated into a medical device, it is considered to be something called Software of Unknown Provenance (SOUP).  In this case, the medical device manufacturer incorporating Linux into their device did not develop Linux and therefore does not fully know what level of quality processes were used to develop Linux in the first place.  Standards like IEC 62304 allow the usage of SOUP such as Linux; however, IEC 62304 requires that risks associated with the failure of SOUP have been considered and addressed by the manufacturer.

The Medical Devices Working Group is in the process of developing a white paper summarizing requirements from IEC 62304 pertaining to SOUP to assist medical device manufacturers.  If you have experience in Linux, medical, or functional safety applications, the Medical Devices Working Group welcomes your input on this white paper.

One of the interesting challenges with medical devices is that often most of the source for the system is restricted, and not openly available.  This presents a challenge when trying to do analysis on how Linux is being used in such systems.   

The OpenAPS project is a hobbyist project to create a feedback system between an insulin pump and glucose monitor to aid the Type 1 diabetes users to build systems to help manage their blood glucose levels.  That the project is open source means that we can see the code and have a starting point for analysis. 

The Medical Devices Working Group has been using System Theoretic Process Analysis (STPA) to analyze the system, which they call a “rig”, and the Linux system interactions within it.  A rig consists of the Raspberry PI (running Linux and algorithms), glucose monitor (commercial) , insulin pump (commercial), and some data logging device.  How to set up a rig and use it is documented by the OpenAPS project, which has significantly aided our analysis. 

At this point,  we’ve applied the STPA analysis through a couple of levels and have iterated on the analysis a few times (STPA process helped us identify some factors we’d not considered in diagraming the system initially). The team is now working on collecting traces of the system interacting with the Linux kernel.   Tracing will let us continue to take the STPA analysis into the kernel subsystems.    

We are interested in learning of other open source projects using Linux in the context of a medical device.   If you know of such a project, or are interested in working with our team of volunteers,  please feel free to reach out at medical-devices@lists.elisa.tech.

ELISA’s New Linux Features for Safety-Critical Systems Working Group

By Blog, Working Group, Workshop

Written by Elana Copperman, ELISA project ambassador, Chair of the Linux Features for Safety-Critical Systems Working Group and System Safety Architect at Mobileye (Intel)

The Linux Features for Safety-Critical Systems (LFSCS) WG aims to feed into the OSEP and other WGs, working together as a team.  LFSCS invites engineers, architects and integrators who actually develop and deploy Linux-based safety-critical systems to contribute from their practical experience and knowledge.  In particular, to identify existing Linux kernel features that may be leveraged for use in safety-critical systems.  

For example:

  1. Mechanisms for protection of various memory types;  e.g. protection from faults due to uninitialized variables or stack overflow.
  2. Dynamic analysis for multi-threaded systems; e.g. tests based on tools such as TSAN or ASAN.
  3. Kernel profiling using ebpf-based tools; e.g.  perf-tools or bpftrace
  4. AER (Advanced Error Reporting) for fault handling; e.g. PCIe fault handling
  5. Safety extensions to Linux drivers; e.g. fault handling support and bridging the gap between hardware-based safety features and application layer fault handling.

The WG mailing list is open to registration here, and we are seeing an amazing group of contributors who can demonstrate use of such features in real systems, and help ELISA to learn from these experiences.  Initially, we will investigate existing features but will also propose enhancements to such features and to work as a community to design / implement / deploy kernel patches.  The goal of such patches will be to help make those features more amenable for use in safety critical systems.  Our Github playground is here.

The alliance with ELISA, and with the new Open Source Engineering Process Working Group in particular, is a critical aspect of this effort.  We will be working together to help ensure that those patches and features can be used by designers and integrators producing safety critical systems.

 The scope of this WG does not include safety qualification or any safety claims on how the integrator can or should use these features or patches.  The only claims that would be made are a description of the feature and its functional impact.

The WG will be formally kicked off at the upcoming ELISA workshop (November 8-10). We will be giving an overview of the working group and answer any questions on November 8 at 3 pm CET.  We will be scheduling weekly meetings following the workshop.  If the technical challenge of enabling real change in deploying open source Linux-based software for safety critical systems excites you, come join and help us meet the challenges!

You can still register for the Fall workshop, which is being held virtually and is free to attend. All registrants will be able to watch the sessions on-demand. Register here today!

ELISA’s Open Source Engineering Process Working Group

By Blog, Working Group, Workshop

Written by Paul Albertella, Chair of the ELISA Project Open Source Engineering Process Working Group

The ELISA Project’s new Open-Source Engineering Process (OSEP) Working Group focuses on the role of engineering processes in creating safety-related systems based on Linux and other FOSS.

Engineering processes are very important in safety, because we rely heavily on them to provide confidence in a system and its components. We achieve confidence by undertaking risk analysis to identify how harm may result from the use (or misuse) of the system, and then constructing a safety argument, which describes how these risks are managed.

When we apply this approach to a specific element of the system, such as a software component like Linux, the argument can be broken down into a number of claims that we want to make regarding that element and its role in the safety of our system. Some claims will relate to the functional responsibilities that the element has in the system; others will relate to the processes that we use to create and refine it.

Importantly, we also need to produce evidence to support these claims. Almost all of this evidence will be produced by an engineering process; some of it will be evidence relating to those processes themselves..

Safety standards like ISO 26262 and IEC 61508 describe reference processes that can act as a template for safety arguments like this. They identify the engineering practices that are seen be necessary (e.g. code review, verification through software testing), the formal processes that are used to control these (e.g. verification management), and the evidence needed to confirm that these have been applied (e.g. test plans, test results).

These reference processes are based on the V-model, which emphasises the formal specification of requirements, architecture and design, and the ability to trace formal verification processes back to these. For software, the standards focus on the processes used when developing new components for a safety-related system, although they include some guidance on applying the principles to pre-existing components, such as software libraries.

Open source projects like Linux have their own development processes,  which may be   sophisticated and make use of sound software engineering practices. However, it is difficult to map these directly to the reference processes described by the safety standards, because open source development models have very different goals and organizational models, which tend to emphasize refinement by rapid iteration, peer review and community contribution.

In order to address this, OSEP aims to identify and evaluate practices, processes and tools that FOSS developers, system integrators and product creators can use to bridge this gap. We plan to accomplish this by:

  • Selecting Linux topics and safety-related claims that we want to make about them
  • Identifying and evaluating practices, processes and tools to answer:
    • What risks are associated with the topic and claims?
    • To what extent are these risks addressed or mitigated by (or for) Linux?
    • How can we manage risks that are not sufficiently addressed or mitigated?
    • How can we show evidence to support our claims?
  • Collaborating with other WGs for technical investigations
  • Documenting and sharing our results as we go

If you would like to learn more about OSEP, join us for an overview presentation on November 8 at 3 pm CET at the ELISA Workshop. The Fall workshop, being held virtually on November 8-10,  is free to attend and all registrants will be able to watch the sessions on-demand. Register here.

If you would like to contribute to OSEP, please join the mailing list here, where you can also find details of weekly meetings on the working group calendar.

ELISA Working Groups

By Blog, Working Group, Workshop

Since launch in February 2019, the ELISA Project has created several working groups that collaborate and work towards providing resources for System integrators to apply and use to analyze qualitatively and quantitatively on their systems. Current groups include an Automotive Working Group, Medical Devices Working Group, Safety Architecture Working Group and Tool Investigation and Code Improvement Sub-Working Group to focus on specific activities and goals. 

If you’re interested in learning more about the goals and objectives for these working groups or asking questions, we invite you to the ELISA Workshop on November 8-10. The virtual workshop, which is free to attend, will host speakers from Arm, Codethink, Elektrobit Automotive GmbH, Evidence Srl, Google, Intel, Mobileye, The Linux Foundation, Red Hat and UL LLC.

On Monday, November 8 at 5-6 am PDT, the working group chairs will provide updates on all activities. Led by Gabriele Paoloni, Lukas Bulwahn, Kate Stewart, Shuah Khan, Milan Lakhani, Jason Smith, Jochen Kall and Philipp Ahmann, you can add this to your schedule here.

Additionally, we also recently announced two more working groups:

Open Source Engineering Process Working GroupThis working group aims to examine safety-related claims that we might like to make about Linux as part of a system, and to explore how we can gather and present evidence to support such claims.

Linux Features for Safety-Critical Systems Working Group: This working group will work to bring together kernel developers and producers of safety critical systems to demonstrate use of such features in real systems, and to learn from these experiences together as a community.

If you want to learn more about these two new working groups, we invite you to the session on November 8 at 6-630 am PDT lead by Paul Albertella and Elana Copperman. Add this to your schedule here.

To register or to review the complete schedule, click here: https://events.linuxfoundation.org/elisa-workshop/program/schedule/.

The ELISA Project Continues to Grow its Global Ecosystem by Welcoming Red Hat as a Premier Member and Banma, Lotus Cars and SUSE

By Announcement, News, Working Group, Workshop

Schedule for the ELISA Fall Workshop on November 8-10 is now live

SAN FRANCISCO – October 20, 2021 – Today, the ELISA (Enabling Linux in Safety Applications) Project, an open source initiative that aims to create a shared set of tools and processes to help companies build and certify Linux-based safety-critical applications and systems, announced that Red Hat has upgraded its membership to premier member and welcomes Banma, Lotus Cars and SUSE as the newest members.

Linux is used in all major industries because it can enable faster time to market for new features and take advantage of the quality of the code development processes.  Launched in February 2019 by the Linux Foundation, ELISA works with Linux kernel and safety communities to agree on what should be considered when Linux is to  be used in safety-critical systems.

“Linux underpins many applications today that have safety-critical and cybersecurity implications,” said Kate Stewart, Vice President of Dependable Embedded Systems at The Linux Foundation. “By collaborating together, the ELISA members are defining the best practices for use of Linux in these systems. We look forward to continuing to build consensus and welcoming expertise and collaboration from these new members.”

Attend the Fall Workshop

Since its inception, ELISA has hosted quarterly workshops that bring together project members and community contributors to discuss working group updates, trends in functional safety, use cases and more. The next workshop will be held virtually on November 8-10 and is free to attend. Speakers include thought leaders from Arm, Codethink, Elektrobit Automotive GmbH, Evidence Srl, Google, Intel, Mobileye, The Linux Foundation, Red Hat and UL LLC. Register and check out the schedule: https://events.linuxfoundation.org/elisa-workshop/

Join the New Working Groups

Since launch, the project has worked to establish a governance model that creates processes and guidance to the focused working groups that aim to provide resources for System integrators to apply and use to analyze qualitatively and quantitatively on their systems. Today, ELISA announces two new working groups:

  • Open Source Engineering Process Working Group: This working group aims to examine safety-related claims that we might like to make about Linux as part of a system, and to explore how we can gather and present evidence to support such claims.
  • Linux Features for Safety-Critical Systems Working Group: This working group will work to bring together kernel developers and producers of safety critical systems to demonstrate use of such features in real systems, and to learn from these experiences together as a community. Learn more about this new working group in this November Workshop session

Learn more about the Global Ecosystem

Red Hat, which is known for its leadership in linux and open source, joined ELISA earlier this year and has been very active in the technical community. With their upgraded membership to Premier, Red Hat welcomes Gabriele Paoloni, Open Source Community Technical Leader at Red Hat, as the ELISA Project Governing Board Chair.

“Red Hat announced our intent to expand our expertise in Linux to safety-critical automotive use cases earlier this year as we work to develop a Linux in-vehicle operating system,” said Francis Chow, vice president, In-Vehicle Operating System, Red Hat. “As such, we’re pleased to extend our participation in ELISA as a Premier member and collaborate with other industry leaders in building up open source software for applications that require extremely high levels of trust and functional safety. We believe a standardized common set of tools and processes can drive innovation toward the software-defined vehicle. ”

Additionally, ELISA welcomes Banma, a Chinese startup specializing in automotive software;  Lotus Cars, a leader in automotive manufacturing in China; and SUSE, a global leader in open source software specializing in enterprise Linux, Kubernetes management, and edge solutions.  These new members join ADIT, AISIN AW CO., arm, Automotive Grade Linux, BMW Car IT GmbH, Codethink, Elektrobit, Horizon Robotics, Huawei Technologies, Intel, Toyota, Kuka, Linuxtronix. Mentor, NVIDIA, Suzuki, Wind River, OTH Regensburg and Toyota.

“Compared with other open software, safety is the key differentiation of automotive OS”, said Sean Xiao, Chief Architect at Banma. “The mission of Banma is to help automotive makers deliver intelligent cars by offering advanced vehicle open software. The ELISA Project combines safety and linux, which offers flexibility and openness, and closely aligns with our goals.”

“For nearly 30 years, SUSE has been a trusted partner supporting systems and essential workloads in some of the most challenging and critical industries in terms of safety requirements, such as automotive and transportation, government, aerospace and defense, industrial and manufacturing, and healthcare,” said Ivo Totev, SUSE COO. “We already collaborate with current ELISA members on important initiatives and are pleased to join ELISA as a formal member to continue to provide innovation in safety-critical domains.”

For more information about ELISA, visit https://elisa.tech/.

About The Linux Foundation

The Linux Foundation is the organization of choice for the world’s top developers and companies to build ecosystems that accelerate open technology development and commercial adoption. Together with the worldwide open source community, it is solving the hardest technology problems by creating the largest shared technology investment in history. Founded in 2000, The Linux Foundation today provides tools, training and events to scale any open source project, which together deliver an economic impact not achievable by any one company. More information can be found at www.linuxfoundation.org.

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see its trademark usage page: www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.

###

Architecture Working Group: A report on Kernel FFI (Freedom From Interference) and some philosophical musings

By Blog, Working Group

Written by Eli Gurvitz,  ELISA Project Ambassador and Functional Safety Architect at Intel (Mobileye)

In a functional safety system FFI is required when the system consists of elements of different Safety Integrity Levels (ASIL).This is to ensure that elements allocated with a lower ASIL do not interfere with elements allocated with a higher ASIL; if FFI cannot be demonstrated the lower ASIL elements must be upgraded to the higher ASIL.

The Architecture Working Group has been discussing “Freedom From Interference (FFI)” in the last several meetings and is considering two aspects:

  • FFI between user space processes allocated with different ASIL
  • FFI between Linux Kernel components/drivers/subsystems allocated with different ASIL

This blog post focuses on the second bullet.

FFI is a key goal of a possible Safety Concept for Linux because Linux is too complex and has too many features, thus considering Linux as a single element of a certain ASIL would result in a very high functional safety qualification effort. If the application runs in a single threaded process and handles interrupts synchronously, then it may be possible to avoid allocating Safety Requirements to the OS and mitigate all failures with application-level safety mechanisms. But this kind of use requires just a simple OS and Linux is an overkill. Using Linux in the way it was meant to be used means it will be the OS of a multi-core SoC that runs many processes with different requirements of different ASILs.

This mode of use is referred to in ISO 26262 part 6 section 7.4.8:

This section refers to ISO 26262 part 9 Clause 6 “Criteria for co-existence of elements”. This clause states:

The Architecture WG investigation considers the Linux kernel partitioned into  sub-elements of mixed criticality, therefore the goal is to show FFI between the sub-elements. The approach to FFI that is currently being discussed in the Architecture WG was developed by ELISA Project members Mobileye and BMW.  

The first step in demonstrating FFI between safety-related and non-safety-related sub-elements is to identify the sub-elments and to allocate them with an ASIL. Since we are analyzing a SW component, the sub-elements are functional areas (or features) of the kernel, e.g. memory management or file systems, and they are made of C language functions. We classify the C functions according to the allocated ASIL by using the Call Tree Tool.

The goal of Call Tree is to statically generate the tree of function calls departing from a specified input one; hence starting for example from a syscall, Call Tree would generate the tree of all invoked functions. Call Tree scans the Linux source code by using the GNU CFlow utility and generates an SQLite database that contains all functions and their calling relations – this provides an almost full call-tree for every C function. 

To classify every Kernel function we allocate Kernel entrypoints (syscalls and interrupt handlers) with safety requirements and associated ASIL; hence every function falling in a certain tree inherits the ASIL associated with the top level entrypoint. If a function is present in multiple trees, it is then assigned with the highest ASIL across those allocated to the different trees.

For example, if there’s a safety requirement for “safe dynamic memory” then we consider the related system calls – mmap, sbrk – as safety related. The union of all functions in the call trees of mmap and sbrk are considered SR and inherit the ASIL allocated to mmap and sbrk.

Once we have partitioned the Kernel the next step is to consider the possible types of interference. These types are defined in Annex D or ISO 26262 part 6. There are three types of interference:

  • Temporal – interference related to time or scheduling. The most common case is when one kernel thread prevents other threads from getting CPU cycles, thereby causing delays. Another example is a process crashing.
  • Spatial – interference related to space, or memory. For example, a lower ASIL driver  may corrupt a kernel data structure.
  • Communication – normally this type of interference relates to transfer of data between two entities over a communication channel. In our analysis we consider static and global variables and pointers as communication channels between sub-elements of the kernel.

The Architecture Working Group plans to deal with all types of interference and currently we are considering the third type – communication interference. We are looking at areas where the internal state of the kernel can be corrupted because of the interaction between NSR and SR C functions (or more generally, C functions of different ASIL ratings). 

The internal state of the kernel consists of many persistent data structures. These data structures, for example linked lists, are pointed to by global and static variables and pointers. Corruption of these data structures can occur in different ways.

Data structures that are accessed via global variables can be corrupted when a lower ASIL function (for example a driver that is rated as ASIL QM) accesses the same data structure that is also used by an ASIL-B function, as depicted in the diagram below. 

Corruption of data structures that are accessed via static variables can occur when a static variable is used by a higher ASIL (or SR) function but this function is used by a lower ASIL (or NSR) function. The NSR function may pass a faulty argument to the SR function and the SR function may use this argument to modify the data structure. The faulty data structure is later used in a safety-related flow. This failure mode is depicted in the diagram below.

This description is only a preliminary formulation of the concept of communication interference within the kernel. The working group is debating the correct use of terms, the concept itself, the correct use of the Call Tree Tool and the selection of ASIL rated system calls for our sample automotive use case – The Tell-Tale signal.

If you are interested in safety engineering, the Linux kernel, or both, then please join us in these discussions. The nice thing about applying the existing Functional Safety standards to the Linux kernel is that there’s plenty of space and freedom for creativity, as these standards were designed for much simpler HW and very much simpler SW. It is as if there’s a written tradition of Safety architecture – the ISO 26262 standard and an Oral interpretation of it which creates a more modern tradition of Safety. You can be a part of creating this tradition. I should also take back the word “creativity” I used four lines above because it will certainly trigger a hot debate around the question of whether Safety likes “creativity” or hates it. So I’ll clarify that we are trying to be creative in a conservative way.

Learn more about the ELISA Architecture Working Group or any of the other groups in this white paper.