Written by Elana Copperman, ELISA project ambassador, Chair of the Linux Features for Safety-Critical Systems Working Group and System Safety Architect at Mobileye (Intel)
The Linux Features for Safety-Critical Systems (LFSCS) WG aims to feed into the OSEP and other WGs, working together as a team. LFSCS invites engineers, architects and integrators who actually develop and deploy Linux-based safety-critical systems to contribute from their practical experience and knowledge. In particular, to identify existing Linux kernel features that may be leveraged for use in safety-critical systems.
For example:
Mechanisms for protection of various memory types; e.g. protection from faults due to uninitialized variables or stack overflow.
Dynamic analysis for multi-threaded systems; e.g. tests based on tools such as TSAN or ASAN.
Kernel profiling using ebpf-based tools; e.g. perf-tools or bpftrace
AER (Advanced Error Reporting) for fault handling; e.g. PCIe fault handling
Safety extensions to Linux drivers; e.g. fault handling support and bridging the gap between hardware-based safety features and application layer fault handling.
The WG mailing list is open to registration here, and we are seeing an amazing group of contributors who can demonstrate use of such features in real systems, and help ELISA to learn from these experiences. Initially, we will investigate existing features but will also propose enhancements to such features and to work as a community to design / implement / deploy kernel patches. The goal of such patches will be to help make those features more amenable for use in safety critical systems. Our Github playground is here.
The alliance with ELISA, and with the new Open Source Engineering Process Working Group in particular, is a critical aspect of this effort. We will be working together to help ensure that those patches and features can be used by designers and integrators producing safety critical systems.
The scope of this WG does not include safety qualification or any safety claims on how the integrator can or should use these features or patches. The only claims that would be made are a description of the feature and its functional impact.
The WG will be formally kicked off at the upcoming ELISA workshop (November 8-10). We will be giving an overview of the working group and answer any questions on November 8 at 3 pm CET. We will be scheduling weekly meetings following the workshop. If the technical challenge of enabling real change in deploying open source Linux-based software for safety critical systems excites you, come join and help us meet the challenges!
You can still register for the Fall workshop, which is being held virtually and is free to attend. All registrants will be able to watch the sessions on-demand. Register here today!
Written by Paul Albertella, Chair of the ELISA Project Open Source Engineering Process Working Group
The ELISA Project’s new Open-Source Engineering Process (OSEP) Working Group focuses on the role of engineering processes in creating safety-related systems based on Linux and other FOSS.
Engineering processes are very important in safety, because we rely heavily on them to provide confidence in a system and its components. We achieve confidence by undertaking risk analysis to identify how harm may result from the use (or misuse) of the system, and then constructing a safety argument, which describes how these risks are managed.
When we apply this approach to a specific element of the system, such as a software component like Linux, the argument can be broken down into a number of claims that we want to make regarding that element and its role in the safety of our system. Some claims will relate to the functional responsibilities that the element has in the system; others will relate to the processes that we use to create and refine it.
Importantly, we also need to produce evidence to support these claims. Almost all of this evidence will be produced by an engineering process; some of it will be evidence relating to those processes themselves..
Safety standards like ISO 26262 and IEC 61508 describe reference processes that can act as a template for safety arguments like this. They identify the engineering practices that are seen be necessary (e.g. code review, verification through software testing), the formal processes that are used to control these (e.g. verification management), and the evidence needed to confirm that these have been applied (e.g. test plans, test results).
These reference processes are based on the V-model, which emphasises the formal specification of requirements, architecture and design, and the ability to trace formal verification processes back to these. For software, the standards focus on the processes used when developing new components for a safety-related system, although they include some guidance on applying the principles to pre-existing components, such as software libraries.
Open source projects like Linux have their own development processes, which may be sophisticated and make use of sound software engineering practices. However, it is difficult to map these directly to the reference processes described by the safety standards, because open source development models have very different goals and organizational models, which tend to emphasize refinement by rapid iteration, peer review and community contribution.
In order to address this, OSEP aims to identify and evaluate practices, processes and tools that FOSS developers, system integrators and product creators can use to bridge this gap. We plan to accomplish this by:
Selecting Linux topics and safety-related claims that we want to make about them
Identifying and evaluating practices, processes and tools to answer:
What risks are associated with the topic and claims?
To what extent are these risks addressed or mitigated by (or for) Linux?
How can we manage risks that are not sufficiently addressed or mitigated?
How can we show evidence to support our claims?
Collaborating with other WGs for technical investigations
Documenting and sharing our results as we go
If you would like to learn more about OSEP, join us for an overview presentation on November 8 at 3 pm CET at the ELISA Workshop. The Fall workshop, being held virtually on November 8-10, is free to attend and all registrants will be able to watch the sessions on-demand. Register here.
If you would like to contribute to OSEP, please join the mailing list here, where you can also find details of weekly meetings on the working group calendar.
Written by Elana Copperman, ELISA project ambassador and System Safety Architect at Mobileye (Intel)
This blog has been updated with the video from the Linux Security Summit (LSS), which took place on September 29-October 1.
Are you attending the upcoming Embedded Linux Conference (ELC) on September 27-30 or the Linux Security Summit (LSS) on September 29-October 1? This year, attendees have the option of joining the conference on-site in Seattle, Washington or virtually from their homes and workplaces.
Security and Safety have common goals, yet often follow divergent development paths. We will take a look at various Linux features which were originally designed for security, investigating if/how these features may be relevant to enable safety critical applications.
For example, we’ll discuss:
Memory protection features
Isolation techniques and FFI (Freedom From Interference)
Timing and execution
ebpf and profiling
Safety extensions to Linux drivers
I will present practical implications – focusing on where security and safety meet and where they don’t meet. The presentation, which is intended for experienced software developers and architects, will focus on how these features may be used in real systems. The goal is to spark discussion on how safety mechanisms may be designed in Linux-based safety critical systems, by learning from solutions in the security domain. Watch the video below or check out the presentation here.
Click here to register for the Linux Security Summit or here to learn more about the conference.
Written by Shuah Khan, Chair of the ELISA Project Technical Steering Committee
Please join me in welcoming Jason Smith, Paul Albertella and Philipp Ahmann to the ELISA TSC. They have made significant contributions to the ELISA project and their addition will strengthen the TSC and help us continue to make progress with our mission.
A brief summary of their background and contributions are as follows:
Jason Smith
Jason Smith, the Principal Engineer for Robotics and Control Systems Consumer Technology at UL LLC., has:
– Contributed a Linux in Basic Safety White paper to the project
– Participates in the Medical Devices Working Group regularly
– Contributed the analysis of 62304 SOUP
– Speaks frequently at ELISA Workshops and is an ELISA ambassador
Paul Albertella
Paul Albertella, Consultant at Codethink, has:
– Participated in ELISA since the inception.
– Presented technical content at the last two ELISA workshops
– Contributes in the TSC and other working groups, where he’s been a constructive collaborator
– 20+ years of software engineering experience and understands the open source ecosystem challenges
Philipp Ahmann
Philipp Ahmann, a manager at ADIT (a joint venture of Robert Bosch GmbH and DENSO Corporation) has:
– Participated in ELISA since the start of the project
– Took the lead in helping the project to improve communication and set up our LinkedIn presence
– Volunteered to be an active ambassador on behalf of the project
– Participates in the TSC and other working groups, where he’s known to be a very constructive contributor with an excellent overview across all the different areas we’re working on
– Demonstrates a helpful attitude by being willing to step in and host meetings when the chair is not available
– He is able to actively listen, and helps bring focus to the key elements we need for the project
I look forward to collaborating more closely with all three of our new TSC members. As a reminder, all are welcome to join the bi-weekly public technical community meeting and to contribute your perspectives. You can find the meeting details and subscribe to the calendar here: https://lists.elisa.tech/g/devel/calendar.
Written by EliGurvitz, ELISA Project Ambassador and Functional Safety Architect at Intel (Mobileye)
In a functional safety system FFI is required when the system consists of elements of different Safety Integrity Levels (ASIL).This is to ensure that elements allocated with a lower ASIL do not interfere with elements allocated with a higher ASIL; if FFI cannot be demonstrated the lower ASIL elements must be upgraded to the higher ASIL.
The Architecture Working Group has been discussing “Freedom From Interference (FFI)” in the last several meetings and is considering two aspects:
FFI between user space processes allocated with different ASIL
FFI between Linux Kernel components/drivers/subsystems allocated with different ASIL
This blog post focuses on the second bullet.
FFI is a key goal of a possible Safety Concept for Linux because Linux is too complex and has too many features, thus considering Linux as a single element of a certain ASIL would result in a very high functional safety qualification effort. If the application runs in a single threaded process and handles interrupts synchronously, then it may be possible to avoid allocating Safety Requirements to the OS and mitigate all failures with application-level safety mechanisms. But this kind of use requires just a simple OS and Linux is an overkill. Using Linux in the way it was meant to be used means it will be the OS of a multi-core SoC that runs many processes with different requirements of different ASILs.
This mode of use is referred to in ISO 26262 part 6 section 7.4.8:
This section refers to ISO 26262 part 9 Clause 6 “Criteria for co-existence of elements”. This clause states:
The Architecture WG investigation considers the Linux kernel partitioned into sub-elements of mixed criticality, therefore the goal is to show FFI between the sub-elements. The approach to FFI that is currently being discussed in the Architecture WG was developed by ELISA Project members Mobileye and BMW.
The first step in demonstrating FFI between safety-related and non-safety-related sub-elements is to identify the sub-elments and to allocate them with an ASIL. Since we are analyzing a SW component, the sub-elements are functional areas (or features) of the kernel, e.g. memory management or file systems, and they are made of C language functions. We classify the C functions according to the allocated ASIL by using the Call Tree Tool.
The goal of Call Tree is to statically generate the tree of function calls departing from a specified input one; hence starting for example from a syscall, Call Tree would generate the tree of all invoked functions. Call Tree scans the Linux source code by using the GNU CFlow utility and generates an SQLite database that contains all functions and their calling relations – this provides an almost full call-tree for every C function.
To classify every Kernel function we allocate Kernel entrypoints (syscalls and interrupt handlers) with safety requirements and associated ASIL; hence every function falling in a certain tree inherits the ASIL associated with the top level entrypoint. If a function is present in multiple trees, it is then assigned with the highest ASIL across those allocated to the different trees.
For example, if there’s a safety requirement for “safe dynamic memory” then we consider the related system calls – mmap, sbrk – as safety related. The union of all functions in the call trees of mmap and sbrk are considered SR and inherit the ASIL allocated to mmap and sbrk.
Once we have partitioned the Kernel the next step is to consider the possible types of interference. These types are defined in Annex D or ISO 26262 part 6. There are three types of interference:
Temporal – interference related to time or scheduling. The most common case is when one kernel thread prevents other threads from getting CPU cycles, thereby causing delays. Another example is a process crashing.
Spatial – interference related to space, or memory. For example, a lower ASIL driver may corrupt a kernel data structure.
Communication – normally this type of interference relates to transfer of data between two entities over a communication channel. In our analysis we consider static and global variables and pointers as communication channels between sub-elements of the kernel.
The Architecture Working Group plans to deal with all types of interference and currently we are considering the third type – communication interference. We are looking at areas where the internal state of the kernel can be corrupted because of the interaction between NSR and SR C functions (or more generally, C functions of different ASIL ratings).
The internal state of the kernel consists of many persistent data structures. These data structures, for example linked lists, are pointed to by global and static variables and pointers. Corruption of these data structures can occur in different ways.
Data structures that are accessed via global variables can be corrupted when a lower ASIL function (for example a driver that is rated as ASIL QM) accesses the same data structure that is also used by an ASIL-B function, as depicted in the diagram below.
Corruption of data structures that are accessed via static variables can occur when a static variable is used by a higher ASIL (or SR) function but this function is used by a lower ASIL (or NSR) function. The NSR function may pass a faulty argument to the SR function and the SR function may use this argument to modify the data structure. The faulty data structure is later used in a safety-related flow. This failure mode is depicted in the diagram below.
This description is only a preliminary formulation of the concept of communication interference within the kernel. The working group is debating the correct use of terms, the concept itself, the correct use of the Call Tree Tool and the selection of ASIL rated system calls for our sample automotive use case – The Tell-Tale signal.
If you are interested in safety engineering, the Linux kernel, or both, then please join us in these discussions. The nice thing about applying the existing Functional Safety standards to the Linux kernel is that there’s plenty of space and freedom for creativity, as these standards were designed for much simpler HW and very much simpler SW. It is as if there’s a written tradition of Safety architecture – the ISO 26262 standard and an Oral interpretation of it which creates a more modern tradition of Safety. You can be a part of creating this tradition. I should also take back the word “creativity” I used four lines above because it will certainly trigger a hot debate around the question of whether Safety likes “creativity” or hates it. So I’ll clarify that we are trying to be creative in a conservative way.
Learn more about the ELISA Architecture Working Group or any of the other groups in this white paper.
In May, the ELISA Project hosted its 7th Workshop with 239 participants from 37 different countries. For a complete recap of the workshop, click here. Today, we’ll take a look at one of the sessions titled “A Guided Tour Through the PREEMPT RT castle” presented by Thomas Gleixner, CTO at Linutronix GmbH.
The tour through the inner workings of PREEMPT_RT will start at the observation deck to give an conceptual overview. From there it will take the participants through the various chambers which contain a broad range of historic and contemporary operating system technologies. The tour will not only take the hallways it is also going to explore some of the secret passages and the brave-hearted can take a glimpse at the horror cabinets.
General knowledge about operating system concepts is recommended for taking the tour, but of course it’s open for everyone and all chambers have exit doors if it gets too spooky.
In May, the ELISA Project hosted its 7th Workshop with 239 participants from 37 different countries. For a complete recap of the workshop, click here. Today, we’ll take a look at one of the sessions titled “ELISA and AUTOSAR Adaptive Brainstorm” led by Philipp Ahmann, ADIT GmbH, Huzaifa Saadat, Autosar Experts & Wolfgang Schramm, Continental AG.
AUTOSAR (AUTomotive Open System ARchitecture) is a worldwide development partnership of vehicle manufacturers, suppliers, service providers and companies from the automotive electronics, semiconductor and software industry. The ELISA Project started working more closely with AUTOSAR and Automotive Grade Linux (AGL) and now has a Working Group dedicated to this collaboration.
The AUTOSAR Consortium Working Group Safety works closely with the ELISA Project Automotive Working Group. Previously, the group collaborated to figure out which requirements are put to the system and if assumptions from the Automotive WG can fit with AAUTOSAR expectations. The discussions with AUTOSTAR should harden the work within the ELISA Automotive WG. In this workshop video, Philipp Ahmann, Huzaifa and Wolfgang discuss the work within the ELISA Project, collaboration and host a brainstorm about next steps and more. Watch it here:
Click here learn more about the ELISA Project, here for the Working Groups and here to join our mailing list.
In May, the ELISA Project hosted its 7th Workshop with 239 participants from 37 different countries. For a complete recap of the workshop, click here. Today, we’ll take a look at one of the sessions led by Daniel Bristot de Oliveira, Principal Software Engineer at Red Hat, and Gabriele Paoloni, Senior Software Architect at Intel, titled “A Maintainable and Scalable Kernel Qualification Approach for Automotive.”
In this presentation, Daniel and Gabriele present an innovative approach that aims for the Functional Safety qualification of Linux by leveraging both ISO26262-6 and ISO26262-8.12.
The goal of this presentation was to get feedback on the approach suitability from a Functional Safety point of view, the approach usability in the current Linux mainline development flow and discuss about the next steps.
If you would like to learn more, Daniel and Gabriele will be giving an updated presentation at Embedded Linux Conference on September 28. Learn more about the session and other ELISA talks here.
In May, the ELISA Project hosted its 7th Workshop with 239 participants from 37 different countries. For a complete recap of the workshop, click here. Today, we’ll take a look at one of the sessions led by Jason Smith, Principal Engineer at UL LLC, about the “Linux in Basic Safety Applications.”
Linux is more often being used in applications with safety relevance:
Complex safety-related functions necessitating the advantages of an OS, or
Complex end application necessitating the advantages of an OS, now being asked to perform one or more basic safety-related functions (for example: voltage, current, temperature monitoring)
In both cases, software used to implement the safety-related functions may be required to conform/comply with applicable functional safety standards.
In this presentation, Jason will discuss linux in basic safety applications, the goals and progress of the white paper the ELISA Project is working on and details about how to get involved.
Click here learn more about the ELISA Project, here for the Working Groups and here to join our mailing list.
Written by Philipp Ahmann, ELISA Project Ambassador and Manager at ADIT
Where it all started – The automotive WG
The ELISA Project was launched two years ago by the Linux Foundation. We had our first workshop in person at the BMW training center (Munich, Germany) and the majority of participants with automotive focuses were screaming, “Enable Linux in safety application within the car!” But what happened then?
Since then, the following workshops as well as our weekly meetings, had a strong focus on automotive use cases. There were a lot of participants and a lot of interest but not a lot of volunteers to help with tasks. We kept receiving requests from Toyota, Suzuki, BMW and Automotive Grade Linux (AGL)… In response to this, the Automotive Working Group was established a little more than a year after the launch of the ELISA Project.
From the beginning, while looking for datasheets, reference designs, documentation, and technical concepts, the words “NDA” and “IP” are something we always have in our minds. As a result, we approached the work cautiously as a group:
Concentrated on what ISO26262 showcased about functional safety;
Focused our work with a simulation that is open for everybody;
Stopped saying “could and should” and started using practical examples; and
Pause lengthy discussions about problems that are not Linux specific.
Gaining momentum – The telltale use case
Following these principles, the Automotive Working Group started making progress. We got a good mixture of safety expertise, Linux know-how and automotive backgrounds. We also frequently talk about new things with the curiosity and questioning mindset of a child, which has helped us create a healthy learning environment that is engaging and productive.
Due to Suzuki’s and AGL’s introduced use case, we decided to concentrate on the enablement of telltales (often referred to also as tell-tale) based on a Linux instrument cluster. Thanks to AGL a demo and some high-level ideas were already available.
As we continued our momentum as a group, we recognized that we were spreading our key learnings around in different formats – a bit of source code in a git, diagrams in PlantUML, PowerPoint, or other tools. Documentation was spread over presentations and google docs, so it was hard to create materials and engage interested participants outside the working group. We were determined to continue our momentum and began leveraging tools that would enable others to reproduce and understand our work.
Public means public – The tools
Functional safety projects typically have a very limited set of tools used in the development flow, which have run through a tool qualification. This is expensive because of the license fees and proprietary tools. Putting everything in plain text is good version control and a good baseline, which is key. But monolithic documents make it hard to maintain relationships and traceability – you may even find yourself lost in long text passages.
To make documentation reviews easier and put them under proper version control, we changed from initial sketches in google docs to documentation in GitHub. While also taking requirements in GitHub, we saw they are hard to maintain, put in the relationship and maintain traceability. So the transition was done to maintain them in Freeplane with a plugin developed by Jochen Kall, who is the Automotive WG lead. This plugin also includes e.g. an export script that renders requirements in markdown. Also, the ReqIF exporter is under preparation.
Similar to text, we also had architectural diagrams that the working group converted. We worked to take initial sketches in slide decks and presentations into a storable format. In this case, PlantUML was efficient and easy for us to use.
After this, we recognized that the use case designs end up in the same issue – no relationship between elements within the single PlantUML diagrams, so it was time to change the tool again. The OSS tool we use now is Papyrus based on Eclipse. The files are stored in XML format and in this way can also be put under proper version control.
In the end, all of this hard work has led us to a steady set of tools:
Github for all source code and documentation;
Freeplane to maintain requirements (storable in version control and exportable to text also stored in version control); and
Papyrus for Eclipse.
We are aware that our tools currently used will not survive a safety assessment out of the box, but this is not our intention. The generated artifacts should be shareable so that they can be re-used by others in their established infrastructure. Also, we are targeting to enable others to build safe Linux-based systems and follow the development process for safety integrity standards accordingly. However, in the end, our telltale example will remain an example. A fully qualified product is out of the scope of the ELISA project.
What’s next
So, here we are. Out of creativity and storming team spirit, we settle and start to standardize the tools we use. Version control, review, traceability became major elements of our work.
The practical demo provided by AGL was enhanced to serve the fundamental demands of the telltale use case with a watchdog and a safety app as a codebase. The build can be reproduced with the help of a docker image and the binary can run on qemu.
We still have a long way to go but our goals for the next quarter are:
The use case will be benchmarked against Autosar Adaptive safety requirements and its demands on the operating system; and
Documentation needs to reach a draft state good enough to share with an external audience and to stand critical questions.
The existing Kernel config will be cleaned up towards a slim config (by throwing out unused things) and feedback on our changes to AGL
To learn more about the Automotive Working Group, please subscribe to the mailing list, join our weekly calls and become an active member. Never underestimate what you can achieve with a group. We are happy to welcome additional contributors – get ready to get your hands dirty and have fun with a passionate group of people.
