Written by Elana Copperman, ELISA project ambassador, Chair of the Linux Features for Safety-Critical Systems Working Group and System Safety Architect at Mobileye (Intel)
The Linux Features for Safety-Critical Systems (LFSCS) WG aims to feed into the OSEP and other WGs, working together as a team. LFSCS invites engineers, architects and integrators who actually develop and deploy Linux-based safety-critical systems to contribute from their practical experience and knowledge. In particular, to identify existing Linux kernel features that may be leveraged for use in safety-critical systems.
For example:
Mechanisms for protection of various memory types; e.g. protection from faults due to uninitialized variables or stack overflow.
Dynamic analysis for multi-threaded systems; e.g. tests based on tools such as TSAN or ASAN.
Kernel profiling using ebpf-based tools; e.g. perf-tools or bpftrace
AER (Advanced Error Reporting) for fault handling; e.g. PCIe fault handling
Safety extensions to Linux drivers; e.g. fault handling support and bridging the gap between hardware-based safety features and application layer fault handling.
The WG mailing list is open to registration here, and we are seeing an amazing group of contributors who can demonstrate use of such features in real systems, and help ELISA to learn from these experiences. Initially, we will investigate existing features but will also propose enhancements to such features and to work as a community to design / implement / deploy kernel patches. The goal of such patches will be to help make those features more amenable for use in safety critical systems. Our Github playground is here.
The alliance with ELISA, and with the new Open Source Engineering Process Working Group in particular, is a critical aspect of this effort. We will be working together to help ensure that those patches and features can be used by designers and integrators producing safety critical systems.
The scope of this WG does not include safety qualification or any safety claims on how the integrator can or should use these features or patches. The only claims that would be made are a description of the feature and its functional impact.
The WG will be formally kicked off at the upcoming ELISA workshop (November 8-10). We will be giving an overview of the working group and answer any questions on November 8 at 3 pm CET. We will be scheduling weekly meetings following the workshop. If the technical challenge of enabling real change in deploying open source Linux-based software for safety critical systems excites you, come join and help us meet the challenges!
You can still register for the Fall workshop, which is being held virtually and is free to attend. All registrants will be able to watch the sessions on-demand. Register here today!
Written by Paul Albertella, Chair of the ELISA Project Open Source Engineering Process Working Group
The ELISA Project’s new Open-Source Engineering Process (OSEP) Working Group focuses on the role of engineering processes in creating safety-related systems based on Linux and other FOSS.
Engineering processes are very important in safety, because we rely heavily on them to provide confidence in a system and its components. We achieve confidence by undertaking risk analysis to identify how harm may result from the use (or misuse) of the system, and then constructing a safety argument, which describes how these risks are managed.
When we apply this approach to a specific element of the system, such as a software component like Linux, the argument can be broken down into a number of claims that we want to make regarding that element and its role in the safety of our system. Some claims will relate to the functional responsibilities that the element has in the system; others will relate to the processes that we use to create and refine it.
Importantly, we also need to produce evidence to support these claims. Almost all of this evidence will be produced by an engineering process; some of it will be evidence relating to those processes themselves..
Safety standards like ISO 26262 and IEC 61508 describe reference processes that can act as a template for safety arguments like this. They identify the engineering practices that are seen be necessary (e.g. code review, verification through software testing), the formal processes that are used to control these (e.g. verification management), and the evidence needed to confirm that these have been applied (e.g. test plans, test results).
These reference processes are based on the V-model, which emphasises the formal specification of requirements, architecture and design, and the ability to trace formal verification processes back to these. For software, the standards focus on the processes used when developing new components for a safety-related system, although they include some guidance on applying the principles to pre-existing components, such as software libraries.
Open source projects like Linux have their own development processes, which may be sophisticated and make use of sound software engineering practices. However, it is difficult to map these directly to the reference processes described by the safety standards, because open source development models have very different goals and organizational models, which tend to emphasize refinement by rapid iteration, peer review and community contribution.
In order to address this, OSEP aims to identify and evaluate practices, processes and tools that FOSS developers, system integrators and product creators can use to bridge this gap. We plan to accomplish this by:
Selecting Linux topics and safety-related claims that we want to make about them
Identifying and evaluating practices, processes and tools to answer:
What risks are associated with the topic and claims?
To what extent are these risks addressed or mitigated by (or for) Linux?
How can we manage risks that are not sufficiently addressed or mitigated?
How can we show evidence to support our claims?
Collaborating with other WGs for technical investigations
Documenting and sharing our results as we go
If you would like to learn more about OSEP, join us for an overview presentation on November 8 at 3 pm CET at the ELISA Workshop. The Fall workshop, being held virtually on November 8-10, is free to attend and all registrants will be able to watch the sessions on-demand. Register here.
If you would like to contribute to OSEP, please join the mailing list here, where you can also find details of weekly meetings on the working group calendar.
ELISA Ambassadors are technical leaders who are passionate about the mission of the ELISA Project, recognized for their expertise in functional safety and linux kernel development, and willing to help others to learn about the community and how to contribute.
Today, we announce two new ambassadors – Jeffrey Osier-Mixon, Principal Community Architect at Red Hat, and John MacGregor, a thought leader with several decades of experience in software technology. Learn more about Jeffrey and John below.
Jeffrey “Jefro” Osier-Mixon:
Jefro currently focuses on automotive efforts. As a community architect, Jefro is responsible for maintaining Red Hat’s relationship with automotive-oriented communities, and he acts as the current chair for the CentOS Automotive Special Interest Group.Jefro has worked in open source for nearly three decades, having started his career as a technical writer with Cygnus Support working on documentation for the GNU tools. He has worked with Wind River and Montavista/Cavium Networks on embedded operating systems, and spent five years at Transmeta. He switched careers in 2011 and went to Intel to serve as the community and program manager for the Yocto Project, where he was the board chair for 7 years. During that time, he also helped launch Zephyr and Project ACRN. Most recently, he spent two years at the Linux Foundation as a program manager for RISC-V International and LF Energy.Jefro has been on the program committee for the Embedded Linux Conference series since 2010, and he speaks regularly at open source conferences. It’s best to catch him after the coffee kicks in.
John MacGregor:
John is currently spicing up his retirement by participating in various ELISA working groups. He started his long career as a scientific programmer, switched to Unix programmer and system architect, then progressed to project manager in telecommunications. He worked for several decades as Senior Expert for Software Technology in the Corporate Research Division of Robert Bosch GmbH. Among other things, he worked on software process improvement, software reuse, automotive software architecture and IoT technologies. Before retiring, John participated in the SIL2LinuxMP project, which focused on certifying Linux under IEC 61508 at the SIL2 level, and then continued to contribute to the ELISA project.
John holds a Bachelor’s degree in Industrial Engineering, specializing in operations research and information systems, as well as an MBA, specializing in marketing and finance.
Learn more about other ELISA Ambassadors here: https://elisa.tech/community/ambassadors/ Or, if you’re currently participating in the project and would like to become an ambassador, you can apply here.
Since launch in February 2019, the ELISA Project has created several working groups that collaborate and work towards providing resources for System integrators to apply and use to analyze qualitatively and quantitatively on their systems. Current groups include an Automotive Working Group, Medical Devices Working Group, Safety Architecture Working Group and Tool Investigation and Code Improvement Sub-Working Group to focus on specific activities and goals.
If you’re interested in learning more about the goals and objectives for these working groups or asking questions, we invite you to the ELISA Workshop on November 8-10. The virtual workshop, which is free to attend, will host speakers from Arm, Codethink, Elektrobit Automotive GmbH, Evidence Srl, Google, Intel, Mobileye, The Linux Foundation, Red Hat and UL LLC.
On Monday, November 8 at 5-6 am PDT, the working group chairs will provide updates on all activities. Led by Gabriele Paoloni, Lukas Bulwahn, Kate Stewart, Shuah Khan, Milan Lakhani, Jason Smith, Jochen Kall and Philipp Ahmann, you can add this to your schedule here.
Open Source Engineering Process Working Group: This working group aims to examine safety-related claims that we might like to make about Linux as part of a system, and to explore how we can gather and present evidence to support such claims.
Linux Features for Safety-Critical Systems Working Group: This working group will work to bring together kernel developers and producers of safety critical systems to demonstrate use of such features in real systems, and to learn from these experiences together as a community.
If you want to learn more about these two new working groups, we invite you to the session on November 8 at 6-630 am PDT lead by Paul Albertella and Elana Copperman. Add this to your schedule here.
ELISA Ambassadors are technical leaders who are passionate about the mission of the ELISA Project, recognized for their expertise in functional safety and linux kernel development, and willing to help others to learn about the community and how to contribute.
Each month, we’ll put a spotlight on an ELISA Ambassador. Today, we’re excited to highlight Philipp Ahmann, ambassador and TSC member within the ELISA project as well as software manager at ADIT (a joint venture of Robert Bosch GmbH and DENSO Corporation).
Background Details:
Philipp Ahmann is manager at ADIT (a joint venture of Robert Bosch GmbH and DENSO Corporation) and has been participating in the ELISA project since the start.
He has more than ten years of experience in automotive infotainment base platforms, utilizing complex multi-core system-on-chips (SoCs). Also, he is leading a group of engineers who are responsible for software integration (CI/CD), testing, development infrastructure and tooling within ADIT.
His automotive expertise started with integration of components in SoC hardware and printed circuit board (PCB) design for the same. From there, Philipp moved over to the field of software development with initial responsibility for bootloader and Linux software board bringup.
After working within the Linaro community and several years as lead of the test development within ADIT, he became software project leader. The projects mainly target OSS based in-vehicle-infotainment base platforms on various hardware variants. Nowadays also build infrastructure as well as software base platforms for autonomous driving products are in his responsibility.
Q&A
How long have you been active in open source?
My first open source work was done as a user. While I was studying in Sweden in 2006, a friend and I collected old university PCs from the electronic scrap and installed Ubuntu 6.10 on them. Afterwards we maintained and distributed them to exchange students who couldn’t afford an own laptop or PC.
Really active in open source, I became a member of the Freescale landing team within Linaro in 2011 to drive ARM Kernel and BSP development for i.mx6 SoC forward.
Tell us about your favorite open source project and what problems did it aim to solve?
It is really hard to define my favorite open source project as they are everywhere in my life. My private NextCloudPi gives me full control over my data. Home Assistant integrated perfectly with ESPHome and is helping me to automate tasks in my flat and surrounding for higher convenience and energy savings. LineageOS, CarbonROM, /e/ brought back new life to old smartphones serving as daily drivers for my kids and parents making technology more sustainable.
Thanks to projects likes Linux Mint, which shows decent performance even on old devices, old PCs and laptops get a second life. On devices tools like LibreOffice, Red Notebook, Freeplane, Arduino IDE, VS Code and others, help me to structure my day and increase productivity. For fun and entertainment there are projects such as Kodi and RetroPie.
Overall, I am pretty sure everyone touches open source at one time or another, since open source software rules the world. It is there, where people need it. From the people for the people. A big thanks and kudos to all of you who participate in open source projects.
What roles and/or working groups do you have or participate in?
I host meetings, act as moderator, write minutes and jump in where I can help to drive topics forward and where my support is needed or wished. For technical content, I mainly contribute within the Automotive Working Group, where I benefit from my many years background in Linux for Automotive.
Where do you see the ELISA Project in three years?
Since I am primarily active in the Automotive WG, I would like to try to make a forecast for this group. In 3 years, we will have completed and showcased our first use case, which is a telltale application.The created work products will act as a blueprint to get the first fully Linux-based instrument cluster on the market.
What is the biggest strength of the ELISA community?
The biggest strength of the ELISA community is the diversity, which we achieve with experts from many different working fields, domains, industries and interests from all across the globe. The diversity of perspectives, coupled with the transparency and communication, is crucial to the success of safety relevant projects. By sharing our concepts, we get a lot of feedback from e.g. the Linux and the safety community. Of course there are passionate in-depth difficult discussions, but these are open and not driven by commercial interests. Risks, potential gaps or also any other concern is addressed from the beginning.
If we continue on this path, the results from the ELISA community can act as state-of-art technology and as a benchmark for many safety critical systems in the future. We contribute to a safer world.
What’s your favorite quote?
“Be yourself (no matter what they say)” – by Sting
3 Fun Facts:
I once repaired an entertainment system of a plane during the flight and asked the pilot if he could use the internet connection in the cockpit to search for a Windows NT dll file for me. If you are curious about the root cause, get in touch with me.
Already twice I have been on an overseas business trip and my luggage was delayed for so long that it got delivered only the day before departing again. Luckily, I bought at least a T-Shirt in Paris before going to NYC.
I built an Arudino based music player with RFID and arcade button control. It took me a year from the first PoC to be robust enough for my kids. At the time I was done, my daughter learnt how to use an Android phone and preferred a touch display and cover flow. So I put a custom rom on a phone, flashed it, and removed any unintended services to make it a kids-ready data-privacy device. This took me only a week in the end.
To learn more about ELISA ambassadors, please click here.
Written by Elana Copperman, ELISA project ambassador and System Safety Architect at Mobileye (Intel)
This blog has been updated with the video from the Linux Security Summit (LSS), which took place on September 29-October 1.
Are you attending the upcoming Embedded Linux Conference (ELC) on September 27-30 or the Linux Security Summit (LSS) on September 29-October 1? This year, attendees have the option of joining the conference on-site in Seattle, Washington or virtually from their homes and workplaces.
Security and Safety have common goals, yet often follow divergent development paths. We will take a look at various Linux features which were originally designed for security, investigating if/how these features may be relevant to enable safety critical applications.
For example, we’ll discuss:
Memory protection features
Isolation techniques and FFI (Freedom From Interference)
Timing and execution
ebpf and profiling
Safety extensions to Linux drivers
I will present practical implications – focusing on where security and safety meet and where they don’t meet. The presentation, which is intended for experienced software developers and architects, will focus on how these features may be used in real systems. The goal is to spark discussion on how safety mechanisms may be designed in Linux-based safety critical systems, by learning from solutions in the security domain. Watch the video below or check out the presentation here.
Click here to register for the Linux Security Summit or here to learn more about the conference.
The Linux Plumbers Conference, which happened virtually on September 20-24, had a packed schedule of microconferences and tracks for the kernel, networking & BPF, GNU Tools, Birds of Feather and more. To see the complete schedule, check out the main conference page at https://www.linuxplumbersconf.org/event/11/.
Shuah Khan, Chair of the ELISA Project Technical Steering Committee and a Kernel Maintainer and Linux Fellow at the Linux Foundation, teamed up with Gabriele Paoloni, Chair of the ELISA Project Governing Board, Safety Architecture Working Group Chair and an Open Source Community Technical Leader at Red Hat, to run the Kernel Dependability and Assurance Microconference on Thursday, September 23. The Kernel Dependability and Assurance Microconference focused on infrastructure to be able to assure software quality and that the Linux kernel is dependable in applications that require predictability and trust.
If you missed the conference, you can watch the video below.
Additionally, several other ELISA Project ambassadors and community members presented sessions including Daniel Bristot de Oliveira, Principal Software Engineer at Red Hat, Sudip Mukherjee, a Kernel Engineer at Codethink, and Lukas Bulwahn with Elektrobit GmbH.
Check out the schedule below for the Microconference on Thursday, September 23 at 7 – 11 am PDT.
Written by Shuah Khan, Chair of the ELISA Project Technical Steering Committee
Please join me in welcoming Jason Smith, Paul Albertella and Philipp Ahmann to the ELISA TSC. They have made significant contributions to the ELISA project and their addition will strengthen the TSC and help us continue to make progress with our mission.
A brief summary of their background and contributions are as follows:
Jason Smith
Jason Smith, the Principal Engineer for Robotics and Control Systems Consumer Technology at UL LLC., has:
– Contributed a Linux in Basic Safety White paper to the project
– Participates in the Medical Devices Working Group regularly
– Contributed the analysis of 62304 SOUP
– Speaks frequently at ELISA Workshops and is an ELISA ambassador
Paul Albertella
Paul Albertella, Consultant at Codethink, has:
– Participated in ELISA since the inception.
– Presented technical content at the last two ELISA workshops
– Contributes in the TSC and other working groups, where he’s been a constructive collaborator
– 20+ years of software engineering experience and understands the open source ecosystem challenges
Philipp Ahmann
Philipp Ahmann, a manager at ADIT (a joint venture of Robert Bosch GmbH and DENSO Corporation) has:
– Participated in ELISA since the start of the project
– Took the lead in helping the project to improve communication and set up our LinkedIn presence
– Volunteered to be an active ambassador on behalf of the project
– Participates in the TSC and other working groups, where he’s known to be a very constructive contributor with an excellent overview across all the different areas we’re working on
– Demonstrates a helpful attitude by being willing to step in and host meetings when the chair is not available
– He is able to actively listen, and helps bring focus to the key elements we need for the project
I look forward to collaborating more closely with all three of our new TSC members. As a reminder, all are welcome to join the bi-weekly public technical community meeting and to contribute your perspectives. You can find the meeting details and subscribe to the calendar here: https://lists.elisa.tech/g/devel/calendar.
Written by EliGurvitz, ELISA Project Ambassador and Functional Safety Architect at Intel (Mobileye)
In a functional safety system FFI is required when the system consists of elements of different Safety Integrity Levels (ASIL).This is to ensure that elements allocated with a lower ASIL do not interfere with elements allocated with a higher ASIL; if FFI cannot be demonstrated the lower ASIL elements must be upgraded to the higher ASIL.
The Architecture Working Group has been discussing “Freedom From Interference (FFI)” in the last several meetings and is considering two aspects:
FFI between user space processes allocated with different ASIL
FFI between Linux Kernel components/drivers/subsystems allocated with different ASIL
This blog post focuses on the second bullet.
FFI is a key goal of a possible Safety Concept for Linux because Linux is too complex and has too many features, thus considering Linux as a single element of a certain ASIL would result in a very high functional safety qualification effort. If the application runs in a single threaded process and handles interrupts synchronously, then it may be possible to avoid allocating Safety Requirements to the OS and mitigate all failures with application-level safety mechanisms. But this kind of use requires just a simple OS and Linux is an overkill. Using Linux in the way it was meant to be used means it will be the OS of a multi-core SoC that runs many processes with different requirements of different ASILs.
This mode of use is referred to in ISO 26262 part 6 section 7.4.8:
This section refers to ISO 26262 part 9 Clause 6 “Criteria for co-existence of elements”. This clause states:
The Architecture WG investigation considers the Linux kernel partitioned into sub-elements of mixed criticality, therefore the goal is to show FFI between the sub-elements. The approach to FFI that is currently being discussed in the Architecture WG was developed by ELISA Project members Mobileye and BMW.
The first step in demonstrating FFI between safety-related and non-safety-related sub-elements is to identify the sub-elments and to allocate them with an ASIL. Since we are analyzing a SW component, the sub-elements are functional areas (or features) of the kernel, e.g. memory management or file systems, and they are made of C language functions. We classify the C functions according to the allocated ASIL by using the Call Tree Tool.
The goal of Call Tree is to statically generate the tree of function calls departing from a specified input one; hence starting for example from a syscall, Call Tree would generate the tree of all invoked functions. Call Tree scans the Linux source code by using the GNU CFlow utility and generates an SQLite database that contains all functions and their calling relations – this provides an almost full call-tree for every C function.
To classify every Kernel function we allocate Kernel entrypoints (syscalls and interrupt handlers) with safety requirements and associated ASIL; hence every function falling in a certain tree inherits the ASIL associated with the top level entrypoint. If a function is present in multiple trees, it is then assigned with the highest ASIL across those allocated to the different trees.
For example, if there’s a safety requirement for “safe dynamic memory” then we consider the related system calls – mmap, sbrk – as safety related. The union of all functions in the call trees of mmap and sbrk are considered SR and inherit the ASIL allocated to mmap and sbrk.
Once we have partitioned the Kernel the next step is to consider the possible types of interference. These types are defined in Annex D or ISO 26262 part 6. There are three types of interference:
Temporal – interference related to time or scheduling. The most common case is when one kernel thread prevents other threads from getting CPU cycles, thereby causing delays. Another example is a process crashing.
Spatial – interference related to space, or memory. For example, a lower ASIL driver may corrupt a kernel data structure.
Communication – normally this type of interference relates to transfer of data between two entities over a communication channel. In our analysis we consider static and global variables and pointers as communication channels between sub-elements of the kernel.
The Architecture Working Group plans to deal with all types of interference and currently we are considering the third type – communication interference. We are looking at areas where the internal state of the kernel can be corrupted because of the interaction between NSR and SR C functions (or more generally, C functions of different ASIL ratings).
The internal state of the kernel consists of many persistent data structures. These data structures, for example linked lists, are pointed to by global and static variables and pointers. Corruption of these data structures can occur in different ways.
Data structures that are accessed via global variables can be corrupted when a lower ASIL function (for example a driver that is rated as ASIL QM) accesses the same data structure that is also used by an ASIL-B function, as depicted in the diagram below.
Corruption of data structures that are accessed via static variables can occur when a static variable is used by a higher ASIL (or SR) function but this function is used by a lower ASIL (or NSR) function. The NSR function may pass a faulty argument to the SR function and the SR function may use this argument to modify the data structure. The faulty data structure is later used in a safety-related flow. This failure mode is depicted in the diagram below.
This description is only a preliminary formulation of the concept of communication interference within the kernel. The working group is debating the correct use of terms, the concept itself, the correct use of the Call Tree Tool and the selection of ASIL rated system calls for our sample automotive use case – The Tell-Tale signal.
If you are interested in safety engineering, the Linux kernel, or both, then please join us in these discussions. The nice thing about applying the existing Functional Safety standards to the Linux kernel is that there’s plenty of space and freedom for creativity, as these standards were designed for much simpler HW and very much simpler SW. It is as if there’s a written tradition of Safety architecture – the ISO 26262 standard and an Oral interpretation of it which creates a more modern tradition of Safety. You can be a part of creating this tradition. I should also take back the word “creativity” I used four lines above because it will certainly trigger a hot debate around the question of whether Safety likes “creativity” or hates it. So I’ll clarify that we are trying to be creative in a conservative way.
Learn more about the ELISA Architecture Working Group or any of the other groups in this white paper.
In May, the ELISA Project hosted its 7th Workshop with 239 participants from 37 different countries. For a complete recap of the workshop, click here. Today, we’ll take a look at one of the sessions titled “A Guided Tour Through the PREEMPT RT castle” presented by Thomas Gleixner, CTO at Linutronix GmbH.
The tour through the inner workings of PREEMPT_RT will start at the observation deck to give an conceptual overview. From there it will take the participants through the various chambers which contain a broad range of historic and contemporary operating system technologies. The tour will not only take the hallways it is also going to explore some of the secret passages and the brave-hearted can take a glimpse at the horror cabinets.
General knowledge about operating system concepts is recommended for taking the tour, but of course it’s open for everyone and all chambers have exit doors if it gets too spooky.
