Our objective is to specify a system context and an example set of safety goals for a safety-related system involving the Linux kernel, in order to enable the safety analysis and specification of a set of safety responsibilities that we may assign to the kernel in that context (and possibly other contexts), at a useful level of detail.
By system context, we mean either a concrete system design, or an abstraction representing a class of system designs.
By safety goals, we mean a set of system-level criteria that must be satisfied in order to avoid specific negative outcomes or consequences.
By safety responsibilities, we mean the behaviour or properties that are required to avoid violating the safety goals for the given system context. This may involve cooperation with other safety mechanisms, which are required to operate when it is not possible to avoid violating a high level safety goal.
In ISO 26262 terminology, this is equivalent to defining the assumptions of use (AoU) for Linux (or any FOSS component, or integration of components) as a Safety Element out of Context (SEooC).
You can find more information about this approach and its intended role as part of a wider engineering process in the Refining the RAFIA approach talk from last year’s ELISA Spring Workshop.
Our purpose with undertaking this kind of analysis in ELISA is to describe and provide examples of a method for identifying and documenting the risks associated with using Linux in a given context, and to examine how its existing features may be used to help to identify, control and/or mitigate these risks. The results of this analysis may then be used to derive the safety requirements that should apply for a system using Linux in such a context
For example, certain Linux configurations may help to address some of the risks that we identify, while others – including default kernel configurations – may introduce additional and possibly unacceptable risks (e.g performance optimisations that may have unintended or unpredictable consequences). Some mitigations for these risks may be outside the scope of Linux, involving AoUs on applications using Linux or on other components integrated with Linux as part of an operating system.
The outputs of this process are expected to be:
A description of the system context and safety goals that we have assumed for the purpose of the analysis
A specification of the risks that are considered in the analysis, and any exclusions from scope
Safety responsibilities for both Linux and the other system components identified in the system context, and specific requirements relating to these
Specific scenarios that might lead to these safety requirements being violated, which can be used to derive test cases and fault injections, and to identify where additional mitigations, safety mechanisms and/or requirements are needed to deal with these scenarios
Documents capturing the results of Linux feature analysis (or analysis of other interacting components) that was undertaken as part of the investigation
The OSEP group plans to start by applying this approach to the Automotive working group’s Telltale use case, documenting and refining the process as well as recording the results of the analysis.
We have invited other ELISA working groups and contributors to consider the following:
What other system contexts and safety goals should we consider for analysis?
What specific Linux features or properties should we focus on in our analyses?
How might ELISA working groups collaborate in applying, refining and documenting this process and its results?
Are there any external communities with which we might collaborate?
We would also welcome input from other open source contributors and communities interested in functional safety.
If you would like to get involved or learn more about this approach, please join the OSEP mailing list, where you can also find details of our meetings and how to participate.
Written by Philipp Ahmann, Chair of the ELISA Project TSC (Robert Bosch GmbH) and Sudip Mukherjee, Member of the Tools Working Group (Codethink Ltd)
This article describes how the ELISA Project has enabled Continuous Integration (CI), using the Automotive working group use case as an example. The key goals are to:
Make it easier for others to onboard the project
Experience deliverables from the various working groups
Make the work reproducible and more dependable
Additionally, it describes which elements are part of the pipeline along with the tools involved in the creation and testing of the release images. A remarkable element is that the complete pipeline is coupled to the documented development flow. This means that the CI reproduces the steps a developer would do on a PC.
In this way, it approaches people who are new to safety as well as those which are new to Linux. It is a collaborative approach of multiple working groups within ELISA and should foster more collaboration in upcoming ELISA activities.
Motivational factors that enable the CI
In a collaborative project, people of different interests and technical backgrounds unite to work together. After a successful onboarding process, it is important that contributions and work results become visible to the wider community. They need to be properly basellined and distributed fastly to stakeholders.
A high degree of automation increases the dependability of the development process and reduces failures introduced by human slip. It also gives more time back to developers as they do not have to do everything from scratch.
The ELISA project also relies on external sources for support – for example – the Linux Foundation sister projects Automotive Grade Linux (AGL) and the Yocto Project. The CI guarantees the rebuild whenever updated packages from AGL or Yocto are provided.
In summary, one can say that these demands drive the enablement of the CI within ELISA Automotive WG and put a set of requirements to it:
A entry point for stakeholder with different (technical) background and experience
A respect of different stakeholder interests in the work products
Faster ramp-up for new project contributors
Better and faster integration and distribution of new features and improvements
Automated and dependable software product creation
Extendable testing to validate concepts and assumptions.
With the successful setup of the prototype for the automotive use case, the goal is to scale towards more complex architectures as tackled within the ELISA Systems WG. The basic approach and tools will remain the same in the extension towards new use cases.
The elements currently involved in the CI include:
All elements are coupled and tied together to build a dependency chain, which makes it easier to automatically detect failures, caused by changes along the elements of the pipeline.
The CI elements and flow within ELISA
The ELISA Project uses GitLab to run the CI pipeline, which interacts with an ELISA hosted build server (which is also used for other purposes as running tools for code improvement). The build sources are taken from Automotive Grade Linux Gerrit, Yocto Project repositories as well as the ELISA GitHub repositories.
Currently, the generated build artifacts are the kernel and root filesystem for QEMU x86_64 image. It is planned to extend these to additional images for physical hardware in the future.
While the CI is running in GitLab, ELISA is hosting source code and documentation in GitHub. This is the starting point of the CI flow.
The build is triggered on a regular basis once per day to provide a fresh image and check the functionality of the generated image. Due to the usage of cached build artifacts, the longer build times of a full yocto build are reduced to a minimum (in the range of 15 minutes). The generated images are only kept for one day, so that a new download will always have the latest changes included. This further supports the idea of continuous integration. The latest image can be downloaded from the GitLab project packages.
Also a manual trigger of the CI can be done by referencing a GitHub branch of meta-elisa from a PR. In this way a pre-check from changes is possible before formal images are released from the master. It further reduces eventual regressions. To avoid misuse, access to trigger new builds from referenced GitHub forks is restricted to ELISA working group members.
The strong bound of build server to documentation also makes sure that the docker file for manual build is used to create a docker image, which is then used to create the build artifacts for the QEMU image. Also, this is newly created on a regular basis, to make sure that documentation and tool updates will always result in a working image using the latest documentation.
This is needed to achieve the different entry points for the stakeholders who want to try ELISA artifacts or contribute to the project work.
Dependency to the AGL instrument cluster demo
As already mentioned before, but not directly visible from the build flow, is the dependency to the AGL instrument cluster demo. As ELISA collaborates with AGL, it enhances the existing work of AGL and uses it for demonstrating. The results are still a work in progress; therefore the generated meta-elisa layer is not integrated into the formal AGL repositories (yet). Missing this upstreaming causes the risk that the ELISA automotive WG tell-tale demo will break in case AGL makes changes to interfaces used by ELISA. Similar changes can happen for changes in the yocto-poky base.
By building on a daily basis and testing the generated images for the added functionality, it is guaranteed that any breaking change will become directly visible and immediate actions can take place. In these scenarios the latest image will not be deployed for download and analysis of the reported errors can be done.
As a full yocto build can be resource intensive and time consuming the use of cached build artifacts with an enabled SSTATE helps to reduce the build time to a minimum.
Benefits of using Yocto’s SSTATE cache
Building a yocto AGL image from scratch is taking time and a lot of computation performance. At least 100GB of disc space is consumed and a build can easily take several hours to complete even on decent hardware.
In order to reduce the build time and limit the necessary disc space, it is possible to turn on cached build results. The so-called SSTATE is created with the first full build and shared for upcoming builds, demanding to only build deltas. Any change in sources and updates are detected and necessary binaries and their dependencies will be recompiled in case needed.
The generated cache is used within the CI, but can also be used for a local build at a developer PCc.
Additionally, from time to time a new cache is generated to check that the build will also work in case someone builds the system from scratch. Building from scratch can make sense for low network speeds or networks with traffic limitations. This further makes sure that the GitLab pipeline fits to the documentation as much as possible.
GitLab pipeline coupling to the meta-elisa documentation
The GitLab pipeline basically represents the steps which users would take when manually creating the ELISA enhanced demo of the instrument cluster provided by Automotive Grade Linux. As the current version is based on QEMU it is easy to reproduce the demo at a local machine and also to perform automated testing. This is illustrated by transferring the created images to the OpenQA server hosted by Codethink.
It is important to confirm that the build was successful. As the original AGL demo has been enhanced with the Signal-Source-Control application, which also renders additional screen content, it makes sense to do an output comparison. For the meta-elisa demo OpenQA was used as a proven tool to check proper boot and functionality of the code modifications. This should also avoid potential regressions, e.g. due to external dependency or interface changes. An example output of a comparison as generated by OpenQA can be found below.
In the OpenQA tool, the orange divider in the presented screenshot can be moved left and right with the cursor. This makes it easy to identify differences between reference and recent execution. The greenish frame shows the region of interest which is checked for content. The 100% shows the fit ratio.
Additionally, during the test execution a screen recording is performed and further logs such as those from serial terminal are provided.
With this option, a fully automated flow from source repository to generated and tested image is achieved. It is possible that working group members can fork the existing meta-elisa repository on GitHub, do modifications, check that it is built properly using GitLab and that the image remains functionally by checking the OpenQA result. This flow would be completely web based without the demand of even having a development environment setup locally on a PC.
However, this may more be used as an exceptional case. The interaction with the system to understand and learn is rather limited in this scenario. As an intention is to develop the system further and gain a larger understanding of the demand of a safety-critical system with Linux based workloads, there are many more entry points for different levels of expertise and personal interest.
The different entry points to experience the automotive use case
In order to make it easier for new contributors to join the ELISA Automotive WG, the demo has multiple entry points.
Closing the loop to the motivation, the aforementioned entry points fulfill the demands of a collaborative project, where people of different interests come together to collaborate. It is possible to understand the system creation from scratch with an own build from source, or just trace a workload by using the pre-built QEMU images. The development flow is accelerated by providing docker images and cached build artifacts. As the outputs of different pipeline steps are available, debugging gets simplified as well (like checking a native build against a docker build or behavior of the own vs. generated QEMU image).
In summary, a comprehensive documentation (including this post) helps to tailor the pipeline and images to individual demands. Though, to have the best compatibility for exchange with others, the usage of a docker container is recommended.
Docker container for better compatibility of CI and manual reproduction
Taking into account the large number of different host machines and Linux distributions that exist, from practical experience the ELISA CI is providing a docker file as a common base for development. The docker file is used to generate the docker image, which again is used to build the meta-elisa. By providing the docker file and docker image also to others, who want to create a local build on their machine, influence of the underlying Linux distribution is reduced. Documentation about the docker container creation and usage can be found in the wg-automotive github repository. To pull the docker container directly use:
Some general remarks regarding container should be mentioned for competition:
There is still a (limited) dependency to the underlying OS outside the container, as the docker image uses for example the Linux kernel from the host system. However, side effects of missing environment setups, conflicting or overwriting packages are minimized by using the docker container.
Regarding build time and performance the impact of the container could be neglected. The duration and system load during the build with and without container usage was very similar.
The docker file will be kept up to date to reflect demands as per meta-elisa or dependency from yocto or AGL updates. This makes sure to always have the latest features and improvements also for the planned next steps reflected for better reproduction of results.
Outlook and next steps
As an evolution of the use case, activities have been started to enable a reference image of an example architecture including Xen and Zephyr on real automotive hardware. Any result from the automotive as well as other working groups within ELISA should feed into the example architecture tailorable to specific use case demands.
Additionally having a proper software bill of material (SBOM) generation will also enhance the quality of the reproducible demo. It will support upcoming regulations which require an understanding of the content of your software product. As AGL also works on the reproducible build to generate binary compatibility of builds this may be an additional enhancement considered as future activity.
✔ By setting up the CI with QEMU image deployment to the openQA testing it is possible to quickly track changes and validate code changes for proper appliance along the chain from source to deployment.
✔ Active contributors in ELISA have the chance to check their own commits or pull requests by using a standardized GitHub development flow and start the CI by a two line change in the pipeline reference to the meta-elisa fork and branch commit.
✔ The different entry points and precompiled build results enable interested people to decide to build an image from scratch, use a docker container to reduce environment setup steps and reduce build time to a fraction by taking cached build artifacts from the server.
✔ By provisioning the final QEMU build image, it is even possible to access and boot the generated image without the need of performing an own build.
As all pipeline elements from documentation to later qa testing are dependent on each other and are coupled to an integrated flow. A high degree of dependability and reproduction is achieved with this solution. With a future enhancement of SBOM generation and utilization of reproducible builds project, the system dependability will further be increased. The automated execution supports that aligned development flows are kept and regressions can be tackled before they occur. Best entry points for reproduction of the results are the meta-elisa and wg-automotive docker container documentation in GitHub.
FOSDEM aka Free and Open Source Software Developers European Meeting is a non-commercial, volunteer-organized European event centered on free and open source software development that is aimed at developers. FOSDEM is held annually, usually during the first weekend of February, at the Université Libre de Bruxelles Solbosch campus in the southeast of Brussels, Belgium.
Philipp Ahmann, Chair of the ELISA Technical Steering Committee and Technical Business Development Manager at Robert Bosch GmbH, gave a presentation at FOSDEM titled, “The ELISA Project – Project insights and overview”. The lecture mainly focused on the goals and technical strategy of the project. It provided information about the different work groups, their interaction, and contributions.
ELISA members are working together to define and maintain a common set of tools and processes that can help companies demonstrate that a specific Linux-based system meets the necessary safety requirements for certification. These existing working groups focus on Linux Features for Safety-Critical Systems, Software Architecture, Open Source Engineering Process, Tool Investigation and Code Improvement. They are complemented by vertical use case working groups dealing with Automotive, Medical, and Aerospace.
During the talk, Philipp explained in detail about each working group. Safety Architecture group monitors the kernel for non safety related issues such as the watchdog kernel analysis and try to improve using tools and documentation and it is driven by use cases and demands of the products.
Linux features for safety critical systems group mainly work on the safety criticality part and not the generic features. The main goal is to look for the elements which could improve safety by doing incremental steps to make the system more dependable and robust.
While doing these processes, code quality is very important and hence the tool investigation and code improvement group. The WG uses tools and CI servers to identify the kernel issues and to make the kernel more reliable and robust.
Open Source Engineering Process WG was developed due to the challenges in Engineering. There are rigorous methods within kernel development. There is a strong demand from traditional project management when it comes to safety products. We can see that not every process compliance agrees with it directly. We need to find an equivalent to the open source development process compared to ISO 26262 requests for automotive products.
Systems work group works cross functional and cross projects and combine these elements involved in the above mentioned processes.
In order to tailor the systems accurately with these groups, ELISA Project has vertical use cases such as Aerospace, Automotive. Medical etc.
He also explains how ELISA Project is interacting with other communities such as Xen Project, Zephyr Project on safety critical topics, Automotive Grade Linux, SOAFEE and SDV on automotive use cases and Yocto project for build tooling and SBOM generation and SPDX for system SBOM data requirements.
In addition to the different work groups and their interactions Philipp also explained on the contributions of the different groups, use cases, used methodologies such as STPA and workload tracing and a status update on what to expect from ELISA Project.
As Philipp mentioned, based on George Bernard Shaw’s quote “If you have an apple and I have an apple and we exchange these apples then you and I will still each have one apple. But if you have an idea and I have an idea and we exchange these ideas, then each of us will have two ideas” we need to get a good understanding and bring things together.
Recently, Philipp Ahmann, an ELISA Ambassador and member of the Technical Steering Committee and Technical Business Development Manager at Bosch, had the chance to speak at the Lund Linux Con about the technical strategy of the ELISA Project, the established work groups as well as the work of the Automotive Working Group.
The Lund Linux Con is a Linux Kernel focused conference in the south of Sweden, typically taking place in May every year. More than 100 participants attended the in person event sponsored by Axis Communications, Western Digital and Volvo Cars to meet and exchange about various Linux Kernel related topics. You can watch Philipp’s presentation below or check out his slides here:
Written by Jeffrey “Jefro” Osier-Mixon, ELISA Project Ambassador and member of the TSC and Senior Principal Community Architect at Red Hat
This blog originally ran on the Red Hat website. For more content like this, click here.
The automotive computing world, like many other industries, is going through a transformation. Traditionally discrete computing systems are becoming more integrated, with workloads consolidated into systems that look remarkably more like edge systems than embedded devices. The ideas driving this shift come from open source, but will Linux be part of this future, given that the existing standards for functional safety do not currently accommodate Linux-based operating systems?
Red Hat safety expert Gabriele Paoloni will present our safety methodology to the Automotive Linux Summit as an update to the presentation given at the recent ELISA Workshop. This methodology outlines the path Red Hat is taking toward creating an in-vehicle OS that incorporates modern ideas around workload orchestration, secure process isolation, and consolidation of mixed-criticality workloads, field-updatable and continuously certified for functional safety. We recognize that this is a huge task, but we believe it to be possible by adhering to these development pillars:
Open methodology development within ELISA
Participation in the ISO 26262 update process
Open code development within the CentOS Automotive SIG
ELISA (Enabling Linux in Safety Applications) is a vital part of the Linux universe, and it provides a community where those of us who care deeply about functional safety can address the challenges of certification and create solutions to resolve those challenges. ELISA is the cornerstone open source community for functional safety, and automotive is a big focus as the industry is clamoring for transformation and advanced computing facilities within the car. Red Hat recognizes that value and has emerged as a leader in the community.
ISO 26262 was originally developed in a time when automotive computing was managed through Electronic Control Units (ECUs), black boxes that had a deterministic output when given specific inputs. The standard is notably lacking support for pre-existing complex systems, including Linux. Red Hat is a member of an ongoing effort to update ISO 26262, known as ISO-PAS 8926, which has been accepted as a new working item proposal to the ISO committee.
ELISA Ambassadors are technical leaders who are passionate about the mission of the ELISA Project, recognized for their expertise in functional safety and linux kernel development, and willing to help others to learn about the community and how to contribute.
Each month, we’ll put a spotlight on an ELISA Ambassador. Today, we’re excited to highlight Elana Copperman, PhD, ELISA ambassador, Chair of the Linux Features for Safety-Critical Systems (LFSCS) Working Group and TSC voting member. Elana works as a System Safety Architect in Mobileye, an Intel company, designing features to support safety as well as security features in Mobileye vision products for the automotive domain.
As a System Safety Architect at Mobileye, Elana provides support for designing critical system features in Mobileye products, including system boot; drivers; and Linux infrastructure. Before working at Mobileye, she worked as a Security Architect for Cisco-Il (formerly NDS) and more recently as a security consultant for major European automotive concerns on behalf of various Israeli startups. Most recently, merging with safety constraints in the automotive domain, to deploy secure as well as safe systems. Her research interests focus on software engineering methodologies and security engineering.
How long have you been active in open source?
I have been involved in “open source” before it was formalized as we know it today. During my time as an undergraduate student, online software source code was commonly shared as freeware. In fact, even early Unix versions were provided at no cost to academics, and collaborative efforts were supported to some degree.
My PhD research focused on Object-Oriented Programming, including some investigations on Java source code and features. Open source software development, Linux in particular, has evolved since then, including many new challenges and opportunities. For example, setting up and working with Apache Server over Linux OS in the early 1990’s.
Over the last 15 years, Linux has grown to what we know & love today, with its amazing powers. As a system architect, I have been designing systems that empower Linux in embedded systems, first in Set-Top Boxes (for digital broadcasting) and currently in Automotive.
Tell us about your favorite open source project and what problems did it aim to solve?
My favorite open source project focused on security code review with ST (chip vendor) engineers for the optee code, which had been released to Linaro. To see first-hand how open source aligns vendor, kernel and user requirements with features to resolve complex security challenges was mind-blowing.
How long have you been active in the ELISA Project?
What roles and/or working groups do you have or participate in?
I was the Chair of the Kernel Development Process Working Group, and am currently the new Linux Features for Safety-Critical Systems WG. I am learning from the safety experts on board, and my primary contribution is to represent the viewpoint of the application designers and developers who will build Linux-based safety critical systems.
Where do you see the ELISA Project in three years?
My dream for ELISA is to see a community of developers providing kernel modules which may be leveraged for use in safety critical systems. For example, a suite of memory protection mechanisms derived from well defined requirements that may be deployed to protect safety-critical data. Another example, which is in our planned agenda, are test suites for concurrency issues such as deadlock and race conditions, focusing on test plans derived from safety requirements.
What is the biggest strength of the ELISA community?
I think ELISA has matured since its early days and we are currently more structured in our analysis and goals. Our strength is in current efforts to progress and making real contributions to the Linux kernel community.
What is the best piece of advice you have ever received?
Never give up. Be flexible, adapt, adjust – but keep up the hard work.
What technology can you not live without? Why?
My home coffee machine. Drop in for a fresh cup and you will know why.
What part of the world do you live in? Why do you love where you live?
I live in Israel, a tiny multicultural nation at the junction of 3 continents (Europe, Asia, Africa). It has also become a major highway for birds – and a pilgrimage site for bird watchers. A wide variety of species can be viewed making their way south in the fall and back north in the spring. Thomas Krumenacker (German journalist and photographer) wrote a book on “Birds in the Holy Land”.
Now is a great time of year to take a trip to the Hula Valley Nature Reserve, to get a view of some of the estimated 500 million migrating birds passing over the Hula Valley and stopping by for a drink of water. This exemplifies the amazing diversity of the people and wildlife of our small fast-paced country.
What’s your favorite quote?
3 Fun facts:
My first computer was a PCJr, with 64 Kb of memory, and with virtually no hardware or software. I had to install all my own hardware add-ons and get it to work, including etching bit maps for the Hebrew alphabet in memory and getting it to work from right to left. After adding more and more components to the motherboard, one day it blew up. But I had a lot of fun with it until that happened.
As a grad student at NYU, we had loads of fun playing with early internet protocols and collaboration tools, including gopher and kermit; getting our hands dirty (“finger”) working with each other’s systems; and Mosaic (an early Web browser). The crazy things we did over the internet taught me a lot about potential security issues related to networking. This expertise was the foundation for my more recent work in security and safety engineering.
As chair of the Computer Department at the Jerusalem College, we aimed high but with a low budget. Trying to keep up to date with educational (i.e., no support or documentation) versions of closed source software was a real challenge. I became adept at finding contacts working with major software vendors who were willing to break down the walls, expose the source code and support my customization for our needs. In a sense, challenging proprietary software in open-source development mode …
To learn more about ELISA ambassadors, please click here.
ELISA Ambassadors are technical leaders who are passionate about the mission of the ELISA Project, recognized for their expertise in functional safety and linux kernel development, and willing to help others to learn about the community and how to contribute.
Today, we announce two new ambassadors – Jeffrey Osier-Mixon, Principal Community Architect at Red Hat, and John MacGregor, a thought leader with several decades of experience in software technology. Learn more about Jeffrey and John below.
Jeffrey “Jefro” Osier-Mixon:
Jefro currently focuses on automotive efforts. As a community architect, Jefro is responsible for maintaining Red Hat’s relationship with automotive-oriented communities, and he acts as the current chair for the CentOS Automotive Special Interest Group.Jefro has worked in open source for nearly three decades, having started his career as a technical writer with Cygnus Support working on documentation for the GNU tools. He has worked with Wind River and Montavista/Cavium Networks on embedded operating systems, and spent five years at Transmeta. He switched careers in 2011 and went to Intel to serve as the community and program manager for the Yocto Project, where he was the board chair for 7 years. During that time, he also helped launch Zephyr and Project ACRN. Most recently, he spent two years at the Linux Foundation as a program manager for RISC-V International and LF Energy.Jefro has been on the program committee for the Embedded Linux Conference series since 2010, and he speaks regularly at open source conferences. It’s best to catch him after the coffee kicks in.
John is currently spicing up his retirement by participating in various ELISA working groups. He started his long career as a scientific programmer, switched to Unix programmer and system architect, then progressed to project manager in telecommunications. He worked for several decades as Senior Expert for Software Technology in the Corporate Research Division of Robert Bosch GmbH. Among other things, he worked on software process improvement, software reuse, automotive software architecture and IoT technologies. Before retiring, John participated in the SIL2LinuxMP project, which focused on certifying Linux under IEC 61508 at the SIL2 level, and then continued to contribute to the ELISA project.
John holds a Bachelor’s degree in Industrial Engineering, specializing in operations research and information systems, as well as an MBA, specializing in marketing and finance.