The Safety-Critical Software Summit, which takes place on June 27-30, 2023 in Prague, Czech Republic, as well as virtually, as part of the new Embedded Open Source Summit conference is packed with technical content.
As open source is found more and more in safety-critical applications, the need to evaluate open source software that meets safety standards has increased. This event, sponsored by the ELISA Project, gathers safety experts and open source developers to enable and advance the use of open source in safety-critical applications. Check out some of the sessions and add them to your schedule:
Special registration rates are available for small businesses, hobbyists, students and virtual attendees.
Members of The Linux Foundation receive a 20 percent discount off registration and can contact events@linuxfoundation.org to request a member discount code.
Stay tuned by subscribing to the ELISA Project newsletter or connect with us on Twitter, LinkedIn or mailing lists to talk with community and TSC members.
Our objective is to specify a system context and an example set of safety goals for a safety-related system involving the Linux kernel, in order to enable the safety analysis and specification of a set of safety responsibilities that we may assign to the kernel in that context (and possibly other contexts), at a useful level of detail.
By system context, we mean either a concrete system design, or an abstraction representing a class of system designs.
By safety goals, we mean a set of system-level criteria that must be satisfied in order to avoid specific negative outcomes or consequences.
By safety responsibilities, we mean the behaviour or properties that are required to avoid violating the safety goals for the given system context. This may involve cooperation with other safety mechanisms, which are required to operate when it is not possible to avoid violating a high level safety goal.
In ISO 26262 terminology, this is equivalent to defining the assumptions of use (AoU) for Linux (or any FOSS component, or integration of components) as a Safety Element out of Context (SEooC).
You can find more information about this approach and its intended role as part of a wider engineering process in the Refining the RAFIA approach talk from last year’s ELISA Spring Workshop.
Our purpose with undertaking this kind of analysis in ELISA is to describe and provide examples of a method for identifying and documenting the risks associated with using Linux in a given context, and to examine how its existing features may be used to help to identify, control and/or mitigate these risks. The results of this analysis may then be used to derive the safety requirements that should apply for a system using Linux in such a context
For example, certain Linux configurations may help to address some of the risks that we identify, while others – including default kernel configurations – may introduce additional and possibly unacceptable risks (e.g performance optimisations that may have unintended or unpredictable consequences). Some mitigations for these risks may be outside the scope of Linux, involving AoUs on applications using Linux or on other components integrated with Linux as part of an operating system.
The outputs of this process are expected to be:
A description of the system context and safety goals that we have assumed for the purpose of the analysis
A specification of the risks that are considered in the analysis, and any exclusions from scope
Safety responsibilities for both Linux and the other system components identified in the system context, and specific requirements relating to these
Specific scenarios that might lead to these safety requirements being violated, which can be used to derive test cases and fault injections, and to identify where additional mitigations, safety mechanisms and/or requirements are needed to deal with these scenarios
Documents capturing the results of Linux feature analysis (or analysis of other interacting components) that was undertaken as part of the investigation
The OSEP group plans to start by applying this approach to the Automotive working group’s Telltale use case, documenting and refining the process as well as recording the results of the analysis.
We have invited other ELISA working groups and contributors to consider the following:
What other system contexts and safety goals should we consider for analysis?
What specific Linux features or properties should we focus on in our analyses?
How might ELISA working groups collaborate in applying, refining and documenting this process and its results?
Are there any external communities with which we might collaborate?
We would also welcome input from other open source contributors and communities interested in functional safety.
If you would like to get involved or learn more about this approach, please join the OSEP mailing list, where you can also find details of our meetings and how to participate.
Written by Shuah Khan, Linux Fellow at the Linux Foundation and member of the ELISA Project TSC
Key Points
Understanding system resources necessary to build and run a workload is important.
Linux tracing and strace can be used to discover the system resources in use by a workload.
Tracing OpenAPS commands with strace generated detailed view of system activity during the common run and helped with generating flowcharts for normal and error paths when commands detected device busy conditions.
Once we discover and understand the workload needs, we can focus on them to avoid regressions and evaluate safety.
OpenAPS is an open source Artificial Pancreas System designed to automatically adjust an insulin pump’s insulin delivery to keep Blood Glucose in a safe range at all times. It is an open and transparent effort to make safe and effective basic Automatic Pancreas System technology widely available to anyone with compatible medical devices who is willing to build their own system.
Broadly speaking, the OpenAPS system can be thought of performing 3 main functions. Monitoring the environment and operational status of devices with as much data relevant to therapy as possible collected, predicting what should happen to glucose levels next, and enacting changes through issuing commands, emails and even phone calls.
The ELISA Medical Devices Working Group has set out to discover the Linux kernel subsystems used by OpenAPS. Understanding the kernel footprint necessary to run a workload helps us focus on the subsystem and modules that make up the footprint for safety. We set out to answer the following questions:
What happens when an OpenAPS workload runs on Linux and discovers the subsystems and modules that are in active use when OpenAPS is running?
What are the interactions between OpenAPS and the kernel when a user checks how much insulin is left in the insulin pump?
Fine grained view of system activity
As mentioned earlier, the approach we gathered higher level of information about the OpenAPS usage. This higher information doesn’t tell us the system usage by individual OpenAPS commands. For example, watching the brain activity during the entire dinner vs. isolating the activity as we take the first bite of a delicious dessert and enjoy it.
Following up on our previous work, we gathered the fine grained information about individual OpenAPS commands and important use-cases. We used the strace commandto trace theOpenAPS commands based on the process we identified to trace a generic workload which is now in the Linux kernel user’s and administrator’s guide. We traced several OpenAPS commands on an OpenAPS instance running on RasPi managing a Medtronic Insulin Pump. “mdt” in the following text refers to the command provided by https://github.com/ecc1/medtronic:
Get Insulin Pump model (mdt model)
Get Insulin Pump time (mdt clock)
Get Insulin Pump battery (mdt battery)
Get Insulin Pump basal rate schedule (mdt basal)
Suspend Insulin Pump (mdt suspend)
Resume Insulin Pump (mdt resume)
Get the remaining insulin in the Insulin Pump reservoir (mdt reservoir)
Get Insulin Pump temporary basal rate (mdt tempbasal)
We ran these commands in normal mode and under strace to get summary (strace -c ) and complete trace (strace) information. The following shows a few selected commands, their output, trace information, and our analysis of the trace.
Running the command to get the remaining insulin in the reservoir
mdt reservoir &> reservoir.out # Get the remaining insulin in the reservoir
strace pumphistory -n 1 &> pumphistory.full # full trace
Output from the get pumphistory command
Run pumphistory command
retrieving pump history since 2022-10-12 10:54:03
cannot connect to CC111x radio on /dev/spidev0.0
null
/dev/spidev0.0: device is in use
Process startup (process mgmt)
Open files (fs -> driver sysfs)
Open pump device “/dev/spidev0.0”
Open “/sys/class/gpio/gpio4/active_low”
Open “/sys/class/gpio/gpio4/direction”
Close files (fs -> driver sysfs)
Exit program (process mgmt)
This output shows that the pump busy path is invoked
Pump history command flowchart
Complete strace -c output
2022/10/12 11:54:03 retrieving pump history since 2022-10-12 10:54:03
2022/10/12 11:54:03 cannot connect to CC111x radio on /dev/spidev0.0
null
2022/10/12 11:54:03 /dev/spidev0.0: device is in use
% time seconds usecs/call calls errors syscall
—— ———– ———– ——— ——— —————-
40.92 0.003106 25 120 rt_sigaction
15.91 0.001208 109 11 futex
8.09 0.000614 204 3 clone
7.84 0.000595 59 10 mprotect
7.52 0.000571 190 3 fcntl
5.23 0.000397 44 9 rt_sigprocmask
5.20 0.000395 17 22 mmap2
4.23 0.000321 321 1 readlinkat
3.77 0.000286 13 21 clock_gettime
1.29 0.000098 24 4 brk
0.00 0.000000 0 5 read
0.00 0.000000 0 4 write
0.00 0.000000 0 7 close
0.00 0.000000 0 1 execve
0.00 0.000000 0 1 getpid
0.00 0.000000 0 1 access
0.00 0.000000 0 1 readlink
0.00 0.000000 0 2 munmap
0.00 0.000000 0 1 uname
0.00 0.000000 0 1 1 flock
0.00 0.000000 0 2 sigaltstack
0.00 0.000000 0 1 ugetrlimit
0.00 0.000000 0 5 fstat64
0.00 0.000000 0 1 gettid
0.00 0.000000 0 1 sched_getaffinity
0.00 0.000000 0 1 set_tid_address
0.00 0.000000 0 7 openat
0.00 0.000000 0 1 set_robust_list
0.00 0.000000 0 1 set_tls
—— ———– ———– ——— ——— —————-
100.00 0.007591 248 1 total
The following diagram shows the run-time context for these commands and their mapping to the Linux subsystems used by them.
System view
The following system view was updated after high level tracing analysis. This system view remains unchanged after the fine grained tracing analysis as expected.
Using strace to trace OpenAPS commands helped us understand the detailed view of system files opened and closed while the commands run. We were able to generate flowcharts for normal and error paths when commands detected device busy conditions. As mentioned earlier, this tracing method gave us insight into the parts of the kernel used by the individual OpenAPS commands.
Credits
The ELISA Medical Working Group would like to sincerely acknowledge Chance Harrison for running the OpenAPS commands and providing information critical for making this effort a successful one.
SPDX-License-Identifier: CC-BY-4.0
This document is released under the Creative Commons Attribution 4.0 International License, available at https://creativecommons.org/licenses/by/4.0/legalcode. Pursuant to Section 5 of the license, please note that the following disclaimers apply (capitalized terms have the meanings set forth in the license). To the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.
To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.
The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.
We’ve got a few more weeks to go before Linaro Connect, which takes place on April 26-28 at the Park Plaza London Riverbank in London. Once a year, engineers, developers, thought leader and software experts come together for technical sessions and hacking. Discussions focus on the future of open source software, solutions and best practices.
This year, ELISA Project is participating in the Linaro Connect 2023 event and will be presenting an update on the progress in Enabling Linux in Safety Applications (ELISA). The session, which is scheduled on Friday, April 28 from 11:20 – 11:35 am, will be presented by Philipp Ahmann, Product Manager Embedded Open Source at Bosch and Chair of the ELISA Technical Steering Committee (TSC) and Kate Stewart, Vice President of Dependable Embedded Systems at The Linux Foundation and Chair of the ELISA Medical Devices Working Group. Both speakers have extensive experience and knowledge in the fields of embedded systems, open-source software and safety-critical applications.
During the session, attendees will get an overview of the goals and technical strategy of the ELISA project. The presentation will cover the different work groups involved in the project, such as Linux Features for Safety-Critical Systems, Software Architecture, Open Source Engineering Process, Tool Investigation, and Code Improvement. The session will also explore how these work groups interact and contribute to the overall project.
The session will provide information on the methodologies and tools in use, existing challenges, and why the different puzzle pieces are all needed for enabling Linux in safety-critical applications. Attendees will leave the session with a better understanding of where the ELISA project stands today and what comes next. A discussion of how this work can complement other Linaro initiatives will be explored.
Overall, the ELISA project’s attendance at Linaro Connect 2023 offers an intriguing chance to learn more about how the project is making Linux-based safety-critical applications possible as well as vertical use case working groups like Automotive, Medical, and Aerospace. These working groups are aimed at addressing specific challenges related to enabling Linux-based safety-critical applications in their respective domains.
If you’re attending Linaro Connect 2023, don’t miss the ELISA Project talk, especially if you’re interested in functional safety. The session will provide valuable insights into the project’s goals, strategies, challenges, and future plans.
To learn more or register for Linaro Connect, click on the main event website.
Written by Philipp Ahmann, Chair of the ELISA Project TSC (Robert Bosch GmbH) and Sudip Mukherjee, Member of the Tools Working Group (Codethink Ltd)
This article describes how the ELISA Project has enabled Continuous Integration (CI), using the Automotive working group use case as an example. The key goals are to:
Make it easier for others to onboard the project
Experience deliverables from the various working groups
Make the work reproducible and more dependable
Additionally, it describes which elements are part of the pipeline along with the tools involved in the creation and testing of the release images. A remarkable element is that the complete pipeline is coupled to the documented development flow. This means that the CI reproduces the steps a developer would do on a PC.
In this way, it approaches people who are new to safety as well as those which are new to Linux. It is a collaborative approach of multiple working groups within ELISA and should foster more collaboration in upcoming ELISA activities.
Motivational factors that enable the CI
In a collaborative project, people of different interests and technical backgrounds unite to work together. After a successful onboarding process, it is important that contributions and work results become visible to the wider community. They need to be properly basellined and distributed fastly to stakeholders.
A high degree of automation increases the dependability of the development process and reduces failures introduced by human slip. It also gives more time back to developers as they do not have to do everything from scratch.
The ELISA project also relies on external sources for support – for example – the Linux Foundation sister projects Automotive Grade Linux (AGL) and the Yocto Project. The CI guarantees the rebuild whenever updated packages from AGL or Yocto are provided.
In summary, one can say that these demands drive the enablement of the CI within ELISA Automotive WG and put a set of requirements to it:
A entry point for stakeholder with different (technical) background and experience
A respect of different stakeholder interests in the work products
Faster ramp-up for new project contributors
Better and faster integration and distribution of new features and improvements
Automated and dependable software product creation
Extendable testing to validate concepts and assumptions.
With the successful setup of the prototype for the automotive use case, the goal is to scale towards more complex architectures as tackled within the ELISA Systems WG. The basic approach and tools will remain the same in the extension towards new use cases.
The elements currently involved in the CI include:
All elements are coupled and tied together to build a dependency chain, which makes it easier to automatically detect failures, caused by changes along the elements of the pipeline.
The CI elements and flow within ELISA
The ELISA Project uses GitLab to run the CI pipeline, which interacts with an ELISA hosted build server (which is also used for other purposes as running tools for code improvement). The build sources are taken from Automotive Grade Linux Gerrit, Yocto Project repositories as well as the ELISA GitHub repositories.
Currently, the generated build artifacts are the kernel and root filesystem for QEMU x86_64 image. It is planned to extend these to additional images for physical hardware in the future.
While the CI is running in GitLab, ELISA is hosting source code and documentation in GitHub. This is the starting point of the CI flow.
The build is triggered on a regular basis once per day to provide a fresh image and check the functionality of the generated image. Due to the usage of cached build artifacts, the longer build times of a full yocto build are reduced to a minimum (in the range of 15 minutes). The generated images are only kept for one day, so that a new download will always have the latest changes included. This further supports the idea of continuous integration. The latest image can be downloaded from the GitLab project packages.
Also a manual trigger of the CI can be done by referencing a GitHub branch of meta-elisa from a PR. In this way a pre-check from changes is possible before formal images are released from the master. It further reduces eventual regressions. To avoid misuse, access to trigger new builds from referenced GitHub forks is restricted to ELISA working group members.
The strong bound of build server to documentation also makes sure that the docker file for manual build is used to create a docker image, which is then used to create the build artifacts for the QEMU image. Also, this is newly created on a regular basis, to make sure that documentation and tool updates will always result in a working image using the latest documentation.
This is needed to achieve the different entry points for the stakeholders who want to try ELISA artifacts or contribute to the project work.
Dependency to the AGL instrument cluster demo
As already mentioned before, but not directly visible from the build flow, is the dependency to the AGL instrument cluster demo. As ELISA collaborates with AGL, it enhances the existing work of AGL and uses it for demonstrating. The results are still a work in progress; therefore the generated meta-elisa layer is not integrated into the formal AGL repositories (yet). Missing this upstreaming causes the risk that the ELISA automotive WG tell-tale demo will break in case AGL makes changes to interfaces used by ELISA. Similar changes can happen for changes in the yocto-poky base.
By building on a daily basis and testing the generated images for the added functionality, it is guaranteed that any breaking change will become directly visible and immediate actions can take place. In these scenarios the latest image will not be deployed for download and analysis of the reported errors can be done.
As a full yocto build can be resource intensive and time consuming the use of cached build artifacts with an enabled SSTATE helps to reduce the build time to a minimum.
Benefits of using Yocto’s SSTATE cache
Building a yocto AGL image from scratch is taking time and a lot of computation performance. At least 100GB of disc space is consumed and a build can easily take several hours to complete even on decent hardware.
In order to reduce the build time and limit the necessary disc space, it is possible to turn on cached build results. The so-called SSTATE is created with the first full build and shared for upcoming builds, demanding to only build deltas. Any change in sources and updates are detected and necessary binaries and their dependencies will be recompiled in case needed.
The generated cache is used within the CI, but can also be used for a local build at a developer PCc.
Additionally, from time to time a new cache is generated to check that the build will also work in case someone builds the system from scratch. Building from scratch can make sense for low network speeds or networks with traffic limitations. This further makes sure that the GitLab pipeline fits to the documentation as much as possible.
GitLab pipeline coupling to the meta-elisa documentation
The GitLab pipeline basically represents the steps which users would take when manually creating the ELISA enhanced demo of the instrument cluster provided by Automotive Grade Linux. As the current version is based on QEMU it is easy to reproduce the demo at a local machine and also to perform automated testing. This is illustrated by transferring the created images to the OpenQA server hosted by Codethink.
OpenQA testing
It is important to confirm that the build was successful. As the original AGL demo has been enhanced with the Signal-Source-Control application, which also renders additional screen content, it makes sense to do an output comparison. For the meta-elisa demo OpenQA was used as a proven tool to check proper boot and functionality of the code modifications. This should also avoid potential regressions, e.g. due to external dependency or interface changes. An example output of a comparison as generated by OpenQA can be found below.
In the OpenQA tool, the orange divider in the presented screenshot can be moved left and right with the cursor. This makes it easy to identify differences between reference and recent execution. The greenish frame shows the region of interest which is checked for content. The 100% shows the fit ratio.
Additionally, during the test execution a screen recording is performed and further logs such as those from serial terminal are provided.
With this option, a fully automated flow from source repository to generated and tested image is achieved. It is possible that working group members can fork the existing meta-elisa repository on GitHub, do modifications, check that it is built properly using GitLab and that the image remains functionally by checking the OpenQA result. This flow would be completely web based without the demand of even having a development environment setup locally on a PC.
However, this may more be used as an exceptional case. The interaction with the system to understand and learn is rather limited in this scenario. As an intention is to develop the system further and gain a larger understanding of the demand of a safety-critical system with Linux based workloads, there are many more entry points for different levels of expertise and personal interest.
The different entry points to experience the automotive use case
In order to make it easier for new contributors to join the ELISA Automotive WG, the demo has multiple entry points.
Closing the loop to the motivation, the aforementioned entry points fulfill the demands of a collaborative project, where people of different interests come together to collaborate. It is possible to understand the system creation from scratch with an own build from source, or just trace a workload by using the pre-built QEMU images. The development flow is accelerated by providing docker images and cached build artifacts. As the outputs of different pipeline steps are available, debugging gets simplified as well (like checking a native build against a docker build or behavior of the own vs. generated QEMU image).
In summary, a comprehensive documentation (including this post) helps to tailor the pipeline and images to individual demands. Though, to have the best compatibility for exchange with others, the usage of a docker container is recommended.
Docker container for better compatibility of CI and manual reproduction
Taking into account the large number of different host machines and Linux distributions that exist, from practical experience the ELISA CI is providing a docker file as a common base for development. The docker file is used to generate the docker image, which again is used to build the meta-elisa. By providing the docker file and docker image also to others, who want to create a local build on their machine, influence of the underlying Linux distribution is reduced. Documentation about the docker container creation and usage can be found in the wg-automotive github repository. To pull the docker container directly use:
Some general remarks regarding container should be mentioned for competition:
There is still a (limited) dependency to the underlying OS outside the container, as the docker image uses for example the Linux kernel from the host system. However, side effects of missing environment setups, conflicting or overwriting packages are minimized by using the docker container.
Regarding build time and performance the impact of the container could be neglected. The duration and system load during the build with and without container usage was very similar.
The docker file will be kept up to date to reflect demands as per meta-elisa or dependency from yocto or AGL updates. This makes sure to always have the latest features and improvements also for the planned next steps reflected for better reproduction of results.
Outlook and next steps
As an evolution of the use case, activities have been started to enable a reference image of an example architecture including Xen and Zephyr on real automotive hardware. Any result from the automotive as well as other working groups within ELISA should feed into the example architecture tailorable to specific use case demands.
Additionally having a proper software bill of material (SBOM) generation will also enhance the quality of the reproducible demo. It will support upcoming regulations which require an understanding of the content of your software product. As AGL also works on the reproducible build to generate binary compatibility of builds this may be an additional enhancement considered as future activity.
Summary
✔ By setting up the CI with QEMU image deployment to the openQA testing it is possible to quickly track changes and validate code changes for proper appliance along the chain from source to deployment.
✔ Active contributors in ELISA have the chance to check their own commits or pull requests by using a standardized GitHub development flow and start the CI by a two line change in the pipeline reference to the meta-elisa fork and branch commit.
✔ The different entry points and precompiled build results enable interested people to decide to build an image from scratch, use a docker container to reduce environment setup steps and reduce build time to a fraction by taking cached build artifacts from the server.
✔ By provisioning the final QEMU build image, it is even possible to access and boot the generated image without the need of performing an own build.
As all pipeline elements from documentation to later qa testing are dependent on each other and are coupled to an integrated flow. A high degree of dependability and reproduction is achieved with this solution. With a future enhancement of SBOM generation and utilization of reproducible builds project, the system dependability will further be increased. The automated execution supports that aligned development flows are kept and regressions can be tackled before they occur. Best entry points for reproduction of the results are the meta-elisa and wg-automotive docker container documentation in GitHub.
Written by Shuah Khan, Linux Fellow at the Linux Foundation and member of the ELISA Project TSC, and Shefali Sharma, a senior student at the Meerut Institute of Engineering and Technology in India and ELISA Project Mentee in 2022
Key Points
Understanding system resources necessary to build and run a workload is important.
Linux tracing and strace can be used to discover the system resources in use by a
workload. The completeness of the system usage information depends on the
completeness of coverage of a workload.
Performance and security of the operating system can be analyzed with the help of tools such as:
Once we discover and understand the workload needs, we can focus on them to avoid regressions and use it to evaluate safety considerations.
We set out to identify methodology to discover system resources necessary to build and run workload and document the process. Easily available workloads are used to test out methodology and to document the process. This process is intended to be used as a guide on how to gather fine-grained information on the resources in use by workloads.
Within the ELISA project the methodology goes into scaling, by hardening and extending the guidelines in other working groups. The achievements from the Medical Devices Working Group will be taken and put into practice within the Automotive WG use case analysis focusing on safety critical display rendering.
The workload-tracing guide is now available in Linux 6.3 release. It is intended to be used as a guide on how to gather fine-grained information on the resources in use by workloads using strace.
As open source is found more and more in safety-critical applications, the need to evaluate open source software that meets safety standards has increased. The Safety-Critical Software Summit, sponsored by the ELISA Project, will take place on June 27-30 in Prague, Czech Republic and virtually. The event gathers safety experts and open source developers to enable and advance the use of open source in safety-critical applications.
If you are new to the ELISA Project, here is what you need to know. ELISA members are defining and maintaining a common set of elements, processes and tools that can be incorporated into specific Linux-based, safety-critical systems amenable to safety certification. ELISA is also working with certification authorities and standardization bodies in multiple industries to establish how Linux can be used as a component in safety-critical systems. This is the first ever Safety-Critical Software event under the Embedded Open Source Summit (EOSS).
Here is a glimpse to the Summit topics:
Open Source Software Interaction with Safety Standards
Best Practices to Work with Regulatory Authorities when using Open Source
Best Practices for Security Updates to Safety-critical Systems
Safety Engineering Applied to Open Source Projects
Quality Assessments based on Data from Development
Development of Technical Features with Safety Relevance
Safety Analysis Approaches and Methodology to Apply to Systems
Case Studies of Existing use of Linux in Safety-critical Domains
To attend, register for Embedded Open Source Summit. 1 registration gives you access to ALL events featured under the Embedded Open Source Summit umbrella. In-person and virtual registration are both available. Register here.
Together with the Embedded Open Source Summit, Safety-Critical Software Summit provides a foundation for collaboration between these embedded development communities. The format will include presentations, BoFs, training, workshops and min-confs designed for real time problem solving and deep discussion.
To get an overview of the event, such as the important dates and the topics that will be covered, click here.
FOSDEM aka Free and Open Source Software Developers European Meeting is a non-commercial, volunteer-organized European event centered on free and open source software development that is aimed at developers. FOSDEM is held annually, usually during the first weekend of February, at the Université Libre de Bruxelles Solbosch campus in the southeast of Brussels, Belgium.
Grand Place, Brussels
CNCF was an official sponsor and experts from various Linux Foundation projects came together to give keynotes and co-host devrooms including Embedded, Mobile & Automotive, Energy and SBOM.
In case you haven’t met the newly launched Linux Foundation Europe team, we invite you to read the latest blog “On the Road in February 2023.”
Philipp Ahmann, Chair of the ELISA Technical Steering Committee and Technical Business Development Manager at Robert Bosch GmbH, gave a presentation at FOSDEM titled, “The ELISA Project – Project insights and overview”. The lecture mainly focused on the goals and technical strategy of the project. It provided information about the different work groups, their interaction, and contributions.
ELISA members are working together to define and maintain a common set of tools and processes that can help companies demonstrate that a specific Linux-based system meets the necessary safety requirements for certification. These existing working groups focus on Linux Features for Safety-Critical Systems, Software Architecture, Open Source Engineering Process, Tool Investigation and Code Improvement. They are complemented by vertical use case working groups dealing with Automotive, Medical, and Aerospace.
During the talk, Philipp explained in detail about each working group. Safety Architecture group monitors the kernel for non safety related issues such as the watchdog kernel analysis and try to improve using tools and documentation and it is driven by use cases and demands of the products.
Linux features for safety critical systems group mainly work on the safety criticality part and not the generic features. The main goal is to look for the elements which could improve safety by doing incremental steps to make the system more dependable and robust.
While doing these processes, code quality is very important and hence the tool investigation and code improvement group. The WG uses tools and CI servers to identify the kernel issues and to make the kernel more reliable and robust.
Open Source Engineering Process WG was developed due to the challenges in Engineering. There are rigorous methods within kernel development. There is a strong demand from traditional project management when it comes to safety products. We can see that not every process compliance agrees with it directly. We need to find an equivalent to the open source development process compared to ISO 26262 requests for automotive products.
Systems work group works cross functional and cross projects and combine these elements involved in the above mentioned processes.
In order to tailor the systems accurately with these groups, ELISA Project has vertical use cases such as Aerospace, Automotive. Medical etc.
He also explains how ELISA Project is interacting with other communities such as Xen Project, Zephyr Project on safety critical topics, Automotive Grade Linux, SOAFEE and SDV on automotive use cases and Yocto project for build tooling and SBOM generation and SPDX for system SBOM data requirements.
In addition to the different work groups and their interactions Philipp also explained on the contributions of the different groups, use cases, used methodologies such as STPA and workload tracing and a status update on what to expect from ELISA Project.
As Philipp mentioned, based on George Bernard Shaw’s quote “If you have an apple and I have an apple and we exchange these apples then you and I will still each have one apple. But if you have an idea and I have an idea and we exchange these ideas, then each of us will have two ideas” we need to get a good understanding and bring things together.
Ready to set your career on an award winning path in February? Make the commitment to invest in yourself and your career today with Linux Foundation training and certifications.
Now through the end of February we are offering serious discounts on our most popular training and certifications, with even bigger discounts on Bootcamps and Bundles:
Not sure which is the best offer for your career? Check out our IT Career Roadmap to see how Linux Foundation certifications guide your career toward: Systems & Infrastructure, Network & Security or Software Development.
Işıl Öz, 2022 ELISA Project Mentee and Assistant Professor atIzmir Institute of Technology
Hardware Trusted Execution Environment
Since remote computation has evolved as the cloud infrastructures offer high-performance and low-cost solutions, security has become a concern due to remote computer’s mainte- nance by untrusted parties. As the security mechanisms aim to maintain protection from attackers and enable execution confidentiality and integrity, they help to achieve safety requirements as well by making sure that both program code and data are not corrupted. Therefore, memory protection solutions support safety by ensuring trusted execution, and they target domains requiring both secure and safe execution [4, 15]. Homomorphic en- cryption, which directly works on the encrypted data, enables the handling of user data by third parties safer. However, its computational requirements lead to impractical perfor- mance issues.
Trusted execution environments (TEEs) maintain secure computation in an isolated area of a processor with hardware-assisted technologies. They offer both performance and security improvements by exposing a smaller trusted compute base (TCB) in the environment. Confidential Computing Consortium (CCC) defines TEE as an environment that provides a level of assurance of data integrity, data confidentiality, and code integrity [19]. Data integrity is preventing unauthorized entities from data modification while in use inside TEE. Data confidentiality means that unauthorized entities cannot view data while in use inside TEE. Code integrity is preventing unauthorized entities from code modification while it is being executed inside TEE. Depending on the implementation, TEE additionally may provide the following features:
Code confidentiality: TEE protects code while in use from being viewed by unautho- rized entities. For instance, an algorithm having intellectual property issues should not be exposed to third parties.
Authenticated launch: TEE enables the launching of a process when only it is verified to be secure.
Programmability: TEE consists of any arbitrary code or code that is loaded by a secure source.
Recoverability: TEE provides a recovery mechanism that resets the status from a corrupted state to a known safe state.
Attestability: TEE measures its origin and current state and maintains evidence that the code has integrity and has not been modified. Different hardware vendors offer TEE implementations in their processor architectures:
Intel Software Guard eXtensions (SGX): With SGX, which exists in Intel Xeon pro- cessors, applications run in a secure memory space without access by the untrusted operating system (enclave) and store the data it uses in this memory area.
ARM TrustZone: TrustZone is in Arm Cortex-A processors and provides a secure environment for memory, software, bus transactions, and interrupts. The applications running on TrustZone can perform trusted operations with API functions.
AMD Memory Encryption Technology: While AMD Secure Memory Encryption (SME) is a hardware-based secure memory encryption technology found in AMD EPYC processors, Secure Encrypted Virtualization (SEV) is the implementation of SME technology used in virtualized systems. We focus on SGX technology and open-source library operating systems that enable its functionality.
Intel Software Guard Extensions (SGX)
Intel’s Software Guard Extensions (SGX) provide hardware-based isolation for secure com- putations by supporting safe execution [18]. SGX reserves isolated memory regions called Processor Reserved Memory (PRM) or enclaves for code and data. The CPU protects the memory regions from all non-enclave memory accesses, including the kernel, hypervisor System Management Mode (SMM) accesses, and DMA accesses from peripherals. Non- addressable memory pages (Enclave Page Cache (EPC) pages) inside enclaves are reserved from the physical RAM and encrypted. Execution flow can only enter an enclave by special CPU instructions, similar to the mechanism for switching from user mode to kernel mode. Enclave execution always happens in protected mode, at ring 3.
Figure 1 presents an example execution flow for the applications utilizing protected memory regions. The initial code and data in an enclave are loaded by untrusted system software. During the loading stage, the system software asks the CPU to copy data from unprotected memory into the enclave and assigns the pages to the enclave. After all the enclave’s pages are loaded, the system software asks the CPU to mark the enclave as initialized, at which point the application can run the code inside the enclave with support for high-level language features. Other parties can perform a software attestation process to verify the identity of the enclave that it is communicating with an enclave and is running in a secure environment.
Open-source LibOS Projects
The library operating system (LibOS) technology enables us to take an application with little or no modifications and protect it in an SGX enclave [7]. While commercial and open-source LibOS options are available, we mention open-source projects and focus on a specific project, Mystikos, our mentorship is built on. The following open-source LibOS projects support Intel SGX:
Gramine [6]: Lightweight libOS, designed to run a single application inside enclaves with minimal host requirements.
Occlum [13]: Memory-safe, multi-process library OS (LibOS) for Intel SGX.
Intel SGX and Linux kernel library [16]: Designed to run existing unmodified Linux binaries inside of Intel SGX enclaves.
Mystikos [12]: Tools and runtime for launching unmodified container images in TEEs.
Mystikos
Mystikos [12] is a runtime and a set of tools for running Linux applications in a hardware- trusted execution environment (TEE). It currently supports Intel SGX. Mystikos aims for the following features: 1) Enable protection of application code and data while in memory through the use of hardware TEEs, 2) Allow users and application developers control over the makeup of the trusted computing base (TCB), 3) Simplify retargeting to other TEE architectures through a plugin architecture.
As shown in Figure 2, Mystikos includes the following components:
C-runtime based on musl libc: Handles passing the runtime functions from unsecure part to the secure part of the application.
LibOS-like kernel: Implements system calls for secure execution.
Kernel-target interface (TCALL): Communicates with the target implementations (SGX target based on Open Enclave SDK or Linux target for only verification on non-SGX platforms).
Command-line interface: Offers an interface for the programs targeting execution inside the enclave (myst exec-sgx or myst exec-linux).
Some related utilities: Tools for building an application for secure execution (myst-appbuilder), creating an archive for target program (myst mkcpio or myst mkext2), packaging an application (myst package-sgx). Figure 2: Mystikos architecture [12]. Figure 3 presents the application workflow when developing application in Mystikos, which consists of four steps:
Figure 3 presents the application workflow when developing application in Mystikos, which consists of four steps:
Develop application: Develop the application using programming models of our choice.
Create appdir: Create and populate an appdir directory that contains compiled application and essential dependencies.
3. Run in Mystikos: First create a root file system (rootfs) from appdir by using myst mkcpio or myst mkext2. Then run the application by using myst exec-sgx or myst exec-linux.
4. Package application: Sign and package the application.
Activities During the Mentorship
At the beginning of the mentorship period, I investigated the open-source libOS projects enabling and implementing SGX features (mentioned in Section 3). With a theoretical background in operating systems and processor architecture, I tried to understand the ex- isting implementations. In the meantime, I learned about some practical issues by following the LFX courses about Linux kernel and open-source project development (like GitHub is- sues and pull requests) to get hands-on experience. Specifically, I enrolled two LFX training courses: A Beginner’s Guide to Open-Source Software Development (LFD102) [3] and A Beginner’s Guide to Linux Kernel Development (LFD103) [2].
After deciding to work on Mystikos project, which is closely related security and safety issues, I started by examining open issues in Mystikos Github repository.
The first issue I have looked at was the following: Memory leak in myst syscall execve ([11]): It was about a potential bug in the function myst syscall execveat inside the file /kernel/syscall.c. The reason seemed to have an unfreed memory allocation in the function myst exec inside /kernel/exec.c file. After discussion with the maintainers re- garding the issue, I created a pull request and submitted it as a fix (Figure 4). However, the maintainers realized that the issue has been already fixed in the previous commits, so my commit has not been merged.
After being introduced in Mystikos and getting familiar with the code, I contacted to the maintainers. Vikas Amar Tikoo replied and offered me to work on improving debug-gability by either writing gdb extensions, improvements to strace mechanism in Mystikos, interactive shell support, or integration with native debugging tools like valgrind. Then he created an issue on github that I can work on: Interactive bash support issues ([9]). The aim is to run bash shell in Mystikos. However, Mystikos complains about it. I could replicate the case in my local machine (Figure 5). The problem seems that the ioctl requests/commands [10] have not been supported by Mystikos. I submitted a pull request with TIOCGPGRP and TIOCSPGRP ioclt system calls (Figure 6). I have been working on the issue and communicating with the maintainers about the implementation details.
Additionally, during my mentorship period, I attended the following online activities:
SGX Community Day Virtual Event, July 26-27, 2022 [1]: Inspirational talks from academics, startups, and big corporations about SGX implementations.
ELISA summit, September 7-8, 2022 [5]: Introductory overview, emerging trends, and current topics in utilizing open-source software for safety-critical applications.
SOAFEE Virtual Symposium, November 16-17, 2022 [17]: Presentations by open- source and independent software vendors organized by the Scalable Open Architecture For Embedded Edge (SOAFEE) Special Interest Group.
Moreover, as a researcher working in the field of computer architecture, the mentorship helped me to improve myself in the hardware security area. I have submitted a national research project titled as Security-Performance Tradeoff Analysis for Embedded Systems with GPUs, where we target to analyze both the security and performance of the embedded applications running inside enclaves. We aim to utilize the open-source LibOS projects and port the program executions partially into the enclaves. Mainly, we target to extend the mentorship work for safety-critical embedded CPU-GPU systems. Additionally, my mentorship encouraged my students to work on open-source projects and be included in mentorship programs.
Thanks to the LFX mentorship program supported by ELISA, I found an opportunity to be introduced to the open-source software community and gained a practical hands-on experience in open-source software development. The mentorship was helpful in terms of both enhancing my hands-on skills and learning the open-source project development pro- cess. While I have a strong background in theoretical operating systems and computer architecture area, especially my operating systems software skills have substantially im- proved. Moreover, the regular meetings with my experienced mentor and communication with the open-source project maintainers were very helpful for both technical and social skill development.
After getting experience on the Mystikos project, I am working on the issue and will continue working on it after my mentorship period. With this experience, I am planning to contribute to other open-source LibOS projects by combining my research interests and practical experience. After being introduced to the open-source community, I believe that I will be able to find more opportunities in the related domains requiring both security and safety. Since I learned that secure execution environments/containers maintain secure and
Safe execution inside enclaves, we can utilize them for different applications in safety-critical systems.
References:
3rd sgx community day virtual event. https://community.intel.com/t5/Blogs/ Tech-Innovation/Data-Center/Third-SGX-Community-Day/post/1393177. Ac- cessed: 2022-11-16.
A beginner’s guide to linux kernel development (lfd102). https://trainingportal.linuxfoundation.org/learn/course/ a-beginners-guide-to-linux-kernel-development-lfd103. Accessed: 2022- 11-16.
A beginner’s guide to open source software development (lfd102). https://trainingportal.linuxfoundation.org/courses/ a-beginners-guide-to-open-source-software-development-lfc102. Accessed: 2022-11-16.
Eclipse software defined vehicle working group. https://sdv.eclipse.org/. Ac- cessed: 2022-11-23.
Victor Costan and Srinivas Devadas. Intel sgx explained. Cryptology ePrint Archive, 2016.
A Publication of The Confidential Computing Consortium. Confidential computing: Hardware-based trusted execution for applications and data. Technical report, January 2021.
