Recently, Philipp Ahmann, an ELISA Ambassador and member of the Technical Steering Committee and Technical Business Development Manager at Bosch, had the chance to speak at the Lund Linux Con about the technical strategy of the ELISA Project, the established work groups as well as the work of the Automotive Working Group.
The Lund Linux Con is a Linux Kernel focused conference in the south of Sweden, typically taking place in May every year. More than 100 participants attended the in person event sponsored by Axis Communications, Western Digital and Volvo Cars to meet and exchange about various Linux Kernel related topics. You can watch Philipp’s presentation below or check out his slides here:
WrittenPhilipp Ahmann, ELISA Ambassador and TSC member and Business Development Manager at Robert Bosch GmbH
Many projects share architectural elements, including container technologies, RTOS requirements, or virtualization, but safety concerns mainly touch single-point elements of such a system rather than sticking everything together. This construction of full system architectures is created by distributors or within companies towards a product. Due to their complexity, these systems utilize the work results of various related projects, either open source or proprietary.
Although the use of these systems heavily differs, their system architecture elements are repeating. As mentioned, the embedded systems world often ends up with a heterogeneous architecture mixed with RTOS and Linux, virtualization and containers. This is true for medical, industrial, automotive and other industries. These systems have to fulfill certain compliance requirements, including a proper Software Bill of Material (SBOM). It is needed to tailor them for various product lines, to enable a quick project start, reduce maintenance and training efforts and ensure faster product ramp ups. Lastly, these systems typically connect with cloud services for features like monitoring, over the air update, or feature extensions, to increase their maintenance time and to architect higher care for safety as well as dependability, reliability and safety.
Working Groups, Project interaction and Architecture
The Systems WG explicitly encourages the collaboration of companies and OSS projects to develop a reference system as a workbench. It will target a community, which either also works toward enabling safety use cases with open source software or which plans to make use of open source mixed-criticality system elements as a base for their product lines.
Currently, the ELISA project already interacts with following community projects:
Taking these puzzle pieces from different projects and bringing them together in a reproducible showcase with goals such as proper SBOM generation is not established in any other open source project so far. There is a lack of an umbrella work group within the extended embedded and edge domain, which provides a reference architecture and a reproducible, dependable system to serve as a blueprint to explore new use cases, functions, or modules. It goes well in line with the recent trend toward software defined vehicles and the first steps into software defined industries, but is not limited to this.
Although the reference architecture may include elements like hypervisors or containers, which have explicitly not been part of ELISA so far, it should enable other working groups to showcase their work on process, features, and tooling with the implementation of a reference system. It can also create ideas and show potential risks or hazards due to a better system understanding; a systems-based understanding of an architecture later on used in a similar way in real world products.
As an important remark, the working group starts with a reference system fully based on “as- is” open source technology. Only a few of the elements contained in the reference system have been developed with considerable safety in mind. This means the created system will act as an example and stimulus environment for mixed criticality Linux based use cases, but is not at all safe or certifiable.
Instead, it should enable users to exchange certain elements like the container technology, RTOS, or hypervisor technology. This will require a modular design, good documentation and a way to reproduce the system with easy to follow steps.
Next steps and Beyond
The starting point for the reference system is based on the work Stefano Stabelini presented during the Open Source Summit North America, hosted in Austin, TX, June 21-24, called “Static Partitioning with Xen, LinuxRT, and Zephyr: A Concrete End-to-end Example”. As a next step, the system architecture presented in this tutorial will be further extended by adding the Yocto Project as build tooling for all system elements and enhancing the base Linux operating system by adding Automotive Grade Linux towards the end of this year. A full SBOM shall be generated as well.
Through early 2023, it is planned to extend the system to a physical hardware next to the existing QEMU image, and to also to add container technology, along with cloud connections represented. By doing so, it is possible that a Debian-based Linux may be used as a guest VM or within a container to pick up a wider community.
The work will be documented in a way that showcases why and how the system is configured as it is, while the underlying tooling will enable reproducibility by a few steps as checkout, build and deploy.
Summary
Along with proper documentation on tailoring and reproduction of the showcase, the reference system can serve as a workbench to challenge concepts and implementations from ELISA and other open source projects. It can be used as a quick start to test new ideas during proofs of concept and to let others easily experience achievements of the ELISA project working groups dealing with Linux features, tooling, architecture and more. It is there to foster collaboration of a wide range of open source projects and enables learning from real world scenarios close to later product line architectures.
Join us
The new Systems WG hosts weekly meetings on Mondays at 15:00 UTC (8 am PT/11 am ET). To receive the meeting invitation (and working group posts), simply subscribe to the mailing list. The meeting invitation will be sent to you after subscription. Also feel free to reach out to one of the ELISA ambassadors to learn about further ELISA activities.
In March, the ELISA Project launched the Monthly Seminar Series, which focuses on hot topics related to ELISA and its mission. Presenters are members, contributors and thought leaders from the ELISA Project and surrounding communities.
For June, Khalid Aziz, Senior Software Engineer from Oracle, gave a presentation titled, “What is a page table and why should we care about it?” Physical memory on a computer is a shared resource. Kernel allocates and reclaims this shared resource to ensure all workloads on a system have adequate resources to complete their tasks. Accurate management of physical memory assignment is required for fairness and data safety. This is the task of Memory Management subsystem in the Linux kernel. We will discuss how the Memory Management subsystem assigns and tracks physical memory, how it ensures isolation between unrelated workloads and how it enables sharing of data in memory for workloads that need it. We will discuss the management overhead associated with isolation and dive deeper into specific approaches on managing overhead for effective sharing.
The March seminar focused on the Real-time Linux Analysis Toolset. ELISA community member Daniel Bristot De Oliveira, Senior Principal Software Engineering at Red Hat, presented the tools provided by rtla. You can find out more or watch the video here.
The Spring ELISA Workshop, which took place on April 5-7 virtually, had more than 130 global registrants that learned more about the various working groups, hot topics related to enabling linux in safety applications and networked with ambassadors. If you missed the workshop, you can check out the materials here or subscribe to the new ELISA Youtube Channel and add these sessions to your watch list.
Christopher Temple, Lead Safety & Reliability Systems Architect at Arm Germany GmbH, and Paul Albertella, ELISA Project TSC member, Chair for Open Source Engineering Process Working Group and Consultant at Codethink, presented a session, “Mixed-Criticality Processing on Linux.”
Check out the video that features the presentation and community discussion about how to create a common understanding of mixed-criticality processing on Linux and the related problems, collect and discuss alternatives for addressing the problems.
In the video, you’ll see there is good engagement from existing ELISA participants and new ones, and the group ended with a clearer understanding of the challenges faced when safety functions co-exist on a system with non-safety functions, and with other safety functions. There was a broad consensus about how ELISA might provide useful guidance for how to tackle some of these, by describing design patterns for systems that include Linux, rather than focussing on what Linux needs in order to be ‘safe’.
The Spring ELISA Workshop, which took place on April 5-7 virtually, had more than 130 global registrants that learned more about the various working groups, hot topics related to enabling linux in safety applications and networked with ambassadors. If you missed the workshop, you can check out the materials here or subscribe to the new ELISA Youtube Channel and add these sessions to your watch list.
Gabriele Paoloni, ELISA Project Governing Board Chair and Senior Principal Software Engineer at Red Hat, and Daniel Bristot, Senior Principal SW Engineer at Red Hat, gave a presentation at the Spring ELISA Workshop titled, “Safety Monitors Inside the Kernel.”
The recently proposed “Runtime Verification Monitor” framework, which can be found here, has the capability of monitoring the Kernel Drivers / Subsystems to behave as expected and to protect them against interference from within the Kernel itself. The video will explain how the RVM framework works with a specific focus on the Watchdog Monitor that has been proposed in the patchset and how it can support a functional safety claim. Watch it here:
The Spring ELISA Workshop, which took place on April 5-7 virtually, had more than 130 global registrants that learned more about the various working groups, hot topics related to enabling linux in safety applications and networked with ambassadors. If you missed the workshop, you can check out the materials here or subscribe to the new ELISA Youtube Channel and add these sessions to your watch list.
For the first-time ever, the ELISA Project featured a keynote presentation. Robert (Bob) Martin, Senior Principal Engineer at the MITRE Corporation, presented a keynote titled, “Software Supply Chain Integrity Transparency & Trustworthiness and Related Community Efforts.” Check out the presentation materials here or watch the video:
Trust, transparency, and integrity of software supply chains is at the center of many of the global security and safety challenges confronting communities around the world, including government agencies and the industries that support them or provide our critical infrastructure. The pandemic, utility ransomware attacks, the attack on SolarWinds, and the Ever Given have brought supply chain security, resilience, integrity, transparency, and trustworthiness into sharpened focus to a broader audience, and the many inadequacies have surfaced regarding timely access to reliable suppliers, software, and stocks of fuel, personal protective equipment, micro-electronics, medical devices, and food supplies, to name a few.
At the same time, the computerization of everything gave rise to pervasive cyber threats for more and more of the capabilities and infrastructure we and our organizations rely upon to function – including those stemming from vulnerabilities inherent in repurposed software of often dubious provenance and unknown pedigree. Further complicating this landscape is the increasingly globalized nature of the technology in these systems and lack of transparency. Adversaries large and small seek to inject themselves into every conceivable stage of software technology development, supply, and support, for disruptive, monetary and intelligence goals of their own.
This video will discuss the capabilities emerging across industry and government to assess and address the challenges to providing trustworthy software supplies with assurance of integrity and transparency to their composition, source, and veracity – the building blocks of software supply chains we can gain justifiable confidence in at scale and speed.
Written by Paul Albertella, ELISA Project TSC member, Chair for Open Source Engineering Process Working Group and Consultant at Codethink
The ELISA Project hosted its annual Spring Workshop on April 5-7. It’s a combination of interesting talks and productive working sessions on Enabling Linux in Safety Applications. I’ve attended a lot of these over the past three years, but for this one there was a perceptible shift towards applying techniques and building solutions. If you couldn’t attend, here’s a quick recap of the workshop.
Day 1 opened with an interesting session from Red Hat’s Daniel Bristot and Gabriele Paoloni titled “Safety Monitors inside the Kernel” about a Realtime Verification Monitor concept, which they have been implementing for inclusion in the Linux kernel. This involves using ‘deterministic automata’ (definition here) modules within the kernel that are generated from directed graph models (defined using Graphviz/dot), which are then driven by events within the kernel using ‘instrumentation’ akin to systrace. The goal is to verify the expected behaviour of specific safety-related functionality within the kernel at runtime, and provide a way to trigger a ‘safe state’ mechanism if a problem is detected.
This was followed by a discussion about whether we should add a new ELISA working group with a focus on industrial and/or IoT safety use cases, led by Philipp Ahmann with Bosch GmbH. This discussion aimed to settle opinions about whether ELISA is a proper place to start such a working group, or if there are better communities to reach out to. Watch the video here:
Gabriele Paoloni then gave a talk about PAS 8296, which is an ISO initiative to provide more detailed guidance on applying the 26262 standard to pre-existing software.
On Day 2, Red Hat’s Christoffer Hall-Frederiksen and Gabriele Paoloni were back again, talking about the work that they have done to document how Linux manages address space integrity. This was very informative, providing an accessible overview of how Linux manages processes, threads and memory.
After that, Alessandro Biasci, Raffaele Giannessi and Fabrizio Tronici from Huawei talked about their use of STPA to analyse dynamic memory functionality for Linux, and how some of the risks they identified might be addressed using memory tagging.
I then did a talk on ‘Refining the RAFIA Approach, which addresses the challenges of creating safety argumentation and supporting evidence for systems involving open source software. This talk provided an update on how this approach is being applied and refined, both as part of ELISA workgroup activities and in Codethink’s projects.
The last session of the day focused on the Kernel Configuration database and was presented by Elana Copperman with Mobileye and Wenhui Zhang from Bytedance. This was an attempt to gather together information available on various kernel configuration items that may be relevant when addressing particular risks, together with some notes on best practice when using these.
This was originally structured using Common Weakness Enumerations (CWEs), and there are plans to include security-related configs as well. One topic of discussion was extending it to include information about the performance impact of the configs, and building automated setups to measure this for reference ‘instances’. At present the ‘database’ itself is just a big Google spreadsheet, but there’s potential for this to become a useful resource. I have been prototyping a solution to migrate the content to a GitHub repository and render it as web pages, which may help.
Day 3 was dominated by two more working sessions. The first, on Mixed Criticality Processing, was a productive discussion which I co-chaired with Chris Temple from Arm. There was some good engagement from existing ELISA participants and new ones, and we ended with a clearer understanding of the challenges faced when safety functions co-exist on a system with non-safety functions, and with other safety functions. There was a broad consensus about how ELISA might provide useful guidance for how to tackle some of these, by describing design patterns for systems that include Linux, rather than focussing on what Linux needs in order to be ‘safe’.
The second session, which I also co-chaired with Philipp, focussed on a possible example of such a pattern, which has been developed for the Telltale use case by the Automotive Working Group using STPA. We talked through the safety concept and control structure, and how we might build on this and implement a reference version based on an existing AGL demo, perhaps using a readily-available hardware platform such as Raspberry Pi.
The last session of the conference is always a working session for next steps and goals that is led by the Linux Foundation’s Shuah Khan, Chair of the ELISA Technical Steering Committee. Shuah and the TSC are currently putting together insights from that discussion, so stay tuned to learn more about the focuses for next quarter.
If any of these pique your interest, you can go to the new ELISA Project Youtube Channel to see some of these videos or click on the schedule to check out the PPT presentations.
The Linux Foundation has had a robust mentorship program for years that invests in new talent and diversity that helps the open source community – no matter what the focus or project – thrive as a whole. Since its formal launch in 2019, the LFX Mentorship has graduated more than 190 mentees and has hosted almost 100 mentorship programs.
This Spring, the ELISA Project is hosting two mentorships that will help developers gain real-world knowledge in a hands-on learning experience with Linux and open source. It also provides a more defined path for ELISA to connect with the next generation to inject more talent into their developer base.
The Spring Mentorship session, which kicked off in March, paired mentees with leaders from Codethink, the Linux Foundation and Mobileye. The ELISA Project is excited to welcome Irenge Jules Bashizi, Shefali Sharma and Wenhui Zhang as the newest mentees in the ELISA community.Please see below for more details about their mentorships and mentors. As they settle into their new roles, we hope to feature their mentorship journey in upcoming blog posts.
To make eBPF programs “safe”, the Linux kernel validates all eBPF code before loading. However, the current validator has many known limitations, leading to rejection of working programs.
Focus in this mentorship will be:
In-depth analysis and review of the eBPF validator, and its use to validate eBPF programs.
Code enhancements to the validator to improve usability.
Identify use cases for kernel profiling in safety critical applications.
Elana Copperman
Mentor: Elana Copperman, Chair of the Linux Features for Safety-Critical Systems Working Group and System Safety Architect at Mobileye (part of Intel)
Elana provides support for designing safety features in Mobileye products, including system boot; drivers; and Linux infrastructure. Before working at Mobileye, she worked as a Security Architect for Cisco-Il (formerly NDS) and more recently as a security consultant for major European automotive concerns on behalf of various Israeli startups. Research interests focus on software engineering methodologies and security engineering.
In particular, focusing on expanding open source and Linux-based tools to support safety critical and life saving product development.
Irenge Jules Bashizi
Mentee: Irenge Jules Bashizi
Jules is a Computer science student at University of Manchester. He is a certified Linux System administrator. Jules is interested in improving his skills in Kernel engineering by contributing to the Linux Kernel community by submitting patches. This internship offers him a unique opportunity tailored to improve and contribute. As a hobby, Jules enjoys jogging..
OpenAPS is an open source Artificial Pancreas System designed to automatically adjust an insulin pump’s insulin delivery to keep Blood Glucose in a safe range at all times. It is an open and transparent effort to make safe and effective basic Automatic Pancreas System technology widely available to anyone with compatible medical devices who is willing to build their own system.
What happens when an OpenAPS workload runs on Linux? What are the subsystems and modules that are in active use when OpenAPS is running? What are the interactions between OpenAPS and the kernel when a user checks how much insulin is left in the insulin pump?
The ELISA Medical Devices Working Group set out to answer these questions. Understanding the kernel footprint necessary to run a workload helps us focus on the subsystem and modules that make up the footprint for safety.
The mentee will:
Use Linux kernel tracing and strace tool to discover Linux kernel subsystems used by OpenAPS.
Find Linux system calls supported on various architectures.
Write a blog/whitepaper on the findings which will aid ELISA Medical Devices WG to focus on the subsystem and modules that make up the footprint for safety.
Shefali Sharma has started working on the project to advance the work Shuah and Milna have shared in their recent blog here.
Mentor: Milan Lakhani, Co-Chair of the Medical Devices Working Group and Systems and Software Engineer at Codethink
In open source, Milan’s contributions to Linux kernel are aimed at achieving ELISA project goals. Other than that, he has previously worked in the Trustable and community – mainly STPA analysis on design and writing requirements and tests and also some patches to help with making a webapp and porting.
There are a lot of aspects and opportunities to really learn through experience and take responsibility to make an impact on a highly approved, tested and growing Closed Loop Open-Source insulin delivery system that is really helping to reduce issues of people with type 1 diabetes. There should also be some variety in the tasks and the approach that the mentee can do. Milan is excited to share his skills and knowledge with STPA (our method of safety analysis for the system), the OpenAPS system and codebase (OpenAPS is the medical device itself) and Linux kernel.
Shuah Khan
Mentor: Shuah Khan, ELISA Project TSC Chair and Linux Foundation Fellow
Shuah is an experienced Linux Kernel developer, maintainer, and contributor. She has extensive experience in open source development, actively working across Linux Kernel sub-systems.
She currently maintains the Kernel Selftest, USB over IP, and cpupower tools. She is an active contributor to the Linux media sub-system.
Shuah has a passion for mentoring and educating the next generation. She loves mentoring and training engineers new to open source and helping them become committers and reviewers.
Shefali Sharma
Mentee: Shefali Sharma
Shefali is a third year Computer Science Engineering student from Meerut Institute of Engineering and Technology, Meerut, India. She likes to explore new technical domains. She is very excited to work on the OpenAPS project as it will give her an opportunity to use her technical skills for the welfare of others and to get involved in the Linux kernel community. Apart from this she is also interested in DevOps and Machine Learning.
Written by Dan Brown, Senior Manager, Content & Social Media, Linux Foundation Training & Certification
It’s that time of year – Linux Foundation Training (LiFT) Scholarships are here! Since 2011, The Linux Foundation has awarded over 1,100 scholarships for millions of dollars in training and certification to deserving individuals around the world who would otherwise be unable to afford it. This is part of our mission to grow the open source community by lowering the barrier to entry and making quality training options accessible to those who want them.
Applications are being accepted through April 30 in 12 different categories:
Open Source Newbies
Teens-in-Training
Women in Open Source
Software Developer Do-Gooder
SysAdmin Super Star
Blockchain Blockbuster
Cloud Captain
Linux Kernel Guru
Networking Notable
Web Development Wiz
Hardware Hero – NEW
Cybersecurity Champion – NEW
Whether you are just starting in your open source career, or you are a veteran developer or sysadmin who is looking to gain new skills, if you feel you can benefit from training and/or certification but cannot afford it, you should apply.
Recipients will receive a Linux Foundation eLearning training course and certification exam. All certification exams, and most training courses, are offered remotely, meaning they can be completed from anywhere.
ELISA Project members will come together for a quarterly Spring Workshop on April 5-7 to learn about the latest developments, working group updates, share best practices and collaborate to drive rapid innovation across the industry. Hosted online, this workshop is free and open to the public. If you haven’t yet checked out the schedule, click here.
As we prepare for the Spring Workshop, we’re taking a look at the most popular sessions from the November event. A full recap by Philipp Ahmann, ELISA Project Ambassador and TSC member, can be found here.
One of the most popular sessions was presented by Rachel Sibley, Senior Principal Software Quality Engineer at Red Hat, titled “Requirements Traceability using Code Coverage.”
In this video, Rachel talks about the existing techniques we use for kernel code coverage and how we would like to apply them towards requirements traceability and verification for Red Hat’s Automotive Initiative. By embedding both code coverage analysis and targeted testing during the verification stage using existing tooling, it will enable us to improve our test coverage starting with requirements. You can watch the full video here:
