Written by George Dunlap, Xen Project Advisory Board Chair
Tailored versions of Xen Hypervisor have been used in mission-critical systems for years, but this was never the case for Xen’s mainline. Starting 2019, a Xen Project Functional Safety Special Interest Group was formed to identify and eliminate obstacles to safety-certify Xen.
Safety certification is one of the essential requirements for software to be used in highly regulated industries. Besides technical and compliance issues (such as ISO 26262 vs IEC 61508) transitioning an existing project to become more easily safety certifiable requires significant changes to development practices within an open source project.
At the upcoming ELISA Workshop on May 18-20, Artem Mygaiev, Director, Technology Solutions, EPAM Systems and Stefano Stabellini, Principal Engineer, Xilinx, will lay out some challenges of making safety certification achievable in open source. The talk, scheduled for May 18 at 7:30 am PDT, will primarily focus on the necessary processes, tooling changes, and community challenges that can prevent progress. Additionally, the talk will offer an in-depth review of how Xen Project is approaching this challenging goal and try to derive lessons for other projects and contributors.
Artem Mygaiev, EPAM Systems
Stefano Stabellini, Xilinx
This talk will provide real-life perspectives from open source community members on achieving safety certification. Audiences will have a clear understanding of what obstacles the group faced and how they are overcoming challenges, as well as how to set realistic expectations when embarking on this task. Add this talk to your schedule here: https://sched.co/j3SO.
The ELISA Project is sponsoring two part-time summer mentorships, which runs from June 1- November. ELISA Project Ambassador Lukas Bulwahn will be mentoring both projects.
Linux Kernel: Checkpatch Documentation
Previous mentees have been evaluating, re-visiting and improving the checkpatch script and its various rules. Towards the end of the mentorship, they have also started to document the rules and their rationales (with references to previous discussions and documentation), but not all rules are fully documented yet. The task in this mentorship is to continue evaluating the rules, identifying the known typical false positive cases, writing the documentation of the rules and explaining the rules’ rationales and known false positives. Apply here: https://mentorship.lfx.linuxfoundation.org/project/a6565ff5-b07c-4c04-98db-3a470917d497
Linux Kernel: Mining for Maintainers
Jonathan Corbet identified in his article MAINTAINERS truth and fiction [https://lwn.net/Articles/842415/] that about 2,800 files in the kernel repository have no dedicated maintainer in the MAINTAINERS file (see https://lwn.net/Articles/842606/ for the full list of files). Jonathan Corbet sets out the call for action: “the vast majority are header files under include/, most of which probably do have maintainers and should be added to the appropriate entries.” The task in this mentorship is to follow this call for action and add the header files under include/ to the appropriate entries. Apply here: https://mentorship.lfx.linuxfoundation.org/project/8f69e012-08d0-4e2b-baa7-9143b5f98823
The deadline to submit applications is Friday, May 14. Submit your application today!
Contributed by Jason R. Smith, Principal Engineer, UL LLC and ELISA Ambassador
In my nearly 16 years as a certification engineer focusing on safety-related software and functional safety, on many occasions I have found myself working with a client with safety-related software who is not only going through the certification process for the first time, but is also incorporating third-party software such as Linux into their application. Even before I have to answer questions like “How long will this take?”, I’ll often have to answer an even more fundamental question: “Is it even possible to certify this application?”
Jason Smith, Certification engineer, expressing doubt
My typical response usually starts with the dubious phrase, “Yes, but…”
After first explaining that functional safety standards require the software to be developed in accordance with a software development life cycle like the V-Model, my attention then focuses on the third-party software: it wasn’t developed by the client, it wasn’t tested by the client, it doesn’t have its own certification, and the client doesn’t know much about its inner workings. It is what some of us certification engineers call SOUP, i.e. Software of Unknown Provenance.
Soup
Also SOUP
So, what is required of SOUP? Much of it depends on the application. Standards intended to be applied to high complexity systems such as IEC 61508 and ISO 26262 require either proof of certification or submittal of evidence that more or less demonstrates an equivalent level of confidence as certification. However, some standards used in the appliance or medical sectors such as UL 1998 or IEC 62304, generally systems of lower complexity, allow a different approach that effectively treats SOUP “as is”.
The SOUP Approach
The SOUP approach employed in standards such as UL 1998 or IEC 62304 focuses on a few topics:
Information about the SOUP such as a detailed description of its purpose, its function, its available interfaces, and its version are available and understood by the client;
The client has conducted a fault analysis that treats the SOUP as a component of the system, has analyzed how failures of the SOUP could impact the safety of the system, and has measures in place to address those failures;
The client has sought out information pertaining to any known issues or bugs related to the SOUP, has analyzed that information, and has shown that those known issues or bugs do not impact the safety of the system; and
The client has conducted and can show evidence of appropriate verification and testing activities, proving that the SOUP and any measures that have been implemented to address failures of the SOUP work correctly in the context of the application.
The ELISA project is currently working on a white paper that explains this approach in further detail and what resources are available to further facilitate this approach for applications that employ LINUX. If you are interested in reading more or contributing to this white paper, it is located on GitHub here.
The following is a brief report of my project carried out as a part of the ELISA/LFX mentorship program.
The primary goal of the mentorship is to extend the Codechecker report converter to support a variety of tools such as:
Coccinelle
Smatch
Sphinx
Kernel-Doc
Sparse
Motivation
Many developers contribute to the Linux kernel. And these kernel developers are not exempted from typical programming errors in their patches, such as null pointer referencing, array buffer overflow. Thus, the kernel community has developed some code analyzers, such as sparse, coccinelle and smatch, for reporting such potential error patterns.
The tools mentioned above are some of the well-known tools for analyzing the code of Linux kernel. These tools however only print the warnings and errors on the command-line interface.Below is the output of some example:
arch/x86/kernel/signal.c:338:9: warning: incorrect type in argument 1 (different address spaces)arch/x86/kernel/signal.c:338:9: expected void const volatile [noderef] __user *ptrarch/x86/kernel/signal.c:338:9: got unsigned long long [usertype] *arch/x86/kernel/signal.c:338:9: warning: cast removes address space ‘__user’ of expressionarch/x86/kernel/signal.c:338:9: warning: cast removes address space ‘__user’ of expressionarch/x86/kernel/pci-dma.c:20:26: warning: symbol ‘dma_ops’ was not declared. Should it be static?arch/x86/kernel/pci-dma.c:27:5: warning: symbol ‘panic_on_overflow’ was not declared. Should it be static?arch/x86/kernel/pci-dma.c:31:5: warning: symbol ‘iommu_merge’ was not declared. Should it be static?
It is tedious for developers to look up and keep track of all the errors through the text file/terminal. Manually searching the line of the error also becomes a tedious job. Furthermore, sending the findings or comments to another developer is not feasible. Thus, it becomes very difficult for developers to keep track of all the errors. CodeChecker offers a nice and convenient web interface for viewing all the errors and even giving them some tags such as confirmed, false positive, etc.
CodeChecker’s report converter tool provides an interface for viewing all the reports produced by the code-analyzing tools in a nice and simple web interface. It also provides the functionality to comment and mark the bugs as confirmed, false positive, etc. Thus, I wanted to extend the report converter to the tools mentioned above and also implement the functionality for importing and exporting the changes to and from CodeChecker.
Report Converters for Analyzer Tools
My first and primary task was to create report converters for the tools mentioned above. These report converters would parse the output of tools, using regex into the format:
File Path
Line Number
Column
Error Message
Checker Name
Once the report converter parses the output file of the tools it stores them into plist files which can be opened and viewed into the browser. All the plist files are stored in a folder specified by the user while running the report converter.
CodeChecker also provides a feature to store these plist files into the CodeChecker server and then we can perform multiple operations on the reports like marking the status( confirmed, false positive, etc) and adding comments on each bug.
Importer and Exporter for CodeChecker
The CodeChecker command-line interface provides a variety of options for listing, filtering all the runs/results present in the CodeChecker server. I have added two more commands to the CodeChecker CLI for Importing and Exporting the results to and from the CodeChecker Server.
The Export command lets the user export the findings i.e comments and review the status of one or more reports specified by the user. Below is the sample output of the reports:
{ “comments”: { “c54004ae9ecfb34b396b46d9e08c4291”: [ { “id”: 7, “author”: “Anonymous”, “message”: “This is a confirmed Bug here”, “createdAt”: “2020-11-28 00:05:02.034035”, “kind”: 0 }, { “id”: 6, “author”: “Anonymous”, “message”: “I am doubtful about this bug”, “createdAt”: “2020-11-28 00:01:48.190914”, “kind”: 0 } }, “reviewData”: { “00eab39f7bb399d446e0794025ab3958”: { “status”: 1, “comment”: “This is for the exporter function testing”, “author”: “Anonymous”, “date”: “2020-12-20 23:01:02.669476” }, }}
The Importer command is used for importing the comments and review statuses sent by another user into CodeChecker Server. For importing the comments, for each comment, we check if the date, kind, and the message of the existing comment in the server and the incoming report. If any of them is different, we replace the existing comment with the incoming comment. Similarly, for review status, if the date of the review status is different, then we update the review status in the server with the incoming review.
Use Case of Review Exchange
Consider there are two users of your System, John, and Maria. Both of them have the output of Coccinelle on the Linux kernel.
They run the report converter of coccinelle on the output as:
John sends the obtained file to Maria via email or any other communication medium. Maria downloads this file and imports the findings into his CodeChecker server:
CodeChecker cmd import -i coccinelle.json
Now, the reports in Maria’s server will be:
Report 1:Tag: False PositiveComment: This is not an actual bugReport 2:Tag: ConfirmedReport 3:Tag: ConfirmedComments: [I am not sure about the status of this bug, This is a confirmed bug]Report 4:Comment: This error needs to be handled
Pull Requests
Coccinelle Parser: Coccinelle report converter tool for parsing coccinelle output of kernel sources.
Fix CodeChecker’s cmd results: Comments in the cmd results command were not fetched properly and even showed empty strings in some cases. Added a separate comments section to the details of the results command.
Importer & Exporter command: Exporter command for exporting the comments and review statuses for given or all runs into a json file. Importer command for importing the findings sent by another developer in a json file.
Currently, the import and export commands of CodeChecker are limited only to the Command Line Interface. I would like to implement a feature to make these available in the Web Interface as well.
I would like to extend CodeChecker’s report converter tools to provide proper warning classes for all the report converters.
I would like to add support for multiple users within a single instance of CodeChecker coming to different assessments and then moderating or reviewing them in some controlled way.
Acknowledgment
I would like to thank my mentor Lukas Bulwahn for giving me this opportunity and helping me come up with workflows and ideas for fulfilling my goals. My heartfelt gratitude towards the maintainers of CodeChecker especially Márton Csordás for being patient with me during code reviews and providing his valuable feedback.
By Elana Copperman at Mobileye, Philipp Ahmann at ADIT
More than 120 registered participants, half of them first time joiners, can look back to 3 days of the 6th ELISA workshop, again held virtually due to the pandemic. It was filled with various sessions focusing on, of course, Linux and safety, but also on medical and automotive use cases as well as which role testing, tooling and development processes play to achieve the ELISA deliverables. And these ELISA deliverables should lead to making it easier to enable Linux in safety applications. This is the actual mission of the ELISA project.
As this is the 3rd time that ELISA held a virtual workshop you can see the learnings from the past. Virtual get together, multiple hosts and cloud recordings of sessions support the rich experience of the workshop.
The virtual workshop also reduces the hurdle to participate, especially as workshops are open for all and free of charge. This led again to a higher number of average participants per session compared to our previous workshops and confirms the interest in a functional safety product based on LInux.
During the workshop, besides the regular content such as working group updates and goal settings also completely new areas of interest were presented by members and external speakers. Topics included cybersecurity expectations in the automotive world, code coverage of glibc and Intel’s Linux test robot. The impact to the Linux (Kernel) community was addressed by talks about measuring code review in Linux Kernel, statistics on patch integration or the kernel testing reference process.
The Safety architecture and automotive work groups agreed on their communication interface by sharing requirements and concepts on the Linux architecture. This enabled the momentum these two groups needed to make progress on their goals. Finally, collaboration and contributions from all our ELISA members resulted in publishing source code and documentation on ELISA github.
Tuesday Feb 2 2021
The first day began with updates from the ELISA Working Groups. As ELISA continues to chart new territory, the collaboration between WGs is being defined. In addition to the new interfaces described by Philipp Ahmann between the Architecture and Automotive WGs, we are beginning a joint effort between the Architecture and Development Process WG to set up a database of kernel configurations/features amenable for safety analysis.
In addition, two presentations from Intel/Mobileye focused on static analysis for compliance with MISRA and proposed test strategies for safety qualification and FFI evidences.
Both talks were very insightful, with a lot of feedback from the audience and participants turning the end of the day really into a workshop feeling like a workshop. It includes exactly the discussion which is needed when working on a Linux system ready for safety applications.
Looking forward to an even more exciting day tomorrow!
Wednesday Feb 3 2021
Andreas Gasch and Joyabrata Ghosh kick started today’s sessions with a presentation on Cybersecurity Expectations in the Automotive World. It certainly was interesting to see how the Cybersecurity community is coming around full circle to align with the standards, processes, documentation and management much closer to what we work with in the safety community. Perhaps in the future, we will (eventually) see Cybersecurity and Safety join forces for risk management and qualification.
Eli Gurvitz then provided a report from the Code Coverage Metrics for GLibC mentorship project (by Ashutosh Pandey) on code coverage analysis for glbc, generating quite some ripples and interest in joining the “Fun and Happiness” group aka Tool Investigation and Code Improvement Sub-WG for further work in this area. Kudos, Ashutosh, for a great job!
Spirited discussions came up in the session on how to handle documentation in git and github. It could be clearly seen that the interest in a proper version controlled documentation repository is needed, but the standard format, if reST, markdown or LaTeX could not be concluded and reminded on the starting phase of ELISA. With the Automotive and Medical Device WGs starting to add their documents to the ELISA git repository, ELISA is taking a good iterative approach to making their work visible and structured to meet the expectations of safety experts as and the safety standards.
Day #2 was wrapped up with an extended session on defining kernel configurations for safety critical applications. Focused on the “top down” alignment of the CONFIGs and their analysis within the context of Shuah’s work on CWE classifications for safety. Medium term (~6 month) goal to establish a basic set of configurations and document effectively on how integrators can potentially use those configurations in safety analysis of their specific use case. There are a lot of challenges in this area.
Thursday Feb 4 2021
The 3rd workshop day is typically focused on setting goals for the next 3 months. In addition, ELISA members spend time collaborating in long deeper dive Development Process WG’s working sessions. These sessions actually make the turn from the conference character to a workshop character. Also other sessions such as the goal settings for the next quarter leave enough room to have alignment among the active ELISA members including the Technical Steering Committee. These sessions see fewer first time attendees. We encourage new members to attend the goal setting sessions in the hope that they might be able to engage and collaborate with us on achieving our goals for the next quarter. We sincerely hope all the first time attendees will join us in our upcoming 7th workshop in May 2021.
Closing thoughts
Recapping the three days of this workshop, it is nice to see that the ELISA project is making steady progress and providing enough technical content so that the different working groups start to align and work together more effectively. The Development Process WG has become large enough to spin out smaller teams to focus on WG goals. The ELISA workshops are instrumental in collaborating on discussing current work and in collecting feedback to gain valuable insights and generate new ideas.
The ELISA Workshop #6 will be held over 3 days, February 2-4, 2021.
Once again the ELISA technical community will gather virtually to continue advancing on topics and work relevant to functional safety and safe linux applications. The ELISA Workshop series are focused on education and outreach for new community members, exchanges of ideas and feedback from the linux kernel and safety communities, as well as productive collaboration to make tangible progress toward achieving the mission and goals of the ELISA Project.
Registration
Workshop registration is now closed.
All workshop attendees must register in order to receive session joining details.
Please contact workshop@elisa.tech for late registration requests or any other workshop related questions.
Session Schedule(UTC)
Day 1: Tuesday, February 2, 2021
12:00 – 12:30 Welcome and ELISA Strategy (Shuah Khan, Kate Stewart)
The ELISA community will collaborate virtually September 29 to October 1.
Over twenty sessions with topics ranging from Working Group updates to deep dives into linux technologies, follow-ups from Linux Plumber Conference, and extended collaborative working sessions are being planned over the course of the 3 days.
Registration
Registration closed at 5pm EDT, Thursday, September 24th.
The project proposals of three Google Summer of Code students contributing to Linux kernel verification have been accepted. The three students, Isaac Avram (Izzy) , Mark Balantzyan, and Himanshu Jha have proposed the following topics:
Isaac Avram (Izzy) : Extending Coccinelle with Complex Types
Mark Balantzyan: Analysing Race Conditions in the Linux Kernel
Himanshu Jha: Applying Clang Thread Safety Analyser to Linux Kernel
They will be mentored by Julia Lawall, Alexey Khoroshilov and Lukas Bulwahn, respectively. These three Google Summer of Code projects are governed under the umbrella of the Linux Foundation and the projects are contributing to generally relevant activities for the ELISA Project.
More specifically, one of the building blocks to the safety compliance argumentation in the ELISA Project is the research, investigation, experimentation, use and establishment of verification measures and tools in the Linux kernel development. The ELISA Project embraces the activities in these projects and is looking forward to the projects’ results and the inclusion of new members to the verification activities around the Linux kernel development through the Google Summer of Code student program.
