The Linux Foundation Projects
Skip to main content
All Posts By

elisaproject

Coding Guidelines – to Comply or Not Comply – Some Myth Busting (video)

By Blog, Safety-Critical Software Summit

The Linux Foundation hosted the Embedded Open Source Summit (EOSS), a new umbrella event for open source embedded projects and developer communities to come together under one roof for important collaboration and education, in Prague, Czech Republic, on June 27-30. More than 1,300 people registered for the conference – representing 375 organizations across 56 countries around the globe.

EOSS hosted the Safety-Critical Software Summit, which was sponsored by the ELISA Project, that gathered safety experts and open source developers to enable and advance the use of open source in safety-critical applications. As part of the Summit, Nicole Pappler, CTO and Founder of AlektoMetis, and Philipp Ahmann, Technical Business Development Manager at Robert Bosch GmbH and Chair of the ELISA Project TSC, presented a session titled, “Coding Guidelines – to Comply or Not Comply – Some Myth Busting.”

While adhering to certain coding styles is a good practice in software projects, adhering to coding guidelines for safety critical applications is still something rather exotic in open source projects. As open source projects now more and more start to address the needs of functional safety applications, considering coding guidelines preferred by existing functional safety projects seems to become necessary. The most used rules for coding guidelines in the safety critical context are MISRA rules. While applying these can be quite beneficial for most applications, there is a significant number of exceptions where blindly following these rules causes more problems than it solves.

In this video, Nicole and Philipp discuss the most common coding guidelines, best practices and arguments when following the MISRA rules conflicts with the expectations of the project. Acceptance criteria for non-compliance cases along with examples of acceptable deviations will be presented. This is not contra coding guidelines, but illustrates how coding guidelines are beneficial for a project, what to consider when designing a project’s coding guidelines and how the lessons learned by the application of MISRA rule sets can be applied to languages that are not (yet?) covered by widely accepted rule sets.

Click here for the presentation slides. Click here to view the other videos from the Safety-Critical Software Summit.

For more ELISA Project updates, subscribe to @ProjectElisa or our LinkedIn page or our Youtube Channel.

ELISA Workshop – A Summary of Berlin

By Blog, Workshop

Written by Philipp Ahmann, Chair of the ELISA Project TSC and Technical Business Development Manager at Robert Bosch GmbH 

In June, the ELISA Project’s core contributors and affiliates came together for three days at the Bosch IoT Campus in Berlin, Germany. We discussed recent achievements, project branding and perception, upcoming goals and next steps.

From left to right: (MBition), Gabriele Paoloni (Red Hat), (Red Hat), Olivier Charrier (Windriver), Dongni Fan (MBition), Leonard Moritz Hübner (NXP), Alex Fomichev (MBition), Christof Petig (Aptiv), Philipp Ahmann (Bosch), Kai Hudalla (Bosch Digital), Johannes Kristan (Bosch Digital), Christopher Temple (ARM), Kate Stewart (Linux Foundation) & Sven Erik Jeroschewski (Bosch Digital)

Quick recap on the three days

The workshop kicked off with a discussion about ELISA’s big picture document. The document serves as an entry point for new contributors to find their path through the ELISA deliverables and approach. It will be a living document which gets updated and enhanced when major achievements are reached. It is structured into 3 major parts and complements the project charter and mission.

  • The project objective
  • The ELISA approach (ongoing work to meet the project objective)
  • Using and putting ELISA results into practice

The second session focused on the creation of a pragmatic guide to best practices for open source contributors to facilitate safety analysis in the future. In this session, Kate Stewart, Vice President of Dependable Embedded Systems and and ELISA Ambassador, shared an overview of existing tools which help to make the kernel development work more discoverable, creating certain traceability, and to make analysis “more provable.” The session addressed a few next steps which the project has to look into: 

  • Capturing current Kernel requirements
  • Using Linux features
  • Testing Frameworks

Some parts of the topics were directly addressed as part of the second day agenda. In the first session, the safety analysis approach uses a combination of risk analysis, fault injection, and a high degree of automation. Part of it is also the System Theoretic Process Analysis (STPA). This was already successfully applied within Codethink and taken forward within the Open Source Engineering Process (OSEP) Working Group. The motivation to go in this direction was also made visible and which initial work has been started.

In the following session certain limits of a traditional STPA when applied to the Linux Kernel were pointed out by Red Hat. Additional tool support may be needed which was one reason to create the ks-nav tool. The objective of this tool is to analyze the Linux kernel for safety by presenting diagrams of call trees. In this way an understanding of the interactions and dependencies among different parts of the kernel can be gained for safety analysis. To speed up the development and make the tool more visible, the ks-nav tool resides now in an own repository within the ELISA github organization.

After that, the workshop participants had a longer discussion, whether manpage derived requirements and manpage driven testing can improve the argument towards usage of Linux in safety-critical applications. 

  • It describes a large part of the software components of Linux usage in products
  • It is the established format to describe and learn the software functionality provided by Linux.
  • It is used by a large audience.

The workshop participants agreed that there is still a lot of work to map the current kernel implementation to the existing manpages and to close the gaps between both. This will be a great contribution to the whole kernel community. Overall the ELISA project plans to take major actions in the field of Kernel documentation improvements.

In the afternoon session “targets for upstreaming to Linux kernel for the remainder of the year” the topic of upstreaming documentation within the user and admin guide of the Kernel was put into practice. The current activities of the Linux Features for Safety Critical Systems (LFSCS) WG were presented to the workshop participants. Shuah Khan (Linux Foundation Fellow) together with Elana Copperman (Mobileye, LFSCS WG lead) illustrated the different configuration parameters of the PREEMPT_RT patches which are now almost completely upstreamed. However, it turned out that the documentation of the parameter and configuration towards desired usage have large room for improvements. As many safety-critical products rely on certain real time capabilities, ELISA judges this topic as high priority and very important.

RT Linux in Safety Critical Systems: the potential and the challenges – Elana Copperman & Shuah Khan

The 3rd day concentrated heavily on internal ELISA activities, project health and growth. There was a session revisiting the project messaging along with a session about review of change management workflow, and a proposed approach document to go to the working groups/TSC for approval. In another session the participants brainstormed ideas for community growth and engagement, adjacent community outreach and mutual alignment.

Although the sessions focused on internal work, especially the contributions by affiliated workshop participants representing e.g. Eclipse Software-Defined-Vehicle, ETAS, MBition and NXP added new perspectives, led to good takeaways and made the workshop a success.

Major Workshop Takeaways

During the various sessions and at the end of each day takeaways from the participants were collected and discussed. An extract of major takeaways are listed below:

  • Rework and structure Kernel documentation is an important element of ELISA
    • Strong risk of diverging, in case you write documentation by another person than the maintainer of the code.
  • Start identifying critical subsystems of the Linux kernel to enhance user documentation similar to “workload” and “realtime” documentation.
  • Identification of the “core” part of the kernel that is present in all set of config images
    • Looking at user APIs for the “core” parts, may be a useful focus for doing detailed analysis that others can use, and build from
    • Any analysis has to be tagged to specific release, as changes are happening through time.
    • Getting the API and subsystem analysis of key pieces upstream, combined with recommendations on testing to demonstrate the user space APIs are consistent. (Maintainer need to agree)
  • ELISA is not providing a safe Linux, but there are interesting tools supporting Safety with using Linux
  • If you push a patch to the Linux kernel you have to follow rules (e.g. checkpatch). Maybe there can be kernel tools to improve the safety part of Linux, e.g. that the proposed change/config is in line with the safety guidelines
  • The kernel alone does not make the operating system, you need other components to create a particular system.
  • Open Sourcing the Red Hat requirement tool would be a great benefit for the wider open source safety community
    • Use the requirements tool to export SPDX safety linkage SBOMs for the Linux Kernel
  • Reach out to Eclipse SDV and AGL with SOAFEE to talk about an example system as part of Systems WG
    • SDPX and System SBOM may be of interest for Eclipse Foundation (SDV)
  • OEM may be a must have to work on a real use case in certain domains (especially automotive).
  • The puzzle pieces on the table may not yet be complete and people may use puzzle pieces differently
    • Workshops are a good place to learn how the different pieces fit together, SBOM, OSEP, ARCH…

Getting involved

The ELISA Project is open to anyone to participate. While membership is not required for participation, we always love to welcome additional  members to join us in the mission of  enabling Linux in safety applications and to collaborate with other members who are committed to this effort.

If you are interested to learn more about ELISA or want to participate in one of the working groups or recently started activities, just send  an email to the technical forum mailing list. Or you can get advice on where to contribute best by joining the Technical Steering Committee (TSC) meeting which is held every other Wednesday at 13:00 UTC.

Last but not least the next in person workshop is only a few months away. ELISA members currently plan to meet again most likely in Munich, Germany, October 16- 18. Please join the mailing list and/or subscribe to @ProjectElisa or our LinkedIn page or our Youtube Channel to learn more about the next workshop.

RAFIA – A Roadmap for Certifying Open Source for Use in Safety-Relevant Systems (video)

By Blog, Safety-Critical Software Summit

The Linux Foundation hosted the Embedded Open Source Summit (EOSS), a new umbrella event for open source embedded projects and developer communities to come together under one roof for important collaboration and education, in Prague, Czech Republic, on June 27-30. More than 1,300 people registered for the conference – representing 375 organizations across 56 countries around the globe.

EOSS hosted the Safety-Critical Software Summit, which was sponsored by the ELISA Project, that gathered safety experts and open source developers to enable and advance the use of open source in safety-critical applications. As part of the Summit, Paul Sherwood, Chairman of Codethink, presented a session titled, “RAFIA – A Roadmap for Certifying Open Source for Use in Safety-Relevant Systems.”

 

Many organizations would like to deploy open source software in safety-relevant systems, but face extreme challenges in demonstrating that the results would be safe and compliant with relevant standards such as ISO 61508 and ISO 26262.

In this video, Paul explains RAFIA (Risk Analysis, Automated Testing, Fault Injection, Mitigation and Compliance), a methodology devised by Codethink and shared in public via the ELISA Project, which helps us to establish confidence in the use of open source software to support specific safety goals and demonstrate compliance with applicable standards. The component steps of RAFIA will be covered in detail with examples, as well as lessons learned by Codethink in developing and applying the process for an embedded Linux-based operating system supporting a safety-relevant in-vehicle workload.

Click here for the presentation slides. Click here to view the other videos from the Safety-Critical Software Summit.

For more ELISA Project updates, subscribe to @ProjectElisa or our LinkedIn page or our Youtube Channel.

 

Real Time Linux in Safety-Critical Systems: The Potential and the Challenges (video)

By Blog, Safety-Critical Software Summit

The Linux Foundation hosted the Embedded Open Source Summit (EOSS), a new umbrella event for open source embedded projects and developer communities to come together under one roof for important collaboration and education, in Prague, Czech Republic, on June 27-30. More than 1,300 people registered for the conference – representing 375 organizations across 56 countries around the globe. 

Shuah Khan

EOSS hosted the Safety-Critical Software Summit, which was sponsored by the ELISA Project, that gathered safety experts and open source developers to enable and advance the use of open source in safety-critical applications. As part of the Summit, Elana Copperman, ELISA Ambassador, Linux Features for Safety-Critical Systems WG Chair and Systems Safety Architect at Mobileye, and Shuah Khan, ELISA Ambassador, member of the ELISA TSC and Linux Fellow at The Linux Foundation, gave a presentation titled, “RTL in Safety-Critical Systems: The Potential and the Challenges.

The Real Time Linux (RTL) collaborative project was established to help coordinate the efforts around mainlining Preempt RT and ensuring that the maintainers have the ability to continue development work, long-term support and future research of RT. The RTL project has been active in adding Preempt RT features in to the mainline kernel. It is time for a closer look on how these features can be used in Safety-Critical Systems.

In this video, we provide a brief overview of Real Time Linux and potential usage in Safety-Critical systems. In addition, we discuss how these features may be relevant to support system safety. We go over the following areas that are most relevant:

1. Tools for analysis of system workload resource usage and performance impact.

2. Kernel configs, guidelines on usage.

3. Relevant system parameters, generic and architecture specific.

4. Test frameworks and how they may be used to investigate and demonstrate safety features.

The PPT presentation can be found here or watch the video below.

https://youtu.be/ShcEarZTcRY?si=5CYTxWOkiOvoHzQ1

 

Click here for the presentation slides. Click here to view the other videos from the Safety-Critical Software Summit.

For more ELISA Project updates, subscribe to @ProjectElisa or our LinkedIn page or our Youtube Channel.

Safety Certifying an Open Source Project: The Example of Xen

By Blog, Safety-Critical Software Summit

The Linux Foundation hosted the Embedded Open Source Summit (EOSS), a new umbrella event for open source embedded projects and developer communities to come together under one roof for important collaboration and education, in Prague, Czech Republic, on June 27-30. More than 1,300 people registered for the conference – representing 375 organizations across 56 countries around the globe. 

EOSS hosted the Safety-Critical Software Summit, which was sponsored by the ELISA Project, that gathered safety experts and open source developers to enable and advance the use of open source in safety-critical applications. As part of the Summit, Stefano Stabellini, Fellow at AMD, and Bertrand Marquis, Principal Software Engineer at ARM, gave a presentation titled,Safety Certifying an Open Source Project: The Example of Xen.

Safety is important to software everywhere human lives are at risk. In these environments, safety standards must be followed to minimize the risk to humans and to follow regulations. Safety standards such as ISO 26262 come with a series of requirements and processes that sometimes clash with well-established Open Source software development practices. How do we reconcile safety certifications and Open Source?

This presentation will provide some insights to answer that question, using the Xen hypervisor as an example. Xen has a micro-kernel design and provides a virtualization solution for embedded and automotive while having a code base small enough to make certifications possible. This presentation will go through the changes to upstream processes that the Xen community adopted during the last 12 months to align community activities with safety-certification requirements. It will discuss any additional changes planned for the near future. The talk will also cover the latest updates from the Xen FuSa working group on MISRA C, traceability, testing, etc. Watch the video below:

Click here for the presentation slides. Click here to view the other videos from the Safety-Critical Software Summit.

For more ELISA Project updates, subscribe to @ProjectElisa or our LinkedIn page or our Youtube Channel.

Linux in Aerospace: Objections and Paths Forward

By Blog, Safety-Critical Software Summit, Working Group

The Linux Foundation hosted the Embedded Open Source Summit, a new umbrella event for open source embedded projects and developer communities to come together under one roof for important collaboration and education, in Prague, Czech Republic, on June 27-30. More than 1,300 people registered for the conference – representing 375 organizations across 56 countries around the globe. 

The event hosted the Safety-Critical Software Summit, which was sponsored by the ELISA Project, that gathered safety experts and open source developers to enable and advance the use of open source in safety-critical applications. As part of the Summit, Peter Brink, Functional Safety Engineering Leader at Underwriter Laboratories (UL) and Steven H. VanderLeest, Chief Technologist for Boeing Linux at Boeing, gave a presentation titled, “Debating Linux in Aerospace: Objections and Paths Forward.”

Traditionally, safety-critical flight software used in aerospace is closed, proprietary code from a handful of commercial vendors. Although open-source software could provide several benefits, there are significant hurdles that prevent widespread adoption. First, we list some of the potential benefits of open source for safety-critical aerospace applications. Second, we present an overview of the key concepts and standards for flight software. Third, we identify the objections and concerns for using Linux as the avionics real-time operating system, which is software that generally needs the highest levels of assurance. For each objection, we suggest a possible path forward to address the concern.

Click here for the presentation slides. Click here to view the other videos from the Safety-Critical Software Summit.

Learn more about linux and aerospace by joining the ELISA Aerospace Working Group. For all upcoming ELISA Working Group meetings and public seminars, please go to https://lists.elisa.tech/calendar.

For more ELISA Project updates, subscribe to @ProjectElisa or our LinkedIn page or our Youtube Channel.

A Development Environment for DO-178C Level D Certified Linux

By Blog, Seminar Series

The ELISA Project Seminar Series focuses on hot topics related to ELISA and its mission. Presenters are members, contributors and thought leaders from the ELISA Project and surrounding communities. To view past presentations, click here.

On July 18,  Chuck Wolber, Software Engineer at The Boeing Company presented a seminar titled, “A Development Environment for DO-178C Level D Certified Linux.”

This video features the use of Yocto/OpenEmbedded as a tool for managing a distributed development environment, automated build and test, and ultimately delivering a DO-178C level D certified Linux platform into revenue service. It also touches on generalized aspects of traceability, team dynamics, “day one developer,” and extensibility. Watch the video:

Learn more about linux and aerospace by joining the ELISA Aerospace Working Group.

For all upcoming ELISA Working Group meetings and public seminars, please go to https://lists.elisa.tech/calendar.

Diving into the Kernel: Introducing ks-nav Tool Set 

By Blog, Working Group, Workshop

Written by Red Hat’s Gabriele Paoloni, Alessandro Carminati & Maurizio Papini

One of the main challenges in using the Linux Kernel for safety-critical systems is conducting safety analyses in the absence of architectural documentation. As outlined in this article, within the ELISA (Enabling Linux in Safety Applications) Project, we are adopting the STPA approach at the system level. Accordingly, the Safety Architecture Working Group has been actively working on implementing and expanding this approach within the Kernel.

To conduct an STPA-inspired analysis, it is necessary to define “controller” entities, along with their corresponding control actions and feedback mechanisms. The Linux Kernel has already been divided into entities, which are maintained by different individuals based on the MAINTAINERS file.

Therefore, the Safety Architecture Working Group has made the decision to experiment with STPA analysis within the Kernel by treating the various subsystems or drivers (as defined in the MAINTAINERS file) as individual controllers. Within this context, the challenge has been to identify the control actions and feedback mechanisms between the drivers and subsystems.

The ks-nav tool set, comprising two complementary tools, is specifically designed to support the identification of such control actions. 

To facilitate this, ks-nav offers subsystem call trees, which visually represent the interactions and dependencies among subsystems, starting from a given symbol. This feature allows users to identify potential interfaces between subsystems or drivers that support relevant control actions within the specific context of the symbol under analysis.

Another key  feature of ks-nav is the identification of function call trees, which list functions potentially encountered starting from a given one . Such a feature could be useful to understand the subsystem or driver behavior following the invocation of a given function. 

In summary, within the context of a specific symbol, ks-nav is capable of initially highlighting potential candidates for control actions between subsystems and drivers. Additionally, it allows users to “zoom in” on each subsystem as necessary to support expert judgment in semantically specifying the control actions.

To accommodate diverse analysis needs, the tool set supports multiple output formats, including dot, raster images (PNG or JPG), and vector images (SVG), facilitating effective visualization.

Flexibility is emphasized with compatibility across different database management systems (DBMS) like PostgreSQL, MySQL, MariaDB, or SQLite. This enables seamless integration with users’ preferred DBMS or existing infrastructure.

Moreover, ks-nav is able to identify indirect calls, including the x86 retpoline technique, within the kernel code, and deals with compiler code optimization.

By offering function call trees, subsystem call trees, versatile output formats, DBMS compatibility, and indirect call detection, the ks-nav tool set provides a comprehensive and efficient solution for ELISA activities in Linux kernel analysis. It provides users with the necessary tools to explore the kernel’s structure, and make informed decisions.

This initial commit of the ks-nav tool set also ensures fair test coverage, guaranteeing reliability and effectiveness in supporting ELISA activities. It marks a milestone, demonstrating the team’s commitment to continuous improvement and future advancements to refine the tool set and meet evolving needs in ELISA activities conducted by the working group.

All are welcome to try out the tools, send pull requests for improvements and bug fixes on the ELISA GitHub here.

There will also be a dedicated session on how to apply this tool at the upcoming ELISA Berlin Workshop June 20-22. Learn more about the Workshop or register for it here.

ELISA Safety Analysis Approach

By Ambassadors, Blog, Working Group

Written By: Paul Albertella, ELISA Project TSC member, Chair for Open Source Engineering Process Working Group and Consultant at Codethink

The ELISA Open Source Engineering Process (OSEP) Working Group plans to document and apply a safety analysis process based on STPA that is suitable for use with Linux and other open source software.

Our objective is to specify a system context and an example set of safety goals for a safety-related system involving the Linux kernel, in order to enable the safety analysis and specification of a set of safety responsibilities that we may assign to the kernel in that context (and possibly other contexts), at a useful level of detail.

  • By system context, we mean either a concrete system design, or an abstraction representing a class of system designs.
  • By safety goals, we mean a set of system-level criteria that must be satisfied in order to avoid specific negative outcomes or consequences.
  • By safety responsibilities, we mean the behaviour or properties that are required to avoid violating the safety goals for the given system context. This may involve cooperation with other safety mechanisms, which are required to operate when it is not possible to avoid violating a high level safety goal.

In ISO 26262 terminology, this is equivalent to defining the assumptions of use (AoU) for Linux (or any FOSS component, or integration of components) as a Safety Element out of Context (SEooC).

You can find more information about this approach and its intended role as part of a wider engineering process in the Refining the RAFIA approach talk from last year’s ELISA Spring Workshop.

Our purpose with undertaking this kind of analysis in ELISA is to describe and provide examples of a method for identifying and documenting the risks associated with using Linux in a given context, and to examine how its existing features may be used to help to identify, control and/or mitigate these risks. The results of this analysis may then be used to derive the safety requirements that should apply for a system using Linux in such a context

For example, certain Linux configurations may help to address some of the risks that we identify, while others – including default kernel configurations – may introduce additional and possibly unacceptable risks (e.g performance optimisations that may have unintended or unpredictable consequences). Some mitigations for these risks may be outside the scope of Linux, involving AoUs on applications using Linux or on other components integrated with Linux as part of an operating system.

The outputs of this process are expected to be:

  • A description of the system context and safety goals that we have assumed for the purpose of the analysis
  • A specification of the risks that are considered in the analysis, and any exclusions from scope
  • Safety responsibilities for both Linux and the other system components identified in the system context, and specific requirements relating to these
  • Specific scenarios that might lead to these safety requirements being violated, which can be used to derive test cases and fault injections, and to identify where additional mitigations, safety mechanisms and/or requirements are needed to deal with these scenarios
  • Documents capturing the results of Linux feature analysis (or analysis of other interacting components) that was undertaken as part of the investigation

The OSEP group plans to start by applying this approach to the Automotive working group’s Telltale use case, documenting and refining the process as well as recording the results of the analysis.

We have invited other ELISA working groups and contributors to consider the following:

  • What other system contexts and safety goals should we consider for analysis?
  • What specific Linux features or properties should we focus on in our analyses?
  • How might ELISA working groups collaborate in applying, refining and documenting this process and its results?
  • Are there any external communities with which we might collaborate?

We would also welcome input from other open source contributors and communities interested in functional safety.

If you would like to get involved or learn more about this approach, please join the OSEP mailing list, where you can also find details of our meetings and how to participate.

What happens when OpenAPS commands run on Linux?

By Blog, Technical Update, Working Group

Written by Shuah Khan, Linux Fellow at the Linux Foundation and member of the ELISA Project TSC

Key Points

  • Understanding system resources necessary to build and run a workload is important.
  • Linux tracing and strace can be used to discover the system resources in use by a workload.
  • Tracing OpenAPS commands with strace generated detailed view of system activity during the common run and helped with generating flowcharts for normal and error paths when commands detected device busy conditions.
  • Once we discover and understand the workload needs, we can focus on them to avoid regressions and evaluate safety.

OpenAPS is an open source Artificial Pancreas System designed to automatically adjust an insulin pump’s insulin delivery to keep Blood Glucose in a safe range at all times. It is an open and transparent effort to make safe and effective basic Automatic Pancreas System technology widely available to anyone with compatible medical devices who is willing to build their own system.

Broadly speaking, the OpenAPS system can be thought of performing 3 main functions. Monitoring the environment and operational status of devices with as much data relevant to therapy as possible collected, predicting what should happen to glucose levels next, and enacting changes through issuing commands, emails and even phone calls.

The ELISA Medical Devices Working Group has set out to discover the Linux kernel subsystems used by OpenAPS. Understanding the kernel footprint necessary to run a workload helps us focus on the  subsystem and modules that make up the footprint for safety. We set out to answer the following questions:

  • What happens when an OpenAPS workload runs on Linux and discovers the subsystems and modules that are in active use when OpenAPS is running?
  • What are the interactions between OpenAPS and the kernel when a user checks how much insulin is left in the insulin pump?

Fine grained view of system activity

As mentioned earlier, the approach we gathered higher level of information about the OpenAPS usage. This higher information doesn’t tell us the system usage by individual OpenAPS commands. For example, watching the brain activity during the entire dinner vs. isolating the activity as we take the first bite of a delicious dessert and enjoy it.

Following up on our previous work, we gathered the fine grained information about individual OpenAPS commands and important use-cases. We used the strace command to trace the OpenAPS commands based on the process we identified to trace a generic workload which is now in the Linux kernel user’s and administrator’s guide. We traced several OpenAPS commands on an OpenAPS instance running on RasPi managing a Medtronic Insulin Pump. “mdt” in the following text refers to the command provided by https://github.com/ecc1/medtronic:

  • Get Insulin Pump model (mdt model)
  • Get Insulin Pump time (mdt clock)
  • Get Insulin Pump battery (mdt battery)
  • Get Insulin Pump basal rate schedule (mdt basal)
  • Suspend Insulin Pump (mdt suspend)
  • Resume Insulin Pump (mdt resume)
  • Get the remaining insulin in the Insulin Pump reservoir (mdt reservoir)
  • Get Insulin Pump temporary basal rate (mdt tempbasal)
  • Get Insulin Pump history (pumphistory)
    • Reference: https://github.com/ecc1/medtronic/tree/main/cmd/pumphistory
  • Send button press to the Insulin Pump  (mdt button)
  • Command the Insulin Pump to deliver a given amount of insulin as bolus  (mdt bolus)
    • Reference: https://github.com/ecc1/medtronic/blob/main/button.go
  • Set Insulin Pump temporary basal rate (mdt settempbasal)
    • Reference: Advanced OpenAPS features: https://openaps.readthedocs.io/en/latest/docs/Customize-Iterate/oref1.html

We ran these commands in normal mode and under strace to get summary (strace -c ) and complete trace (strace) information. The following shows a few selected commands, their output, trace information, and our analysis of the trace.

Running the command to get  the remaining insulin in the reservoir

mdt reservoir &> reservoir.out      # Get the remaining insulin in the reservoir 

strace -c mdt reservoir &> reservoir.summary  # trace summary

strace mdt reservoir &> reservoir.full  # full trace

Output from the get the remaining insulin in the reservoir command

Run the get the reservoir status command

connected to CC111x radio on /dev/spidev0.0

setting frequency to 916.600

waking pump

model 722 pump

108.5

disconnecting CC111x radio on /dev/spidev0.0

Process startup (process mgmt)

Open files (fs -> driver sysfs)

Open pump device “/dev/spidev0.0”

Open “/sys/class/gpio/gpio4/active_low”

Open “/sys/class/gpio/gpio4/direction”

Open “/sys/class/gpio/gpio4/value”

Open “/sys/class/gpio/gpio4/value”

Send SPI_IOC_MESSAGE(s) to talk to the pump (driver ioctl)

Close files (fs -> driver sysfs)

Exit program  (process mgmt)

This output shows that the Pump wakeup path was invoked.

mdt reservoir command flowchart

Complete strace -c output

2022/10/12 11:50:59 connected to CC111x radio on /dev/spidev0.0

2022/10/12 11:50:59 setting frequency to 916.600

2022/10/12 11:51:00 model 722 pump

108.5

2022/10/12 11:51:00 disconnecting CC111x radio on /dev/spidev0.0

% time     seconds  usecs/call     calls    errors syscall

—— ———– ———– ——— ——— —————-

 38.55    0.012979         270        48           ioctl

 31.50    0.010605          87       121         1 futex

  9.90    0.003333          27       120           rt_sigaction

  3.86    0.001300          54        24           clock_gettime

  3.04    0.001022         170         6           write

  1.95    0.000658         658         1           readlinkat

  1.73    0.000584          26        22           mmap2

  1.66    0.000558         111         5           read

  1.63    0.000548          49        11           openat

  1.61    0.000543         181         3           clone

  1.55    0.000521          47        11           fcntl

  1.08    0.000362          40         9           rt_sigprocmask

  0.77    0.000259          25        10           close

  0.72    0.000243          24        10           mprotect

  0.26    0.000088          22         4           brk

  0.14    0.000046          23         2           sigaltstack

  0.06    0.000020          20         1           gettid

  0.00    0.000000           0         1           execve

  0.00    0.000000           0         1           getpid

  0.00    0.000000           0         1           access

  0.00    0.000000           0         1           readlink

  0.00    0.000000           0         2           munmap

  0.00    0.000000           0         1           uname

  0.00    0.000000           0         1           flock

  0.00    0.000000           0         1           ugetrlimit

  0.00    0.000000           0         5           fstat64

  0.00    0.000000           0         1           sched_getaffinity

  0.00    0.000000           0         8           epoll_ctl

  0.00    0.000000           0         1           set_tid_address

  0.00    0.000000           0         2           fstatat64

  0.00    0.000000           0         1           set_robust_list

  0.00    0.000000           0         1           epoll_create1

  0.00    0.000000           0         1           set_tls

—— ———– ———– ——— ——— —————-

100.00    0.033669                   437         1 total

Get temp basal rate

mdt tempbasal &> tempbasal.out # Get temp basal rate

strace -c mdt tempbasal &> tempbasal.summary # trace summary

strace mdt tempbasal &> tempbasal.full # full trace

Output from the get temp basal rate command

Run get basal temperature command

connected to CC111x radio on /dev/spidev0.0

setting frequency to 916.600

model 722 pump

“duration”: 28,

  “temp”: “absolute”,

  “rate”: 0.9

}

disconnecting CC111x radio on /dev/spidev0.0

Process startup (process mgmt)

Open files (fs -> driver sysfs)

Open pump device “/dev/spidev0.0”

Open “/sys/class/gpio/gpio4/active_low”

Open “/sys/class/gpio/gpio4/direction”

Open “/sys/class/gpio/gpio4/value”

Open “/sys/class/gpio/gpio4/value”

Sends SPI_IOC_MESSAGE(s) to talk to the pump (driver ioctl SPI_IOC_MESSAGE)

Close files (fs -> driver sysfs)

Exit program  (process mgmt)

mdt tempbasal command flowchart

Complete strace -c output

2022/10/12 11:51:06 connected to CC111x radio on /dev/spidev0.0

2022/10/12 11:51:06 setting frequency to 916.600

2022/10/12 11:51:06 model 722 pump

{

  “duration”: 28,

  “temp”: “absolute”,

  “rate”: 0.9

}

2022/10/12 11:51:06 disconnecting CC111x radio on /dev/spidev0.0

% time     seconds  usecs/call     calls    errors syscall

—— ———– ———– ——— ——— —————-

 44.07    0.015195         316        48           ioctl

 31.19    0.010755          93       115         1 futex

  8.35    0.002879          23       120           rt_sigaction

  2.43    0.000837          34        24           clock_gettime

  2.10    0.000723          65        11           openat

  2.10    0.000723         723         1           readlinkat

  1.90    0.000655         109         6           write

  1.46    0.000503         167         3           clone

  1.05    0.000362          72         5           read

  0.95    0.000326          32        10           close

  0.92    0.000317          28        11           fcntl

  0.80    0.000277          34         8           epoll_ctl

  0.67    0.000232          10        22           mmap2

  0.62    0.000215          23         9           rt_sigprocmask

  0.52    0.000181          18        10           mprotect

  0.41    0.000140          70         2           fstatat64

  0.27    0.000094          23         4           brk

  0.11    0.000039          39         1           flock

  0.08    0.000029          29         1           epoll_create1

  0.00    0.000000           0         1           execve

  0.00    0.000000           0         1           getpid

  0.00    0.000000           0         1           access

  0.00    0.000000           0         1           readlink

  0.00    0.000000           0         2           munmap

  0.00    0.000000           0         1           uname

  0.00    0.000000           0         2           sigaltstack

  0.00    0.000000           0         1           ugetrlimit

  0.00    0.000000           0         5           fstat64

  0.00    0.000000           0         1           gettid

  0.00    0.000000           0         1           sched_getaffinity

  0.00    0.000000           0         1           set_tid_address

  0.00    0.000000           0         1           set_robust_list

  0.00    0.000000           0         1           set_tls

—— ———– ———– ——— ——— —————-

100.00    0.034482                   431         1 total

Get insulin pump history

pumphistory -n 1 &> pumphistory.out # Get insulin pump history

strace -c pumphistory -n 1 &> pumphistory.summary # trace summary

strace pumphistory -n 1 &> pumphistory.full # full trace

Output from the get pumphistory command

Run pumphistory command

retrieving pump history since 2022-10-12 10:54:03

cannot connect to CC111x radio on /dev/spidev0.0

null

/dev/spidev0.0: device is in use

Process startup (process mgmt)

Open files (fs -> driver sysfs)

Open pump device “/dev/spidev0.0”

Open “/sys/class/gpio/gpio4/active_low”

Open “/sys/class/gpio/gpio4/direction”

Close files (fs -> driver sysfs)

Exit program  (process mgmt)

This output shows that the pump busy path is invoked

Pump history command flowchart

Complete strace -c output

2022/10/12 11:54:03 retrieving pump history since 2022-10-12 10:54:03

2022/10/12 11:54:03 cannot connect to CC111x radio on /dev/spidev0.0

null

2022/10/12 11:54:03 /dev/spidev0.0: device is in use

% time     seconds  usecs/call     calls    errors syscall

—— ———– ———– ——— ——— —————-

 40.92    0.003106          25       120           rt_sigaction

 15.91    0.001208         109        11           futex

  8.09    0.000614         204         3           clone

  7.84    0.000595          59        10           mprotect

  7.52    0.000571         190         3           fcntl

  5.23    0.000397          44         9           rt_sigprocmask

  5.20    0.000395          17        22           mmap2

  4.23    0.000321         321         1           readlinkat

  3.77    0.000286          13        21           clock_gettime

  1.29    0.000098          24         4           brk

  0.00    0.000000           0         5           read

  0.00    0.000000           0         4           write

  0.00    0.000000           0         7           close

  0.00    0.000000           0         1           execve

  0.00    0.000000           0         1           getpid

  0.00    0.000000           0         1           access

  0.00    0.000000           0         1           readlink

  0.00    0.000000           0         2           munmap

  0.00    0.000000           0         1           uname

  0.00    0.000000           0         1         1 flock

  0.00    0.000000           0         2           sigaltstack

  0.00    0.000000           0         1           ugetrlimit

  0.00    0.000000           0         5           fstat64

  0.00    0.000000           0         1           gettid

  0.00    0.000000           0         1           sched_getaffinity

  0.00    0.000000           0         1           set_tid_address

  0.00    0.000000           0         7           openat

  0.00    0.000000           0         1           set_robust_list

  0.00    0.000000           0         1           set_tls

—— ———– ———– ——— ——— —————-

100.00    0.007591                   248         1 total

The following diagram shows the run-time context for these commands and their mapping to the Linux subsystems used by them.

System view

The following system view was updated after high level tracing analysis. This system view remains unchanged after the fine grained tracing analysis as expected.

References

https://github.com/ecc1/medtronic/blob/main/button.go

Conclusion

Using strace to trace OpenAPS commands helped us understand the detailed view of system files opened and closed while the commands run. We were able to generate flowcharts for normal and error paths when commands detected device busy conditions. As mentioned earlier, this tracing method gave us insight into the parts of the kernel used by the individual OpenAPS commands.

Credits

The ELISA Medical Working Group would like to sincerely acknowledge Chance Harrison for running the OpenAPS commands and providing information critical for making this effort a successful one.

SPDX-License-Identifier: CC-BY-4.0

This document is released under the Creative Commons Attribution 4.0 International License, available at https://creativecommons.org/licenses/by/4.0/legalcode. Pursuant to Section 5 of the license, please note that the following disclaimers apply (capitalized terms have the meanings set forth in the license). To the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.

To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.

The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.