Skip to main content

What happens when OpenAPS commands run on Linux?

Written by Shuah Khan, Linux Fellow at the Linux Foundation and member of the ELISA Project TSC

Key Points

  • Understanding system resources necessary to build and run a workload is important.
  • Linux tracing and strace can be used to discover the system resources in use by a workload.
  • Tracing OpenAPS commands with strace generated detailed view of system activity during the common run and helped with generating flowcharts for normal and error paths when commands detected device busy conditions.
  • Once we discover and understand the workload needs, we can focus on them to avoid regressions and evaluate safety.

OpenAPS is an open source Artificial Pancreas System designed to automatically adjust an insulin pump’s insulin delivery to keep Blood Glucose in a safe range at all times. It is an open and transparent effort to make safe and effective basic Automatic Pancreas System technology widely available to anyone with compatible medical devices who is willing to build their own system.

Broadly speaking, the OpenAPS system can be thought of performing 3 main functions. Monitoring the environment and operational status of devices with as much data relevant to therapy as possible collected, predicting what should happen to glucose levels next, and enacting changes through issuing commands, emails and even phone calls.

The ELISA Medical Devices Working Group has set out to discover the Linux kernel subsystems used by OpenAPS. Understanding the kernel footprint necessary to run a workload helps us focus on the  subsystem and modules that make up the footprint for safety. We set out to answer the following questions:

  • What happens when an OpenAPS workload runs on Linux and discovers the subsystems and modules that are in active use when OpenAPS is running?
  • What are the interactions between OpenAPS and the kernel when a user checks how much insulin is left in the insulin pump?

Fine grained view of system activity

As mentioned earlier, the approach we gathered higher level of information about the OpenAPS usage. This higher information doesn’t tell us the system usage by individual OpenAPS commands. For example, watching the brain activity during the entire dinner vs. isolating the activity as we take the first bite of a delicious dessert and enjoy it.

Following up on our previous work, we gathered the fine grained information about individual OpenAPS commands and important use-cases. We used the strace command to trace the OpenAPS commands based on the process we identified to trace a generic workload which is now in the Linux kernel user’s and administrator’s guide. We traced several OpenAPS commands on an OpenAPS instance running on RasPi managing a Medtronic Insulin Pump. “mdt” in the following text refers to the command provided by https://github.com/ecc1/medtronic:

  • Get Insulin Pump model (mdt model)
  • Get Insulin Pump time (mdt clock)
  • Get Insulin Pump battery (mdt battery)
  • Get Insulin Pump basal rate schedule (mdt basal)
  • Suspend Insulin Pump (mdt suspend)
  • Resume Insulin Pump (mdt resume)
  • Get the remaining insulin in the Insulin Pump reservoir (mdt reservoir)
  • Get Insulin Pump temporary basal rate (mdt tempbasal)
  • Get Insulin Pump history (pumphistory)
    • Reference: https://github.com/ecc1/medtronic/tree/main/cmd/pumphistory
  • Send button press to the Insulin Pump  (mdt button)
  • Command the Insulin Pump to deliver a given amount of insulin as bolus  (mdt bolus)
    • Reference: https://github.com/ecc1/medtronic/blob/main/button.go
  • Set Insulin Pump temporary basal rate (mdt settempbasal)
    • Reference: Advanced OpenAPS features: https://openaps.readthedocs.io/en/latest/docs/Customize-Iterate/oref1.html

We ran these commands in normal mode and under strace to get summary (strace -c ) and complete trace (strace) information. The following shows a few selected commands, their output, trace information, and our analysis of the trace.

Running the command to get  the remaining insulin in the reservoir

mdt reservoir &> reservoir.out      # Get the remaining insulin in the reservoir 

strace -c mdt reservoir &> reservoir.summary  # trace summary

strace mdt reservoir &> reservoir.full  # full trace

Output from the get the remaining insulin in the reservoir command

Run the get the reservoir status command

connected to CC111x radio on /dev/spidev0.0

setting frequency to 916.600

waking pump

model 722 pump

108.5

disconnecting CC111x radio on /dev/spidev0.0

Process startup (process mgmt)

Open files (fs -> driver sysfs)

Open pump device “/dev/spidev0.0”

Open “/sys/class/gpio/gpio4/active_low”

Open “/sys/class/gpio/gpio4/direction”

Open “/sys/class/gpio/gpio4/value”

Open “/sys/class/gpio/gpio4/value”

Send SPI_IOC_MESSAGE(s) to talk to the pump (driver ioctl)

Close files (fs -> driver sysfs)

Exit program  (process mgmt)

This output shows that the Pump wakeup path was invoked.

mdt reservoir command flowchart

Complete strace -c output

2022/10/12 11:50:59 connected to CC111x radio on /dev/spidev0.0

2022/10/12 11:50:59 setting frequency to 916.600

2022/10/12 11:51:00 model 722 pump

108.5

2022/10/12 11:51:00 disconnecting CC111x radio on /dev/spidev0.0

% time     seconds  usecs/call     calls    errors syscall

—— ———– ———– ——— ——— —————-

 38.55    0.012979         270        48           ioctl

 31.50    0.010605          87       121         1 futex

  9.90    0.003333          27       120           rt_sigaction

  3.86    0.001300          54        24           clock_gettime

  3.04    0.001022         170         6           write

  1.95    0.000658         658         1           readlinkat

  1.73    0.000584          26        22           mmap2

  1.66    0.000558         111         5           read

  1.63    0.000548          49        11           openat

  1.61    0.000543         181         3           clone

  1.55    0.000521          47        11           fcntl

  1.08    0.000362          40         9           rt_sigprocmask

  0.77    0.000259          25        10           close

  0.72    0.000243          24        10           mprotect

  0.26    0.000088          22         4           brk

  0.14    0.000046          23         2           sigaltstack

  0.06    0.000020          20         1           gettid

  0.00    0.000000           0         1           execve

  0.00    0.000000           0         1           getpid

  0.00    0.000000           0         1           access

  0.00    0.000000           0         1           readlink

  0.00    0.000000           0         2           munmap

  0.00    0.000000           0         1           uname

  0.00    0.000000           0         1           flock

  0.00    0.000000           0         1           ugetrlimit

  0.00    0.000000           0         5           fstat64

  0.00    0.000000           0         1           sched_getaffinity

  0.00    0.000000           0         8           epoll_ctl

  0.00    0.000000           0         1           set_tid_address

  0.00    0.000000           0         2           fstatat64

  0.00    0.000000           0         1           set_robust_list

  0.00    0.000000           0         1           epoll_create1

  0.00    0.000000           0         1           set_tls

—— ———– ———– ——— ——— —————-

100.00    0.033669                   437         1 total

Get temp basal rate

mdt tempbasal &> tempbasal.out # Get temp basal rate

strace -c mdt tempbasal &> tempbasal.summary # trace summary

strace mdt tempbasal &> tempbasal.full # full trace

Output from the get temp basal rate command

Run get basal temperature command

connected to CC111x radio on /dev/spidev0.0

setting frequency to 916.600

model 722 pump

“duration”: 28,

  “temp”: “absolute”,

  “rate”: 0.9

}

disconnecting CC111x radio on /dev/spidev0.0

Process startup (process mgmt)

Open files (fs -> driver sysfs)

Open pump device “/dev/spidev0.0”

Open “/sys/class/gpio/gpio4/active_low”

Open “/sys/class/gpio/gpio4/direction”

Open “/sys/class/gpio/gpio4/value”

Open “/sys/class/gpio/gpio4/value”

Sends SPI_IOC_MESSAGE(s) to talk to the pump (driver ioctl SPI_IOC_MESSAGE)

Close files (fs -> driver sysfs)

Exit program  (process mgmt)

mdt tempbasal command flowchart

Complete strace -c output

2022/10/12 11:51:06 connected to CC111x radio on /dev/spidev0.0

2022/10/12 11:51:06 setting frequency to 916.600

2022/10/12 11:51:06 model 722 pump

{

  “duration”: 28,

  “temp”: “absolute”,

  “rate”: 0.9

}

2022/10/12 11:51:06 disconnecting CC111x radio on /dev/spidev0.0

% time     seconds  usecs/call     calls    errors syscall

—— ———– ———– ——— ——— —————-

 44.07    0.015195         316        48           ioctl

 31.19    0.010755          93       115         1 futex

  8.35    0.002879          23       120           rt_sigaction

  2.43    0.000837          34        24           clock_gettime

  2.10    0.000723          65        11           openat

  2.10    0.000723         723         1           readlinkat

  1.90    0.000655         109         6           write

  1.46    0.000503         167         3           clone

  1.05    0.000362          72         5           read

  0.95    0.000326          32        10           close

  0.92    0.000317          28        11           fcntl

  0.80    0.000277          34         8           epoll_ctl

  0.67    0.000232          10        22           mmap2

  0.62    0.000215          23         9           rt_sigprocmask

  0.52    0.000181          18        10           mprotect

  0.41    0.000140          70         2           fstatat64

  0.27    0.000094          23         4           brk

  0.11    0.000039          39         1           flock

  0.08    0.000029          29         1           epoll_create1

  0.00    0.000000           0         1           execve

  0.00    0.000000           0         1           getpid

  0.00    0.000000           0         1           access

  0.00    0.000000           0         1           readlink

  0.00    0.000000           0         2           munmap

  0.00    0.000000           0         1           uname

  0.00    0.000000           0         2           sigaltstack

  0.00    0.000000           0         1           ugetrlimit

  0.00    0.000000           0         5           fstat64

  0.00    0.000000           0         1           gettid

  0.00    0.000000           0         1           sched_getaffinity

  0.00    0.000000           0         1           set_tid_address

  0.00    0.000000           0         1           set_robust_list

  0.00    0.000000           0         1           set_tls

—— ———– ———– ——— ——— —————-

100.00    0.034482                   431         1 total

Get insulin pump history

pumphistory -n 1 &> pumphistory.out # Get insulin pump history

strace -c pumphistory -n 1 &> pumphistory.summary # trace summary

strace pumphistory -n 1 &> pumphistory.full # full trace

Output from the get pumphistory command

Run pumphistory command

retrieving pump history since 2022-10-12 10:54:03

cannot connect to CC111x radio on /dev/spidev0.0

null

/dev/spidev0.0: device is in use

Process startup (process mgmt)

Open files (fs -> driver sysfs)

Open pump device “/dev/spidev0.0”

Open “/sys/class/gpio/gpio4/active_low”

Open “/sys/class/gpio/gpio4/direction”

Close files (fs -> driver sysfs)

Exit program  (process mgmt)

This output shows that the pump busy path is invoked

Pump history command flowchart

Complete strace -c output

2022/10/12 11:54:03 retrieving pump history since 2022-10-12 10:54:03

2022/10/12 11:54:03 cannot connect to CC111x radio on /dev/spidev0.0

null

2022/10/12 11:54:03 /dev/spidev0.0: device is in use

% time     seconds  usecs/call     calls    errors syscall

—— ———– ———– ——— ——— —————-

 40.92    0.003106          25       120           rt_sigaction

 15.91    0.001208         109        11           futex

  8.09    0.000614         204         3           clone

  7.84    0.000595          59        10           mprotect

  7.52    0.000571         190         3           fcntl

  5.23    0.000397          44         9           rt_sigprocmask

  5.20    0.000395          17        22           mmap2

  4.23    0.000321         321         1           readlinkat

  3.77    0.000286          13        21           clock_gettime

  1.29    0.000098          24         4           brk

  0.00    0.000000           0         5           read

  0.00    0.000000           0         4           write

  0.00    0.000000           0         7           close

  0.00    0.000000           0         1           execve

  0.00    0.000000           0         1           getpid

  0.00    0.000000           0         1           access

  0.00    0.000000           0         1           readlink

  0.00    0.000000           0         2           munmap

  0.00    0.000000           0         1           uname

  0.00    0.000000           0         1         1 flock

  0.00    0.000000           0         2           sigaltstack

  0.00    0.000000           0         1           ugetrlimit

  0.00    0.000000           0         5           fstat64

  0.00    0.000000           0         1           gettid

  0.00    0.000000           0         1           sched_getaffinity

  0.00    0.000000           0         1           set_tid_address

  0.00    0.000000           0         7           openat

  0.00    0.000000           0         1           set_robust_list

  0.00    0.000000           0         1           set_tls

—— ———– ———– ——— ——— —————-

100.00    0.007591                   248         1 total

The following diagram shows the run-time context for these commands and their mapping to the Linux subsystems used by them.

System view

The following system view was updated after high level tracing analysis. This system view remains unchanged after the fine grained tracing analysis as expected.

References

https://github.com/ecc1/medtronic/blob/main/button.go

Conclusion

Using strace to trace OpenAPS commands helped us understand the detailed view of system files opened and closed while the commands run. We were able to generate flowcharts for normal and error paths when commands detected device busy conditions. As mentioned earlier, this tracing method gave us insight into the parts of the kernel used by the individual OpenAPS commands.

Credits

The ELISA Medical Working Group would like to sincerely acknowledge Chance Harrison for running the OpenAPS commands and providing information critical for making this effort a successful one.

SPDX-License-Identifier: CC-BY-4.0

This document is released under the Creative Commons Attribution 4.0 International License, available at https://creativecommons.org/licenses/by/4.0/legalcode. Pursuant to Section 5 of the license, please note that the following disclaimers apply (capitalized terms have the meanings set forth in the license). To the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.

To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.

The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.