The Linux Foundation Projects
Skip to main content
search
All Posts By

elisaproject

Apr 05

ELISA CI enablement – Automation tools for easier collaboration

By Ambassadors, Blog, Working Group

Written by Philipp Ahmann, Chair of the ELISA Project TSC (Robert Bosch GmbH) and Sudip Mukherjee, Member of the Tools Working Group (Codethink Ltd)

This article describes how the ELISA Project has enabled Continuous Integration (CI), using the Automotive working group use case as an example. The key goals are to:

  • Make it easier for others to onboard the project 
  • Experience deliverables from the various working groups 
  • Make the work reproducible and more dependable

Additionally, it describes which elements are part of the pipeline along with the tools involved in the creation and testing of the release images. A remarkable element is that the complete pipeline is coupled to the documented development flow. This means that the CI reproduces the steps a developer would do on a PC.

In this way, it approaches people who are new to safety as well as those which are new to Linux. It is a collaborative approach of multiple working groups within ELISA and should foster more collaboration in upcoming ELISA activities.

Motivational factors that enable the CI

In a collaborative project, people of different interests and technical backgrounds unite to work together. After a successful onboarding process, it is important that contributions and work results become visible to the wider community. They need to be properly basellined and distributed fastly to stakeholders. 

A high degree of automation increases the dependability of the development process and reduces failures introduced by human slip. It also gives more time back to developers as they do not have to do everything from scratch.

The ELISA project also relies on external sources for support – for example – the Linux Foundation sister projects Automotive Grade Linux (AGL) and the Yocto Project.  The CI guarantees the rebuild whenever updated packages from AGL or Yocto are provided.

In summary, one can say that these demands drive the enablement of the CI within ELISA Automotive WG and put a set of requirements to it:

  • A entry point for stakeholder with different (technical) background and experience
  • A respect of different stakeholder interests in the work products
  • Faster ramp-up for new project contributors 
  • Better and faster  integration and distribution of new features and improvements
  • Automated and dependable software product creation
  • Extendable testing to validate concepts and assumptions.

With the successful setup of the prototype for the automotive use case, the goal is to scale towards more complex architectures as tackled within the ELISA Systems WG. The basic approach and tools will remain the same in the extension towards new use cases. 

The elements currently involved in the CI include:

All elements are coupled and tied together to build a dependency chain, which makes it easier to automatically detect failures, caused by changes along the elements of the pipeline.

The CI elements and flow within ELISA

The ELISA Project uses GitLab to run the CI pipeline, which interacts with an ELISA hosted build server (which is also used for other purposes as running tools for code improvement). The build sources are taken from Automotive Grade Linux Gerrit, Yocto Project repositories as well as the ELISA GitHub repositories.

Currently, the generated build artifacts are the kernel and root filesystem for QEMU x86_64 image. It is planned to extend these to additional images for physical hardware in the future.

While the CI is running in GitLab, ELISA is hosting source code and documentation in GitHub. This is the starting point of the CI flow. 

The build is triggered on a regular basis once per day to provide a fresh image and check the functionality of the generated image. Due to the usage of cached build artifacts, the longer build times of a full yocto build are reduced to a minimum (in the range of 15 minutes). The generated images are only kept for one day, so that a new download will always have the latest changes included. This further supports the idea of continuous integration. The latest image can be downloaded from the GitLab project packages. 

Also a manual trigger of the CI can be done by referencing a GitHub branch of meta-elisa from a PR. In this way a pre-check from changes is possible before formal images are released from the master. It further reduces eventual regressions. To avoid misuse, access to trigger new builds from referenced GitHub forks is restricted to ELISA working group members.

The strong bound of build server to documentation also makes sure that the docker file for manual build is used to create a docker image, which is then used to create the build artifacts for the QEMU image. Also, this is newly created on a regular basis, to make sure that documentation and tool updates will always result in a working image using the latest documentation.

This is needed to achieve the different entry points for the stakeholders who want to try ELISA artifacts or contribute to the project work.

Dependency to the AGL instrument cluster demo

As already mentioned before, but not directly visible from the build flow, is the dependency to the AGL instrument cluster demo. As ELISA collaborates with AGL, it enhances the existing work of AGL and uses it for demonstrating. The results are still a work in progress; therefore the generated meta-elisa layer is not integrated into the formal AGL repositories (yet). Missing this upstreaming causes the risk that the ELISA automotive WG tell-tale demo will break in case AGL makes changes to interfaces used by ELISA. Similar changes can happen for changes in the yocto-poky base.

By building on a daily basis and testing the generated images for the added functionality, it is guaranteed that any breaking change will become directly visible and immediate actions can take place. In these scenarios the latest image will not be deployed for download and analysis of the reported errors can be done.

As a full yocto build can be resource intensive and time consuming the use of cached build artifacts with an enabled SSTATE helps to reduce the build time to a minimum.

Benefits of using Yocto’s SSTATE cache

Building a yocto AGL image from scratch is taking time and a lot of computation performance. At least 100GB of disc space is consumed and a build can easily take several hours to complete even on decent hardware. 

In order to reduce the build time and limit the necessary disc space, it is possible to turn on cached build results. The so-called SSTATE is created with the first full build and shared for upcoming builds, demanding to only build deltas. Any change in sources and updates are detected and necessary binaries and their dependencies will be recompiled in case needed.

The generated cache is used within the CI, but can also be used for a local build at a developer PCc. 

Additionally, from time to time a new cache is generated to check that the build will also work in case someone builds the system from scratch. Building from scratch can make sense for low network speeds or networks with traffic limitations. This further makes sure that the GitLab pipeline fits to the documentation as much as possible. 

GitLab pipeline coupling to the meta-elisa documentation

The GitLab pipeline basically represents the steps which users would take when manually creating the ELISA enhanced demo of the instrument cluster provided by Automotive Grade Linux. As the current version is based on QEMU it is easy to reproduce the demo at a local machine and also to perform automated testing. This is illustrated by transferring the created images to the OpenQA server hosted by Codethink. 

OpenQA testing

It is important to confirm that the build was successful. As the original AGL demo has been enhanced with the Signal-Source-Control application, which also renders additional screen content, it makes sense to do an output comparison. For the meta-elisa demo OpenQA was used as a proven tool to check proper boot and functionality of the code modifications. This should also avoid potential regressions, e.g. due to external dependency or interface changes. An example output of a comparison as generated by OpenQA can be found below.

In the OpenQA tool, the orange divider in the presented screenshot can be moved left and right with the cursor. This makes it easy to identify differences between reference and recent execution. The greenish frame shows the region of interest which is checked for content. The 100% shows the fit ratio. 

Additionally, during the test execution a screen recording is performed and further logs such as those from serial terminal are provided.

With this option, a fully automated flow from source repository to generated and tested image is achieved. It is possible that working group members can fork the existing meta-elisa repository on GitHub, do modifications, check that it is built properly using GitLab and that the image remains functionally by checking the OpenQA result. This flow would be completely web based without the demand of even having a development environment setup locally on a PC.

However, this may more be used as an exceptional case. The interaction with the system to understand and learn is rather limited in this scenario. As an intention is to develop the system further and gain a larger understanding of the demand of a safety-critical system with Linux based workloads, there are many more entry points for different levels of expertise and personal interest.  

The different entry points to experience the automotive use case 

In order to make it easier for new contributors to join the ELISA Automotive WG, the demo has multiple entry points. 

  • Natively on a development PC using the Readme within meta-elisa
  • Starting from a docker file on a development PC
  • Using a docker image based on the ELISA provided docker file
  • Enhance the build time by using the yocto SSTATE 
  • Download QEMU based build images

Closing the loop to the motivation, the aforementioned entry points fulfill the demands of a collaborative project, where people of different interests come together to collaborate. It is possible to understand the system creation from scratch with an own build from source, or just trace a workload by using the pre-built QEMU images. The development flow is accelerated by providing docker images and cached build artifacts. As the outputs of different pipeline steps are available, debugging gets simplified as well (like checking a native build against a docker build or behavior of the own vs. generated QEMU image).

In summary, a comprehensive documentation (including this post) helps to tailor the pipeline and images to individual demands. Though, to have the best compatibility for exchange with others, the usage of a docker container is recommended. 

Docker container for better compatibility of CI and manual reproduction

Taking into account the large number of different host machines and Linux distributions that exist, from practical experience the ELISA CI is providing a docker file as a common base for development. The docker file is used to generate the docker image, which again is used to build the meta-elisa. By providing the docker file and docker image also to others, who want to create a local build on their machine, influence of the underlying Linux distribution is reduced. Documentation about the docker container creation and usage can be found in the wg-automotive github repository. To pull the docker container directly use:

docker pull registry.gitlab.com/elisa-tech/docker-image/elisa

Some general remarks regarding container should be mentioned for competition:

  • There is still a (limited) dependency to the underlying OS outside the container, as the docker image uses for example the Linux kernel from the host system. However, side effects of missing environment setups, conflicting or overwriting packages are minimized by using the docker container.
  • Regarding build time and performance the impact of the container could be neglected. The duration and system load during the build with and without container usage was very similar. 

The docker file will be kept up to date to reflect demands as per meta-elisa or dependency from yocto or AGL updates. This makes sure to always have the latest features and improvements also for the planned next steps reflected for better reproduction of results. 

Outlook and next steps

As an evolution of the use case, activities have been started to enable a reference image of an example architecture including Xen and Zephyr on real automotive hardware. Any result from the automotive as well as other working groups within ELISA should feed into the example architecture tailorable to specific use case demands.

Additionally having a proper software bill of material (SBOM) generation will also enhance the quality of the reproducible demo. It will support upcoming regulations which require an understanding of the content of your software product. As AGL also works on the reproducible build to generate binary compatibility of builds this may be an additional enhancement considered as future activity.

Summary

  • ✔ By setting up the CI with QEMU image deployment to the openQA testing it is possible to quickly track changes and validate code changes for proper appliance along the chain from source to deployment. 
  • ✔ Active contributors in ELISA have the chance to check their own commits or pull requests by using a standardized GitHub development flow and start the CI by a two line change in the pipeline reference to the meta-elisa fork and branch commit.
  • ✔ The different entry points and precompiled build results enable interested people to decide to build an image from scratch, use a docker container to reduce environment setup steps and reduce build time to a fraction by taking cached build artifacts from the server. 
  • ✔ By provisioning the final QEMU build image, it is even possible to access and boot the generated image without the need of performing an own build.

As all pipeline elements from documentation to later qa testing are dependent on each other and are coupled to an integrated flow. A high degree of dependability and reproduction is achieved with this solution. With a future enhancement of SBOM generation and utilization of reproducible builds project, the system dependability will further be increased. The automated execution supports that aligned development flows are kept and regressions can be tackled before they occur. Best entry points for reproduction of the results are the meta-elisa and wg-automotive docker container documentation in GitHub.

Mar 29

Discovering Linux kernel subsystems used by a workload

By Blog, Technical Update

Written by Shuah Khan, Linux Fellow at the Linux Foundation and member of the ELISA Project TSC, and Shefali Sharma, a senior student at the Meerut Institute of Engineering and Technology in India and ELISA Project Mentee in 2022

Key Points

  • Understanding system resources necessary to build and run a workload is important.
  • Linux tracing and strace can be used to discover the system resources in use by a     

            workload. The completeness of the system usage information depends on the  

            completeness of coverage of a workload.

  • Performance and security of the operating system can be analyzed with the help of tools such as:
    •    `perf <https://man7.org/linux/man-pages/man1/perf.1.html>`_,
    •    `stress-ng <https://www.mankier.com/1/stress-ng>`_,
    •    `paxtest <https://github.com/opntr/paxtest-freebsd>`_.
  • Once we discover and understand the workload needs, we can focus on them to avoid regressions and use it to evaluate safety considerations.

We set out to identify methodology to discover system resources necessary to build and run workload and document the process. Easily available workloads are used to test out methodology and to document the process. This process is intended to be used as a guide on how to gather fine-grained information on the resources in use by workloads.

Within the ELISA project the methodology goes into scaling, by hardening and extending the guidelines in other working groups. The achievements from the Medical Devices Working Group will be taken and put into practice within the Automotive WG use case analysis focusing on safety critical display rendering.

The workload-tracing guide is now available in Linux 6.3 release. It is intended to be used as a guide on how to gather fine-grained information on the resources in use by workloads using strace.

Feb 22

Invest in Yourself With Best in Class Bootcamps – Save 50% Today!

By Blog, LF Training & Certification

Written by LF Training & Certification

Ready to set your career on an award winning path in February? Make the commitment to invest in yourself and your career today with Linux Foundation training and certifications.

Now through the end of February we are offering serious discounts on our most popular training and certifications, with even bigger discounts on Bootcamps and Bundles:

Not sure which is the best offer for your career? Check out our IT Career Roadmap to see how Linux Foundation certifications guide your career toward: Systems & Infrastructure, Network & Security or Software Development.

Don’t wait, offers end February 28, 2023.

Feb 01

Trusted Execution Inside Secure Enclaves – LFX Mentorship Report

By Blog, Mentorship

Işıl Öz, 2022 ELISA Project Mentee and Assistant Professor at Izmir Institute of Technology

Hardware Trusted Execution Environment

Since remote computation has evolved as the cloud infrastructures offer high-performance and low-cost solutions, security has become a concern due to remote computer’s mainte- nance by untrusted parties. As the security mechanisms aim to maintain protection from attackers and enable execution confidentiality and integrity, they help to achieve safety requirements as well by making sure that both program code and data are not corrupted. Therefore, memory protection solutions support safety by ensuring trusted execution, and they target domains requiring both secure and safe execution [4, 15]. Homomorphic en- cryption, which directly works on the encrypted data, enables the handling of user data by third parties safer. However, its computational requirements lead to impractical perfor- mance issues.

Trusted execution environments (TEEs) maintain secure computation in an isolated area of a processor with hardware-assisted technologies. They offer both performance and security improvements by exposing a smaller trusted compute base (TCB) in the environment. Confidential Computing Consortium (CCC) defines TEE as an environment that provides a level of assurance of data integrity, data confidentiality, and code integrity [19]. Data integrity is preventing unauthorized entities from data modification while in use inside TEE. Data confidentiality means that unauthorized entities cannot view data while in use inside TEE. Code integrity is preventing unauthorized entities from code modification while it is being executed inside TEE. Depending on the implementation, TEE additionally may provide the following features:

  • Code confidentiality: TEE protects code while in use from being viewed by unautho- rized entities. For instance, an algorithm having intellectual property issues should not be exposed to third parties.
  • Authenticated launch: TEE enables the launching of a process when only it is verified to be secure.
  • Programmability: TEE consists of any arbitrary code or code that is loaded by a secure source.
  • Recoverability: TEE provides a recovery mechanism that resets the status from a corrupted state to a known safe state.
  • Attestability: TEE measures its origin and current state and maintains evidence that the code has integrity and has not been modified. Different hardware vendors offer TEE implementations in their processor architectures:
  • Intel Software Guard eXtensions (SGX): With SGX, which exists in Intel Xeon pro- cessors, applications run in a secure memory space without access by the untrusted operating system (enclave) and store the data it uses in this memory area.
  • ARM TrustZone: TrustZone is in Arm Cortex-A processors and provides a secure environment for memory, software, bus transactions, and interrupts. The applications running on TrustZone can perform trusted operations with API functions.
  • AMD Memory Encryption Technology: While AMD Secure Memory Encryption (SME) is a hardware-based secure memory encryption technology found in AMD EPYC processors, Secure Encrypted Virtualization (SEV) is the implementation of SME technology used in virtualized systems. We focus on SGX technology and open-source library operating systems that enable its functionality.

Intel Software Guard Extensions (SGX)

Intel’s Software Guard Extensions (SGX) provide hardware-based isolation for secure com- putations by supporting safe execution [18]. SGX reserves isolated memory regions called Processor Reserved Memory (PRM) or enclaves for code and data. The CPU protects the memory regions from all non-enclave memory accesses, including the kernel, hypervisor System Management Mode (SMM) accesses, and DMA accesses from peripherals. Non- addressable memory pages (Enclave Page Cache (EPC) pages) inside enclaves are reserved from the physical RAM and encrypted. Execution flow can only enter an enclave by special CPU instructions, similar to the mechanism for switching from user mode to kernel mode. Enclave execution always happens in protected mode, at ring 3.

Figure 1 presents an example execution flow for the applications utilizing protected memory regions. The initial code and data in an enclave are loaded by untrusted system software. During the loading stage, the system software asks the CPU to copy data from unprotected memory into the enclave and assigns the pages to the enclave. After all the enclave’s pages are loaded, the system software asks the CPU to mark the enclave as initialized, at which point the application can run the code inside the enclave with support for high-level language features. Other parties can perform a software attestation process to verify the identity of the enclave that it is communicating with an enclave and is running in a secure environment.

Open-source LibOS Projects

The library operating system (LibOS) technology enables us to take an application with little or no modifications and protect it in an SGX enclave [7]. While commercial and open-source LibOS options are available, we mention open-source projects and focus on a specific project, Mystikos, our mentorship is built on. The following open-source LibOS projects support Intel SGX:

Gramine [6]: Lightweight libOS, designed to run a single application inside enclaves with minimal host requirements.

Occlum [13]: Memory-safe, multi-process library OS (LibOS) for Intel SGX.

Intel SGX and Linux kernel library [16]: Designed to run existing unmodified Linux binaries inside of Intel SGX enclaves.

Mystikos [12]: Tools and runtime for launching unmodified container images in TEEs.

Mystikos

Mystikos [12] is a runtime and a set of tools for running Linux applications in a hardware- trusted execution environment (TEE). It currently supports Intel SGX. Mystikos aims for the following features: 1) Enable protection of application code and data while in memory through the use of hardware TEEs, 2) Allow users and application developers control over the makeup of the trusted computing base (TCB), 3) Simplify retargeting to other TEE architectures through a plugin architecture.

As shown in Figure 2, Mystikos includes the following components:

  • C-runtime based on musl libc: Handles passing the runtime functions from unsecure part to the secure part of the application.
  • LibOS-like kernel: Implements system calls for secure execution.
  • Kernel-target interface (TCALL): Communicates with the target implementations (SGX target based on Open Enclave SDK or Linux target for only verification on non-SGX platforms).
  • Command-line interface: Offers an interface for the programs targeting execution inside the enclave (myst exec-sgx or myst exec-linux).
  • Some related utilities: Tools for building an application for secure execution (myst-appbuilder), creating an archive for target program (myst mkcpio or myst mkext2), packaging an application (myst package-sgx). Figure 2: Mystikos architecture [12]. Figure 3 presents the application workflow when developing application in Mystikos, which consists of four steps:

Figure 3 presents the application workflow when developing application in Mystikos, which consists of four steps:

  1. Develop application: Develop the application using programming models of our choice.
  2. Create appdir: Create and populate an appdir directory that contains compiled application and essential dependencies.

3. Run in Mystikos: First create a root file system (rootfs) from appdir by using myst mkcpio or myst mkext2. Then run the application by using myst exec-sgx or myst exec-linux.

4. Package application: Sign and package the application.

Activities During the Mentorship

At the beginning of the mentorship period, I investigated the open-source libOS projects enabling and implementing SGX features (mentioned in Section 3). With a theoretical background in operating systems and processor architecture, I tried to understand the ex- isting implementations. In the meantime, I learned about some practical issues by following the LFX courses about Linux kernel and open-source project development (like GitHub is- sues and pull requests) to get hands-on experience. Specifically, I enrolled two LFX training courses: A Beginner’s Guide to Open-Source Software Development (LFD102) [3] and A Beginner’s Guide to Linux Kernel Development (LFD103) [2].

After deciding to work on Mystikos project, which is closely related security and safety issues, I started by examining open issues in Mystikos Github repository.

The first issue I have looked at was the following: Memory leak in myst syscall execve ([11]): It was about a potential bug in the function myst syscall execveat inside the file /kernel/syscall.c. The reason seemed to have an unfreed memory allocation in the function myst exec inside /kernel/exec.c file. After discussion with the maintainers re- garding the issue, I created a pull request and submitted it as a fix (Figure 4). However, the maintainers realized that the issue has been already fixed in the previous commits, so my commit has not been merged.

After being introduced in Mystikos and getting familiar with the code, I contacted to the maintainers. Vikas Amar Tikoo replied and offered me to work on improving debug-gability by either writing gdb extensions, improvements to strace mechanism in Mystikos, interactive shell support, or integration with native debugging tools like valgrind. Then he created an issue on github that I can work on: Interactive bash support issues ([9]). The aim is to run bash shell in Mystikos. However, Mystikos complains about it. I could replicate the case in my local machine (Figure 5). The problem seems that the ioctl requests/commands [10] have not been supported by Mystikos. I submitted a pull request with TIOCGPGRP and TIOCSPGRP ioclt system calls (Figure 6). I have been working on the issue and communicating with the maintainers about the implementation details.

Additionally, during my mentorship period, I attended the following online activities:

  • SGX Community Day Virtual Event, July 26-27, 2022 [1]: Inspirational talks from academics, startups, and big corporations about SGX implementations.
  • ELISA summit, September 7-8, 2022 [5]: Introductory overview, emerging trends, and current topics in utilizing open-source software for safety-critical applications.
  • SOAFEE Virtual Symposium, November 16-17, 2022 [17]: Presentations by open- source and independent software vendors organized by the Scalable Open Architecture For Embedded Edge (SOAFEE) Special Interest Group.

Moreover, as a researcher working in the field of computer architecture, the mentorship helped me to improve myself in the hardware security area. I have submitted a national research project titled as Security-Performance Tradeoff Analysis for Embedded Systems with GPUs, where we target to analyze both the security and performance of the embedded applications running inside enclaves. We aim to utilize the open-source LibOS projects and port the program executions partially into the enclaves. Mainly, we target to extend the mentorship work for safety-critical embedded CPU-GPU systems. Additionally, my mentorship encouraged my students to work on open-source projects and be included in mentorship programs.

Thanks to the LFX mentorship program supported by ELISA, I found an opportunity to be introduced to the open-source software community and gained a practical hands-on experience in open-source software development. The mentorship was helpful in terms of both enhancing my hands-on skills and learning the open-source project development pro- cess. While I have a strong background in theoretical operating systems and computer architecture area, especially my operating systems software skills have substantially im- proved. Moreover, the regular meetings with my experienced mentor and communication with the open-source project maintainers were very helpful for both technical and social skill development.

After getting experience on the Mystikos project, I am working on the issue and will continue working on it after my mentorship period. With this experience, I am planning to contribute to other open-source LibOS projects by combining my research interests and practical experience. After being introduced to the open-source community, I believe that I will be able to find more opportunities in the related domains requiring both security and safety. Since I learned that secure execution environments/containers maintain secure and

Safe execution inside enclaves, we can utilize them for different applications in safety-critical systems.

References:

  1. 3rd sgx community day virtual event. https://community.intel.com/t5/Blogs/ Tech-Innovation/Data-Center/Third-SGX-Community-Day/post/1393177. Ac- cessed: 2022-11-16.
  2. A beginner’s guide to linux kernel development (lfd102). https://trainingportal.linuxfoundation.org/learn/course/ a-beginners-guide-to-linux-kernel-development-lfd103. Accessed: 2022- 11-16.
  3.  A beginner’s guide to open source software development (lfd102). https://trainingportal.linuxfoundation.org/courses/ a-beginners-guide-to-open-source-software-development-lfc102. Accessed: 2022-11-16.
  4. Eclipse software defined vehicle working group. https://sdv.eclipse.org/. Ac- cessed: 2022-11-23.
  5. Elisa summit. https://events.linuxfoundation.org/elisa-summit/. Accessed: 2022-11-16.
  1. Gramine – a library os for unmodified applications. https://gramineproject.io/. Accessed: 2022-11-16.
  2. Intel software guard extensions. https://www.intel.com/content/www/us/ en/developer/tools/software-guard-extensions/get-started.html. Accessed: 2022-11-16.
  3. Intel software guard extensions (intel sgx) web-based training. https: //www.intel.com/content/www/us/en/developer/articles/technical/ intel-sgx-web-based-training.html. Accessed: 2022-11-16.
  4. Interactive bash support issues. https://github.com/deislabs/mystikos/issues/ 1423. Accessed: 2022-11-16.
  5. ioctl(2) — linux manual page. https://man7.org/linux/man-pages/man2/ioctl. 2.html. Accessed: 2022-11-16.
  6. Memory leak in myst syscall execve. https://github.com/deislabs/mystikos/ issues/946. Accessed: 2022-11-16.
  7. Mystikos. https://github.com/deislabs/mystikos. Accessed: 2022-11-16.
  8. Occlum – a library os empowering everyone to run every application in secure enclaves. https://occlum.io/. Accessed: 2022-11-16.
  9. Pull request – ioctl system calls. https://github.com/deislabs/mystikos/pull/ 1439. Accessed: 2022-11-24.
  10. Scalable open architecture for embedded edge (soafee) project. https://soafee.io/. Accessed: 2022-11-23.
  11. Sgx-lkl library os for running linux applications inside of intel sgx enclaves. https: //github.com/lsds/sgx-lkl. Accessed: 2022-11-16.
  12. Soafee virtual symposium. https://soafee.io/blog/2022/virtual_symposium/. Accessed: 2022-11-16.
  13. Victor Costan and Srinivas Devadas. Intel sgx explained. Cryptology ePrint Archive, 2016.
  14. A Publication of The Confidential Computing Consortium. Confidential computing: Hardware-based trusted execution for applications and data. Technical report, January 2021.
Jan 25

Why We Need Open Source Mentorship Programs

By Blog, Mentorship

Written by Jason Perlow, Editorial Director at the Linux Foundation

Mentorship programs in open source are critical for the growth and development of the open source community, and the LFX Mentorship program is no exception. The program’s participants find it so valuable that a whopping 99% of the graduates felt the program to be beneficial, and 47% said it helped them get a job.

In a recent study conducted by Linux Foundation Research, Mentorship in Open Source, with a report authored by Linux Foundation Editorial Director Jason Perlow, over 100 mentees from the LFX Mentorship graduating classes of 2020 and 2021 were surveyed and interviewed about their experiences during participation and postgraduation. This includes several mentorships with the ELISA Project.

The study explores the intrinsic, economic, and career value of mentorship programs in open source, highlighting the importance of such programs for both mentees and mentors.

Benefits of mentorship programs

One of the key benefits of mentorship programs is the intrinsic value they provide. Mentors can share their knowledge and experience with mentees, helping them develop the skills and knowledge they need to succeed in the open source community. Mentees, on the other hand, can learn from experienced professionals, gaining valuable insights and advice that they can use to advance their careers.

In addition to the intrinsic value of mentorship programs, there are also significant economic benefits. Open source projects rely on the contributions of a wide range of individuals, from developers to designers and testers. By providing mentorship opportunities, organizations can help attract and retain talented individuals, which can help ensure the project’s long-term success. This is particularly important for open source projects that rely on community contributions, as these projects need to attract and retain a steady stream of contributors to be successful.

Here are a few more examples of why mentorship programs are important for the both the participants and the open source communities: 

  • Hands-on experience: Open source mentorship programs allow individuals to work on real-world projects, which can help them gain practical skills and experience.
  • Networking opportunities: Participants in open source mentorship programs can interact and collaborate with experienced developers and other professionals in the field. This can lead to valuable networking opportunities and potential job leads.
  • Flexibility: Participating in mentorship programs can be flexible, allowing individuals to work on projects at their own pace and in their own time. This allows for a better work-life balance and allows individuals to continue working on other projects or responsibilities. An example of flexibility is having training sessions recorded so that participants can access them at more convenient times.
  • Learning from experts: Having access to open source project maintainers with deep experience in coding is one of the key benefits of mentorship programs. Participants can learn from their mentors and benefit from their knowledge and experience.
  • Building a portfolio: Participating in mentorship programs provides the opportunity to build a portfolio of work, which can demonstrate to potential employers the skills and experience that an individual has gained.
  • Building a community: Running mentorship programs helps to build a community of developers and professionals who can provide support, guidance, and mentorship to others. This can be a valuable resource for career advancement.

Challenges faced by mentorship programs

One of the challenges mentorship programs face is the lack of funding and support. Many mentorship programs are run by volunteers, who may not have the resources or support to mentor mentees effectively. This can make it difficult for mentees to get the support and guidance they need, leading to frustration and disengagement.

Another challenge faced by mentorship programs is the lack of structure and support. Many mentorship programs are informal, with mentors and mentees often left to figure out how to work together best. This can make it difficult for mentees to get the support and guidance they need, leading to frustration and disengagement.

The LFX Mentorship program, sponsored by the Linux Foundation, aims to address these challenges by providing mentees with the support and guidance they need to be successful in the open source community. The program provides mentees access to a wide range of resources and support, including training and development opportunities, mentoring and coaching, and networking opportunities. The program also provides mentors with the support and guidance they need to be effective mentors, including training and development opportunities, mentoring and coaching, and networking opportunities.

Insights from the report

Here are a few highlights of valuable insights gained from the mentorship study:

  • Improving Diversity: Mentorship creates opportunities for a healthy succession of open source project contributions and leadership.
  • Career Advancement Opportunities: 69% of mentees have seen their career advance because of mentorship, with 47% saying that the program helped them get a job.
  • Increased Income: 67% of employed mentees report increased income after program participation.
  • Improved Contributions: 85% of mentees are or are willing to contribute to the project they were involved in after mentorship.
Download Report

Who should read the report?

One of the report’s primary audiences is developers interested in open source mentorship. The report provides detailed information on the types of mentorship programs available and the skills and experience typically required for mentors and mentees. Developers interested in becoming a mentor or mentees in open source can use the report to understand the process better and what is involved.

Another important audience for the report is managers and leaders within open source communities. The report provides valuable insights into the challenges and opportunities of mentorship in open source and can help managers and leaders identify areas where they can improve their mentorship programs. Additionally, the report guides how to design and implement mentorship programs that are effective and sustainable.

Conclusion

The Linux Foundation’s report on mentorship in open source is an important resource for many individuals and organizations involved in open source software development. In partnership with CNCF, ELISA Project, Hyperledger, Open Mainframe Project and OpenSSF, the report provides valuable insights into the current state of mentorship in open source and best practices for mentorship programs. Developers, managers, community leaders, educators and trainers, and researchers can all benefit from reading the report and using the information it contains to improve their own mentorship programs and contribute to the overall success of the open source community.

Jan 11

My ELISA Mentorship

By Blog, ELISA Summit, Mentorship

Shefali Sharma, a senior student at the Meerut Institute of Engineering and Technology in India, was a mentee of the ELISA Project starting in March 2022. Her mentors were Shuah Khan, Linux Fellow and Kernel Maintainer at the Linux Foundation and Chair of the ELISA Technical Steering Committee at that time, and Milan Lakhani, Co-Chair of the ELISA Medical Devices Working Group and Software Engineer at Codethink. During her mentorship, Shefali learned a lot about the Linux kernel, Core C programming, and various tools and techniques for analyzing the kernel like strace, ftrace, cscope, perf. 

Shefali will be sharing her key learnings on Thursday, January 19 at 12:33-12:43 am ET at the LFX Mentorship Showcase, a virtual event that gives Linux Foundation mentees an opportunity to present their experience with their mentorship. This virtual event, which takes place on January 18-19, is free to attend and open to anyone who would like to learn more about the experiences of LF Mentorship Program mentees, find out more about some of the programs our projects are working on, recruit new talent, and support new developer contributions. Register for the free event here.

If you can’t make the LFX Mentorship Showcase, you can check out her findings in this video from the ELISA Summit (September 2022) about kernel tracing.

You can also learn more in Shefali’s Medium blog here.

Dec 07

Linux in Aerospace: A Personal Journey

By Blog, Working Group

Written by Steven H. VanderLeest, Software Engineering Technical Lead at The Boeing Company and Chair of the ELISA Aerospace Working Group

Introduction

From the early days of Linux, I was a fan of this innovative, open-source Operating System (OS). I appreciated it as a hobbyist, helping me run Linux at home. I appreciated it as an educator, helping my computer engineering students walk with Linux through OS concepts. However, as a professional working in the safety-critical domain of aerospace, I wondered: could Linux fly?

My Pre-flight Taxi with Linux

My journey with Linux had its roots in the 1980s before Linus Torvalds introduced his new OS to the world in 1991. During my undergraduate degree in the 1980s, my engineering program had some labs equipped with the relatively recent IBM Personal Computer (PC). The machines were amazing, but my ability to command their power was somewhat limited by the OS, which was the Microsoft Disk Operating System (MS-DOS). When I reached my third year, I gained access to a Sun Workstation running SunOS, a variant of Unix. I quickly learned to appreciate the rich menagerie of shell commands, the power of combining them with redirection such as pipes, and the aesthetics of the fledgling X-Windows GUI.

I first heard about Linux in graduate school in the early 1990s at the University of Illinois at Urbana-Champaign. My doctoral thesis was on Input/Output (I/O) performance, especially on multiprocessor systems. My research analyzed and quantified I/O performance on OSs such as SunOS, SGI IRIX, DEC OSF/1, HP-UX, and Linux. One key finding of my research was that I/O performance could be impacted by the interference caused by unrelated transactions contending for shared resources within a multi-processor system. The magnitude of the impact was heavily dependent not only on the computing hardware architecture but also on the architecture of the OS. Interference could even occur on a uni-processor where independent processes had I/O tasks clustered in time.

As an educator, I applied Linux in my teaching. After finishing my Ph.D., I returned to my alma mater, Calvin College (now University), to take a position as a professor of engineering, teaching computer engineering topics. Linux provided a rich learning environment where my students could look under the hood while learning about operating systems. The transparency of open-source code made an ideal environment for learning and innovation. I also wanted to share my love for working at the interface between computer hardware and software. Studying the Linux kernel provided key insights into how the OS manages the hardware on behalf of applications. The overall system’s performance will improve if the OS is reasonably tuned to take advantage of the hardware architecture.

As a hobbyist, I used Linux at home. I set it up on any extra desktop or laptop I could get my hands on. The whole family got involved when I set up MythTV, an open-source streaming media system, and installed it on a spare Linux desktop system along with an expansion card to capture and record live television. We were asynchronously watching programs and never missing an episode well before any of our friends or neighbors followed suit with ReplayTV or TiVo.

As an engineering professional, I found opportunities to bolster my work with Linux. The challenge was that my employers often required MS Windows as the standard a bureaucratic IT department imposed. Nevertheless, I discovered ways to use Linux by dual-booting or a LiveCD approach and eventually run Linux in a virtual machine using hypervisors like VirtualBox. Like its Unix forebears, Linux was much more stable and reliable than Windows. Even if an application program went astray, I got a segmentation fault warning at most, and the other processes continued. Windows was prone to the Blue Screen of Death, bringing the system to a halt much too often. While it might be distressing to lose your work when this happened, losing a few minutes of labor (or hours if you didn’t save often) was a minor albeit annoying inconvenience. I couldn’t expect higher reliability since that wasn’t a use case for office desktop systems. I quickly realized that Windows doesn’t apply to safety-critical systems.

I also would not expect an operating system designed for an office desktop/laptop to work for embedded systems where the available main memory and secondary storage are limited. Embedded computing platforms are all around us but hidden inside our vehicles, more sophisticated consumer electronics, and smart devices. Windows might not work in these use cases, but Linux could! I started using Linux on embedded development boards when chip manufacturers such as Freescale (later NXP), Intel, Texas Instruments, and others began providing a Linux Board Support Package. The chip makers found this approach was the most effective way to get developers up and running quickly on their new hardware.

Taking Flight with Linux

Within safety-critical domains such as aerospace, Linux provides the foundation for multiple software development environments that run on desktops and laptops. As we move toward distributed development, Linux is a ubiquitous cloud guest OS.

For embedded, safety-critical applications, Linux is less common than a Real-Time Operating System (RTOS). However, a group of Linux developers has been slowly improving real-time performance since the 1990s. Attention coalesced into the PREEMPT-RT patch since 2004, with key parts of the patch making their way to the mainline kernel code. Today, almost all PREEMPT-RT functionality is mainlined but must be enabled through kernel configuration parameters. As for the safety-critical need, in the early 2010s, several research groups examined Linux as a foundation for an Integrated Modular Avionics (IMA) system. I led one of these efforts as the Principal Investigator for a Small Business Innovation Research (SBIR) contract with the US Defense Advanced Research Projects Agency (DARPA). We developed a proof-of-concept safety-critical system that combined the Xen hypervisor with Linux as a guest OS, to provide ARINC 653 partitioning, a key standard related to IMA.

Over the past decade, multiple private endeavors have applied Linux in aeronautical and astronautical computing systems, even platforms with modest safety criticality, though only a few of these efforts have been publicized. Demonstrating that software is reliable enough for flight is ambitious. I work for Boeing, one of the aerospace companies tackling that challenge. The next section provides an overview of the four key characteristics necessary to put aircraft using Linux into the air.

Developing Software for Aerospace is Challenging

For use in avionics (an electronic computing platform used on an aircraft), the software must be fast, deterministic, embedded, and assured.

Fast

For use in avionics, Linux must be fast. The Linux developer community is already heavily focused on speed, constantly innovating kernel performance improvements.

The aerospace industry can largely leverage the Linux community effort toward high performance. There may be a few specialized devices where drivers must be further optimized. However, those devices will almost always follow the existing design patterns and take advantage of community innovations, such as io_uring. Another example of an area that might need more attention is boot time. For aerospace, certain fault-tolerance techniques require a fast boot-up (or in-air re-boot) time. In these cases, the system must be operational in only a few seconds or even less.

Deterministic

For use in avionics, Linux must be deterministic. Remember the action thriller series 24? Jack Bauer (played by Kiefer Sutherland) would introduce the series with a voice-over claiming “events occur in real-time”. The audience understood that we were watching as if it were airing live. This commonly understood definition of real-time is not quite the same idea as a real-time computing system. For an RTOS, real-time means that the response to critical events will occur within a deterministic amount of time, even in the worst case. Most computing systems- hardware and software- are tuned to optimize the average response time. Most users and actions enjoy a rapid response, but sometimes at the expense of a slow response for certain users or certain actions. A deterministic system is not necessarily fast — it simply means that we can bound, with confidence, the maximum for critical response times. We want a guaranteed maximum response time in a real-time system, even in the worst case. If we were grading responses like students, we don’t care if the best score was an A+ or the average score was a C. We care that the worst score is still a passing grade in real-time systems. Let’s say the system must always respond within 50 milliseconds, or something bad happens. Over a series of tests, perhaps you find that the fastest response is 12 milliseconds, the average is 27 milliseconds, and the worst is 42 milliseconds. For determinism, we only care that the worst response is still under the requirement (in this example, it appears to be meeting our needs).

The aerospace industry can leverage the Linux community’s effort toward determinism. The PREEMPT_RT patches developed over the last 20 years have largely been mainlined, but must still be configured to enable them. Deterministic boot time has received less attention than deterministic response time, but both are important for aerospace applications.

Embedded

For use in avionics, Linux must be embedded. Embedded use cases are constrained with limited size, weight, and power. The most widely deployed embedded instance of Linux is probably the Android OS, used on the largest number of smartphones around the globe today. The vast majority of the billions of embedded devices that make our digital world run smoothly are not this visible — they are under the hood in your car, behind the panel of your home thermostat, and in many other behind-the-scenes locations.

Many industries, including the aerospace industry, continue to turn to Linux for embedded systems. Chip manufacturers continue to support Linux, often the first OS for which they provide starter software development kits. Developers from across the open-source community continue to develop drivers for new devices.

Assured

Regulatory agencies often oversee safety-critical systems to ensure the software is correct to a high confidence level. Because public safety is at stake, the agencies generally have the authority to enforce standards before a product can be released. For use in avionics, Linux must be assured. For avionics software in civilian aircraft, the authority to approve flight certification is specific to a geographic region. For example, in the United States, it is the Federal Aviation Administration (FAA); in most of Europe, it is the European Union Aviation Safety Agency (EASA).

The details of safety standards vary across industries such as nuclear, automotive, medical, aeronautical, rail, and others. However, the same basic concepts are found in all of them, such as expert peer review or formal means of verification and validation to show the software is suited to purpose. Most have two aspects: ensuring the software is reliable (it does the things we want) and safe (it does not do things we do not want).

A key standard for avionics software is DO-178C, which describes software development life cycle processes and objectives that must be met. DO-178C defines five software levels. The lowest is level E, where a software bug has no impact on the safety of the crew or passengers. An example might be the passenger entertainment system. The highest is level A, where a software bug could have catastrophic results. An example might be the flight control software that responds to pilot commands.

The aerospace industry can leverage much less from the Linux community regarding assurance than the other criteria stated earlier. On the one hand, Linux has been extensively field-tested, so it has a strong product history. Due to the crowd-sourcing nature of open source, Linux likely has more expert peer reviews than any other existing software. Assurance of Linux also benefits from the reasonably large number of tests available within several test frameworks. On the other hand, Linux was not designed expressly for aerospace, nor even for safety-critical use cases in general. The design has been much more iterative and ad-hoc, making it more challenging to demonstrate the correct design to software safety regulatory authorities.

Conclusion

Linux is already being used in flight-certified systems at level D. Aerospace companies like Boeing are now poised to use Linux more broadly and at higher levels of assurance, with groups like ELISA leading the effort. ELISA is the Enabling Linux In Safety Applications project under the Linux Foundation. Its mission is to make it easier for companies to build and certify Linux-based safety-critical applications. ELISA recently formed a new working group focused on Aerospace, which will tackle some of the challenges outlined above. We are just getting this group started and welcome new members!

I have crawled, walked, and run with Linux. Now it is time to fly!

For more information

This article previously ran on Linux.com.

Nov 28

Cyber Monday

By Blog, LF Training & Certification, Linux Foundation

With the new year approaching, what better time is there to plan for the future? 

Many technology professionals are looking for ways to level up in their career in 2023. In the spirit of planning for big moves and meeting opportunities well-prepared, the Linux Foundation is once again offering CyberMonday promotions – 30-65% off – on our most popular courses, certifications, bootcamps and bundles. 

Courses, Certifications and Bundles

Linux Foundation Training & Certification courses and certs are among the most well-recognized and respected in the technology industry. 

Right now you can save 50% off any course or certification in our catalog, including our popular CloudNative certs – CKA, CKNA, CKS and KCNA. (While you’re browsing our course catalog, be sure to check out our brand new offering – the Linux Foundation Cloud Certified Technician (LFCT) that was launched just this week!)

Even better, you can save even more – 65% off – when you combine certifications and courses, or purchase multiple certification exams with bundles and power bundles. 

Once purchased you’ll have access to your course and will be able to schedule your exam for up to a year. Our certifications come with verifiable digital badges that are good for up to three years once the exams are passed. 

Bootcamps and In-Person Training

Looking for a deeper-dive into the subject matter? Linux Foundation bootcamps offer incredible value with course materials, online mentorship and instructor feedback, and a learning community to help you master DevOps, Cloud Native Development and Cloud Engineering skills. At 65% off, you won’t find a better deal on a quality bootcamp to help you learn these subjects anywhere.

And, if you’re missing the in-person training experience – good news! The Linux Foundation also offers virtual, instructor-led training. 

Introducing Skillcreds

Finally, this week the Linux Foundation Training & Certification team has launched an entirely new credential series focused on practical, tool-specific technologies. Skillcreds for Vim, Git, Helm, YAML and Bash are available at 30% off now through 12/5/22.

This is your chance to get incredible savings on world-class training and certifications that can boost your earning potential and help you grow in your career. Talk to your employer about whether or not your company will help pay for your training. But act quickly – these incredible savings come only once a year and are good only through December 5, 2022.

Oct 05

Addressing Space Isolation for Enhanced Safety of the Linux Kernel (Video)

By Blog, Technical Update, Working Group

Written by Igor Stoppa, Senior Software Architect at Nvidia

For more than two decades, Linux has made inroad in new fields of applications, from data centres, to embedded. We see now a growing demand for Linux in safety critical applications, ranging from automotive to robotics, to medical appliances.

However, Linux was not designed with these applications in mind, and unsurprisingly it is not an ideal fit, at the moment.In particular, one major pain point is the very limited resilience to spatial interferences originating from within the kernel itself.

Furthermore, the code base if much larger than what can be found in other operating systems traditionally found in safe applications. This is also compounded by the fact that Linux does not follow the processes traditionally in use for Functional Safety.

Summary

In the video, I describe my ongoing experiment of modifying the Linux kernel, to introduce a form of Address Space Isolation, meant to provide a mechanism enforcing freedom from interference. The presentation describes the problems, possible means to address it, and the current progress with the implementation. You’ll see a methodology for the safety analysis of a Linux system and mechanism for improving the safety of selected components.

This presentation ties both into the scope of the Linux Features for Safety-Critical Systems Working Group and the Critical SW track at Open Source Summit Europe. Though this work is not formally sponsored nor endorsed by ELISA, it is something I shared with the community for brainstorm and discussion purposes.

If you’d like to learn more about the Linux Features for Safety-Critical Systems Working Group or you’d like to continue this conversation, please join the mailing list or a WG meeting here.

Sep 23

Open Source Automation Development Lab Survey

By Blog, Industry Partners

Written by Philipp Ahmann, an ELISA Ambassador, Chair of the Automotive Working Group, Chair of the Systems Working Group, and Technical Business Development Manager at Bosch 

One of ELISA’s industry partners Open Source Automation Development Lab (OSADL), which is the leading competence network providing services to use Open Source Software in industrial products, is hosting a survey to get a better picture on demands and pain points of industrial use of embedded Linux.

Their mission is to:

  • Support with services and products to use Open Source Software in industrial products sustainably.
  • Provide broad knowledge and a wide network of experts to accompany your product development throughout all stages.
  • Assist in complying with legal requirements, safety and security standards and we certify your products and processes.
  • Defend the interests of our members and those of the Open Source community.

They are independent and invite any interested party to join. Currently, they are asking for help in a survey until the end of September: https://www.osadl.org/Linux4Industry.

The idea behind the survey is simple:

The landscape of Linux distributions currently in use is very heterogeneous. A way forward can be to pick one or more existing distributions that come closest to an ideal industry-grade Linux and enhance them by adding missing components. In this way the current (unsatisfactory) situation of Linux distributions for embedded systems in industry applications can be significantly improved. The survey should help to identify these potential distributions, missing components and key additions needed. 

The results will be published to those who participated in the survey. Fill out the survey today by September 30 here: https://www.osadl.org/Linux4Industry.

Close Menu