Introduction to Singularity

Welcome and Introductions

Overview

Teaching: 10 min
Exercises: 0 min
Questions
  • Instructor introductions

Objectives
  • Overview of the course

FIXME

Welcome and Introductions

Welcome to this tutorial on using the Singularity containerization framework to install customizable applications.

This tutorial is a primer on how to use the Singularity containerization framework to install customizable applications and use those containers on Linux based computer systems.

Software containers and containerization technologies are emerging as a valuable research artifact to facilitate and promote scientific reproducibility. Software containers provide a separation of the containerized application environment from the host system, acting as an abstracted virtualization layer within the operating system of the host machine. One of the primary reasons behind this rapid adoption of containerization frameworks is that this abstracted virtualization layer allows containers to be easily migrated from one system and executed on another as long as the underlying containerization framework is present on a given system. Another major factor driving adoption, particularly in the computational biology and bioinformatics domain, is that this abstraction and virtualization allows application and container developers to bundle up the necessary prerequisites and software dependencies required by their applications, allowing their applications to be easily transferred and executed by others seeking to utilize them on their own datasets.

This tutorial will allow participants to interact with and build Singularity containers for various computational biology and bioinformatics applications using cloud hosted resources (ex: instances in AWS, GCP, or Azure) provided to each participant for the duration of the tutorial and configured with a host system Linux operating system and Singularity containerization framework.

Presenters

Objectives

Connecting to the Workshop Resources

Please use the instructions below to connect to the Google Cloud virtual machine machine instances used for this course.

1. Click here to view a list of virtual machines (VMs) for this workshop. Each one is represented by a unique Public IP address.

2. Find your name in the list or place your name in an empty space in the “Workshop User” column. This ensures only one user will be logged in per virtual machine instance.

image

3 Copy the public IP address associated with your VM.

Mac Users

image

Windows User

image

image

image

Click Here to view a PDF version of the instructions above

Key Points

  • First key point. Brief Answer to questions. (FIXME)


What is a Container

Overview

Teaching: 15 min
Exercises: 0 min
Questions
  • Containers and Scientific Reproducibility

Objectives
  • How containers contribute to scientific reproducibility

FIXME

What is a Container?

Slides available (download)

Scientific Reproducibility

There is rising concern about a lack of repeatability, replicability, and reproducibility in science and engineering with a large number of public failings across domains ranging from algorithm development and design to cancer genomics, medicine and economics, where the results of a number of high profile publications failing to be replicated when reproducibility studies were conducted. Variation in data collection methodologies, experimental environments, computational configuration, the lack of detailed and intricate documentation, and more are leading to a call for enhanced peer review and validation of experimentally produced artifacts. These experimentally produced artifacts represent the digital artifacts produced during the course of research that are either the inputs or outputs of the study, and can take the form of input data sets, raw data, software systems, scripts to run experiments or analyze results.

As part of this process to provide better validation of experimental works, a number of publishers are engaged in efforts to encourage and promote better scientific reproducibility. The Association of Computing Machinery has begun and embraced a campaign to produce digital badges representing various levels of digital artifact validation to accompany publications. These badges indicate if the digital artifacts have been found to be functional, reusable, available, and if the underlying studies have validated results and reproduced results.

Containerization

In recent years, containers and containerization technologies have emerged and gained significant adoption and importance among developers within industry and within the software development landscape. Applications distributed as containers have the same installation overhead as a normal installation of the container, but with a slight additional overhead requiring knowledge of the containerization framework. However, once the applications are containerized and the containers are made available for others, the full burden of application installation and configuration is abstracted from the end users of those containers, so long as they can utilize the underlying containerization framework on their systems. This allows users to download and run these pre-built containers without the need for often lengthy and detailed application installation procedures. Containers operate as a virtualization layer within the operating system of the host machine, and provide a separation of the containerized application environment from the physical resources of the hosts. Previous work has shown that applications bundled and distributed as containers and executed through containerization frameworks have very little execution overhead and exhibit almost no difference in application performance than the same application executing non-containerized on a “bare-metal” system. This separation and abstraction allows containers to be easily migrated from system to system as long as the underlying containerization framework is capable of being executed and has given significant traction to their adoption, particularly in cloud computing ecosystems. They typically include all the software and dependencies to run a single application, allowing them to facilitate DevOps principles and support for microservices (an environment dedicated to a single task).

The microservice development strategy, encapsulating all the necessary dependencies for a single application into a lightweight virtualization environment makes containers a natural fit for the software environments, processing scripts, and applications produced as digital artifacts in the course of scientific and academic research. A large portion of the artifacts generated by research groups are produced by graduate students and postdoctoral associates, are developed in unique or isolated environments, would not be considered production-level in terms of support and documentation, and often, after publication or when the student or postdoctoral associate moves on to other positions, are at best poorly updated and maintained. This leads to situations where researchers working to reproduce or build on this previous work are forced to engage in almost archaeological expedition level efforts with the identification and configuration of various undocumented application dependencies.

Docker has emerged as the clear leader in containerization frameworks, particularly for cloud computing ecosystems, but information security concerns about possible root privilege escalation exploitations have slowed its adoption within on premise data centers and academic high performance computing centers. To alleviate these potential security concerns, an alternative containerization framework, Singularity, was developed at Lawrence Berkeley National Laboratory, and has begun to make significant inroads as the a leader in containerization frameworks for non-cloud deployments. The Singularity containerization framework differs from the Docker containerization framework in that by design it does not provide a mechanism to allow for privilege escalation within the runtime environment, so a user within the container is the same user as outside the container. This makes it an ideal candidate for execution on large multi-tenant HPC systems, such as those found in academic and research institutions.

Key Points

  • First key point. Brief Answer to questions. (FIXME)


Singularity Basics

Overview

Teaching: 30 min
Exercises: 0 min
Questions
  • Singularity basics: pull, shell, exec, search, bind mounts

Objectives
  • Singularity pull command

  • Singularity shell command

  • Singularity exec command

  • Singularity search command

  • Singularity bind mounts

FIXME

Singularity Basics

Learn Singularity basics: pull, shell, exec, search, bind mounts (25-30 min)

Now that we have a brief overview of what containers are, let’s get into how we can actually interact with them.

$ singularity --help
Linux container platform optimized for High Performance Computing (HPC) and
Enterprise Performance Computing (EPC)

Usage:
  singularity [global options...]

Description:
  Singularity containers provide an application virtualization layer enabling
  mobility of compute via both application and environment portability. With
  Singularity one is capable of building a root file system that runs on any 
  other Linux system where Singularity is installed.

Options:
  -c, --config string   specify a configuration file (for root or
                        unprivileged installation only) (default
                        "/etc/singularity/singularity.conf")
  -d, --debug           print debugging information (highest verbosity)
  -h, --help            help for singularity
      --nocolor         print without color output (default False)
  -q, --quiet           suppress normal output
  -s, --silent          only print errors
  -v, --verbose         print additional information
      --version         version for singularity

Available Commands:
  build       Build a Singularity image
  cache       Manage the local cache
  capability  Manage Linux capabilities for users and groups
  config      Manage various singularity configuration (root user only)
  delete      Deletes requested image from the library
  exec        Run a command within a container
  help        Help about any command
  inspect     Show metadata for an image
  instance    Manage containers running as services
  key         Manage OpenPGP keys
  oci         Manage OCI containers
  overlay     Manage an EXT3 writable overlay image
  plugin      Manage Singularity plugins
  pull        Pull an image from a URI
  push        Upload image to the provided URI
  remote      Manage singularity remote endpoints, keyservers and OCI/Docker registry credentials
  run         Run the user-defined default command within a container
  run-help    Show the user-defined help for an image
  search      Search a Container Library for images
  shell       Run a shell within a container
  sif         siftool is a program for Singularity Image Format (SIF) file manipulation
  sign        Attach digital signature(s) to an image
  test        Run the user-defined tests within a container
  verify      Verify cryptographic signatures attached to an image
  version     Show the version for Singularity

Examples:
  $ singularity help <command> [<subcommand>]
  $ singularity help build
  $ singularity help instance start


For additional help or support, please visit https://www.sylabs.io/docs/
$ man singularity shell
singularity(1)                                                                                               singularity(1)

NAME
       singularity-shell - Run a shell within a container

SYNOPSIS
       singularity shell [shell options...]

DESCRIPTION
       singularity shell supports the following formats:

       *.sif               Singularity Image Format (SIF). Native to Singularity 3.0+

       *.sqsh              SquashFS format.  Native to Singularity 2.4+

       *.img               ext3 format. Native to Singularity versions < 2.4.

       directory/          sandbox format. Directory containing a valid root file
                             system and optionally Singularity meta-data.

       instance://*        A local running instance of a container. (See the instance
                             command group.)

       library://*         A SIF container hosted on a Library
                             (default https://cloud.sylabs.io/library)

       docker://*          A Docker/OCI container hosted on Docker Hub or another
                             OCI registry.

       shub://*            A container hosted on Singularity Hub.

       oras://*            A SIF container hosted on an OCI registry that supports
                             the OCI Registry As Storage (ORAS) specification.

OPTIONS
       --add-caps=""      a comma separated capability list to add

       --allow-setuid[=false]      allow setuid binaries in container (root only)

       --app=""      set an application to run inside a container

       --apply-cgroups=""      apply cgroups from file for container processes (root only)

       -B,  --bind=[]       a  user-bind path specification.  spec has the format src[:dest[:opts]], where src and dest are
       outside and inside paths.  If dest is not given, it is set equal to src.  Mount options ('opts') may be specified as
       'ro'  (read-only)  or 'rw' (read/write, which is the default). Multiple bind paths can be given by a comma separated
       list.

       -e, --cleanenv[=false]      clean environment before running container

       -c, --contain[=false]      use minimal /dev and empty other directories (e.g. /tmp and  $HOME)  instead  of  sharing
       filesystems from your host

       -C, --containall[=false]      contain not only file systems, but also PID, IPC, and environment

       --disable-cache[=false]      dont use cache, and dont create cache

       --dns=""      list of DNS server separated by commas to add in resolv.conf

       --docker-login[=false]      login to a Docker Repository interactively

       --drop-caps=""      a comma separated capability list to drop

       --env=[]      pass environment variable to contained process

       --env-file=""      pass environment variables from file to contained process

       -f, --fakeroot[=false]      run container in new user namespace as uid 0

       --fusemount=[]      A FUSE filesystem mount specification of the form ': ' - where  is 'container' or 'host', speci‐
       fying where the mount will be performed ('container-daemon' or 'host-daemon' will run the  FUSE  process  detached).
       is  the path to the FUSE executable, plus options for the mount.  is the location in the container to which the FUSE
       mount will be attached. E.g. 'container:sshfs 10.0.0.1:/ /sshfs'. Implies --pid.

       -h, --help[=false]      help for shell

       -H, --home="/builddir"      a home directory specification.  spec can either be a src path or src:dest pair.  src is
       the  source  path  of the home directory outside the container and dest overrides the home directory within the con‐
       tainer.

       --hostname=""      set container hostname

       -i, --ipc[=false]      run container in a new IPC namespace

       --keep-privs[=false]      let root user keep privileges in container (root only)

       -n, --net[=false]      run container in a new network namespace (sets up a bridge network interface by default)

       --network="bridge"      specify desired network type separated by commas, each network will  bring  up  a  dedicated
       interface inside container

       --network-args=[]      specify network arguments to pass to CNI plugins

       --no-home[=false]      do NOT mount users home directory if /home is not the current working directory

       --no-init[=false]      do NOT start shim process with --pid

       --no-mount=[]      disable one or more mount xxx options set in singularity.conf

       --no-privs[=false]      drop all privileges from root user in container)

       --no-umask[=false]      do not propagate umask to the container, set default 0022 umask

       --nohttps[=false]       do  NOT use HTTPS with the docker:// transport (useful for local docker registries without a
       certificate)

       --nonet[=false]      disable VM network handling

       --nv[=false]      enable experimental Nvidia support

       -o, --overlay=[]      use an overlayFS image for persistent data storage or as read-only layer of container

       --passphrase[=false]      prompt for an encryption passphrase

       --pem-path=""      enter an path to a PEM formated RSA key for an encrypted container

       -p, --pid[=false]      run container in a new PID namespace

       --pwd=""      initial working directory for payload process inside the container

       --rocm[=false]      enable experimental Rocm support

       -S, --scratch=[]      include a scratch directory within the container that is linked to a temporary dir (use -W  to
       force location)

       --security=[]      enable security features (SELinux, Apparmor, Seccomp)

       -s, --shell=""      path to program to use for interactive shell

       --syos[=false]      execute SyOS shell

       -u, --userns[=false]      run container in a new user namespace, allowing Singularity to run completely unprivileged
       on recent kernels. This disables some features of Singularity, for example it only works with sandbox images.

       --uts[=false]      run container in a new UTS namespace

       --vm[=false]      enable VM support

       --vm-cpu="1"      number of CPU cores to allocate to Virtual Machine (implies --vm)

       --vm-err[=false]      enable attaching stderr from VM

       --vm-ip="dhcp"      IP Address to assign for container usage. Defaults to DHCP within bridge network.
       
       --vm-ram="1024"      amount of RAM in MiB to allocate to Virtual Machine (implies --vm)

       -W, --workdir=""      working directory to be used for /tmp, /var/tmp and $HOME (if -c/--contain was also used)

       -w, --writable[=false]      by default all Singularity containers are available as read only. This option makes  the
       file system accessible as read/write.

       --writable-tmpfs[=false]       makes the file system accessible as read-write with non persistent data (with overlay
       support only)

EXAMPLE
                $ singularity shell /tmp/Debian.sif
                Singularity/Debian.sif> pwd
                /home/gmk/test
                Singularity/Debian.sif> exit

                $ singularity shell -C /tmp/Debian.sif
                Singularity/Debian.sif> pwd
                /home/gmk
                Singularity/Debian.sif> ls -l
                total 0
                Singularity/Debian.sif> exit

                $ sudo singularity shell -w /tmp/Debian.sif
                $ sudo singularity shell --writable /tmp/Debian.sif

                $ singularity shell instance://my_instance

                $ singularity shell instance://my_instance
                Singularity: Invoking an interactive shell within container...
                Singularity container: > ps -ef
                UID        PID  PPID  C STIME TTY          TIME CMD
                ubuntu       1     0  0 20:00 ?        00:00:00 /usr/local/bin/singularity/bin/sinit
                ubuntu       2     0  0 20:01 pts/8    00:00:00 /bin/bash --norc
                ubuntu       3     2  0 20:02 pts/8    00:00:00 ps -ef

SEE ALSO
       singularity(1)

HISTORY
       15-Jun-2021 Auto generated by spf13/cobra

Auto generated by spf13/cobra                             Jun 2021                                           singularity(1)

How do we find a container to use?

$ singularity search samtools 
Found 8 container images for amd64 matching "samtools":

	library://daanjg98/rnaseq/samtools:1.11

	library://icaoberg/default/samtools:v1.10,latest
		Samtools is a suite of programs for interacting with high-throughput sequencing data.
		Signed by: 5BB086918F6EAFF2EA6CAAE06DDC4BC448961610

	library://joelnulsen/default/samtools:v1

	library://kgillinder/analysis_pipelines/samtools:v1.1.12
		Signed by: b2396fa5a9d3d18ca20e82950e5ea802a07d87a3

	library://marialitovchenko/default/samtools:v.1.10

	library://summerwang/default/samtools:1.7

	library://vi.ya/rnaseq-dbs/samtools-v1.12:latest

	library://weizhu365/mocca-sv/samtools_1-9:1.0.0

Or search for the star application

$ singularity search star
Found 12 container images for amd64 matching "star":

	library://dtrudg-sylabs/default/testarch4179:latest

	library://dtrudg-sylabs/default/testarch:bob,latest

	library://jemten/mip_containers/star:2.7.3a

	library://khodeir/default/decstar:sha256.66d9dda1a075a011537d6e8dcac7589e8e9c73989a298b1fa0519ac4e7c7e796
		Signed by: f45f0aaa38f52f285151bcb71b7a20de7d68a80c

	library://marialitovchenko/default/sra_to_bam_star:v18jan2021

	library://marialitovchenko/default/star:v.2.7.1a

	library://vi.ya/rnaseq-dbs/star_2.7.7a:latest

	library://vi.ya/rnaseq-dbs/star_2.7.7a:sha256.bef07a9d979e6150fd6f4fdca557cb38b967773eb13fcc593bd2818029140869

	library://viya/rnaseq-dbs/star-2.7.7a:latest

	library://wchoston/project1/stars2csv_env:latest

	library://yh549848/rnaseqde/star:2.6.1d

	library://yh549848/rnaseqde/star:latest,2.7.8a

What repo are we searching? This is the generic Singularity basic cloud library repo where anyone can upload their containers found at https://cloud.sylabs.io/library. (browse to this site in your browser and show how the same containers are available via this site)

Pulling containers . We are going to be working with a humorous container today called lolcow to lay the foundation for using containers.

$ singularity search lolcow | grep demo (this is the container we’ll pull)
library://godlovedc/demo/lolcow:latest

Let’s pull the container down.

singularity pull library://godlovedc/demo/lolcow:latest  (disregard the WARNING message that appears)
INFO:    Downloading library image
89.2MiB / 89.2MiB [=======================================================================================] 100 % 58.6 MiB/s 0s
WARNING: integrity: signature not found for object group 1
WARNING: Skipping container verification

When we use the URI library:// we are specifying the default Singularity sylabs cloud library. Others are available which we’ll cover in just a moment

Run ls -l to show that lolcow_latest.sif was pulled or downloaded from the Singularity cloud repo.

$ ls -l
total 91384
-rwxrwxr-x. 1 labuser labuser 93574075 Jul 28 17:31 lolcow_latest.sif

The .sif extension stands for “singularity image format” and is the default image format singularity uses.

Notice how singularity images appear as single files unlike Docker images which have multiple layers

Run singularity pull docker://biocontainers/bamtools:v2.4.0_cv4 If you notice with this container we are pulling, Singularity is downloading multiple image source layers and converting them into a single .sif file. (this takes a minute or 2 to complete)

$ singularity pull docker://biocontainers/bamtools:v2.4.0_cv4
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 34667c7e4631 done  
Copying blob d18d76a881a4 done  
Copying blob 119c7358fbfc done  
Copying blob 2aaf13f3eff0 done  
Copying blob d344a036449c done  
Copying blob 77f405472a1e done  
Copying blob 07459e7bed97 done  
Copying blob 4b62c33c231b done  
Copying blob d504e55e444c done  
Copying blob ab19a5691df7 done  
Copying blob 8740c4399f9a done  
Copying blob 374d024f102a done  
Copying blob b45a352333c1 done  
Copying blob 1e8c2c56d6ad done  
Copying blob b4ebad9eaf90 done  
Copying blob 62db1a3bbafa done  
Copying config 10d539d117 done  
Writing manifest to image destination
Storing signatures
2021/07/28 17:45:33  info unpack layer: sha256:34667c7e4631207d64c99e798aafe8ecaedcbda89fb9166203525235cc4d72b9
2021/07/28 17:45:34  warn rootless{dev/agpgart} creating empty file in place of device 10:175
2021/07/28 17:45:34  warn rootless{dev/audio} creating empty file in place of device 14:4
2021/07/28 17:45:34  warn rootless{dev/audio1} creating empty file in place of device 14:20
2021/07/28 17:45:34  warn rootless{dev/audio2} creating empty file in place of device 14:36
2021/07/28 17:45:34  warn rootless{dev/audio3} creating empty file in place of device 14:52
2021/07/28 17:45:34  warn rootless{dev/audioctl} creating empty file in place of device 14:7
2021/07/28 17:45:34  warn rootless{dev/console} creating empty file in place of device 5:1
2021/07/28 17:45:34  warn rootless{dev/dsp} creating empty file in place of device 14:3
2021/07/28 17:45:34  warn rootless{dev/dsp1} creating empty file in place of device 14:19
2021/07/28 17:45:34  warn rootless{dev/dsp2} creating empty file in place of device 14:35
2021/07/28 17:45:34  warn rootless{dev/dsp3} creating empty file in place of device 14:51
2021/07/28 17:45:34  warn rootless{dev/full} creating empty file in place of device 1:7
2021/07/28 17:45:34  warn rootless{dev/kmem} creating empty file in place of device 1:2
2021/07/28 17:45:34  warn rootless{dev/loop0} creating empty file in place of device 7:0
2021/07/28 17:45:34  warn rootless{dev/loop1} creating empty file in place of device 7:1
2021/07/28 17:45:34  warn rootless{dev/loop2} creating empty file in place of device 7:2
2021/07/28 17:45:34  warn rootless{dev/loop3} creating empty file in place of device 7:3
2021/07/28 17:45:34  warn rootless{dev/loop4} creating empty file in place of device 7:4
2021/07/28 17:45:34  warn rootless{dev/loop5} creating empty file in place of device 7:5
2021/07/28 17:45:34  warn rootless{dev/loop6} creating empty file in place of device 7:6
2021/07/28 17:45:34  warn rootless{dev/loop7} creating empty file in place of device 7:7
2021/07/28 17:45:34  warn rootless{dev/mem} creating empty file in place of device 1:1
2021/07/28 17:45:34  warn rootless{dev/midi0} creating empty file in place of device 35:0
2021/07/28 17:45:34  warn rootless{dev/midi00} creating empty file in place of device 14:2
2021/07/28 17:45:34  warn rootless{dev/midi01} creating empty file in place of device 14:18
2021/07/28 17:45:34  warn rootless{dev/midi02} creating empty file in place of device 14:34
2021/07/28 17:45:34  warn rootless{dev/midi03} creating empty file in place of device 14:50
2021/07/28 17:45:34  warn rootless{dev/midi1} creating empty file in place of device 35:1
2021/07/28 17:45:34  warn rootless{dev/midi2} creating empty file in place of device 35:2
2021/07/28 17:45:34  warn rootless{dev/midi3} creating empty file in place of device 35:3
2021/07/28 17:45:34  warn rootless{dev/mixer} creating empty file in place of device 14:0
2021/07/28 17:45:34  warn rootless{dev/mixer1} creating empty file in place of device 14:16
2021/07/28 17:45:34  warn rootless{dev/mixer2} creating empty file in place of device 14:32
2021/07/28 17:45:34  warn rootless{dev/mixer3} creating empty file in place of device 14:48
2021/07/28 17:45:34  warn rootless{dev/mpu401data} creating empty file in place of device 31:0
2021/07/28 17:45:34  warn rootless{dev/mpu401stat} creating empty file in place of device 31:1
2021/07/28 17:45:34  warn rootless{dev/null} creating empty file in place of device 1:3
2021/07/28 17:45:34  warn rootless{dev/port} creating empty file in place of device 1:4
2021/07/28 17:45:34  warn rootless{dev/ram0} creating empty file in place of device 1:0
2021/07/28 17:45:34  warn rootless{dev/ram1} creating empty file in place of device 1:1
2021/07/28 17:45:34  warn rootless{dev/ram10} creating empty file in place of device 1:10
2021/07/28 17:45:34  warn rootless{dev/ram11} creating empty file in place of device 1:11
2021/07/28 17:45:34  warn rootless{dev/ram12} creating empty file in place of device 1:12
2021/07/28 17:45:34  warn rootless{dev/ram13} creating empty file in place of device 1:13
2021/07/28 17:45:34  warn rootless{dev/ram14} creating empty file in place of device 1:14
2021/07/28 17:45:34  warn rootless{dev/ram15} creating empty file in place of device 1:15
2021/07/28 17:45:34  warn rootless{dev/ram16} creating empty file in place of device 1:16
2021/07/28 17:45:34  warn rootless{dev/ram2} creating empty file in place of device 1:2
2021/07/28 17:45:34  warn rootless{dev/ram3} creating empty file in place of device 1:3
2021/07/28 17:45:34  warn rootless{dev/ram4} creating empty file in place of device 1:4
2021/07/28 17:45:34  warn rootless{dev/ram5} creating empty file in place of device 1:5
2021/07/28 17:45:34  warn rootless{dev/ram6} creating empty file in place of device 1:6
2021/07/28 17:45:34  warn rootless{dev/ram7} creating empty file in place of device 1:7
2021/07/28 17:45:34  warn rootless{dev/ram8} creating empty file in place of device 1:8
2021/07/28 17:45:34  warn rootless{dev/ram9} creating empty file in place of device 1:9
2021/07/28 17:45:34  warn rootless{dev/random} creating empty file in place of device 1:8
2021/07/28 17:45:34  warn rootless{dev/rmidi0} creating empty file in place of device 35:64
2021/07/28 17:45:34  warn rootless{dev/rmidi1} creating empty file in place of device 35:65
2021/07/28 17:45:34  warn rootless{dev/rmidi2} creating empty file in place of device 35:66
2021/07/28 17:45:34  warn rootless{dev/rmidi3} creating empty file in place of device 35:67
2021/07/28 17:45:34  warn rootless{dev/sequencer} creating empty file in place of device 14:1
2021/07/28 17:45:34  warn rootless{dev/smpte0} creating empty file in place of device 35:128
2021/07/28 17:45:34  warn rootless{dev/smpte1} creating empty file in place of device 35:129
2021/07/28 17:45:34  warn rootless{dev/smpte2} creating empty file in place of device 35:130
2021/07/28 17:45:34  warn rootless{dev/smpte3} creating empty file in place of device 35:131
2021/07/28 17:45:34  warn rootless{dev/sndstat} creating empty file in place of device 14:6
2021/07/28 17:45:34  warn rootless{dev/tty} creating empty file in place of device 5:0
2021/07/28 17:45:34  warn rootless{dev/tty0} creating empty file in place of device 4:0
2021/07/28 17:45:34  warn rootless{dev/tty1} creating empty file in place of device 4:1
2021/07/28 17:45:34  warn rootless{dev/tty2} creating empty file in place of device 4:2
2021/07/28 17:45:34  warn rootless{dev/tty3} creating empty file in place of device 4:3
2021/07/28 17:45:34  warn rootless{dev/tty4} creating empty file in place of device 4:4
2021/07/28 17:45:34  warn rootless{dev/tty5} creating empty file in place of device 4:5
2021/07/28 17:45:34  warn rootless{dev/tty6} creating empty file in place of device 4:6
2021/07/28 17:45:34  warn rootless{dev/tty7} creating empty file in place of device 4:7
2021/07/28 17:45:34  warn rootless{dev/tty8} creating empty file in place of device 4:8
2021/07/28 17:45:34  warn rootless{dev/tty9} creating empty file in place of device 4:9
2021/07/28 17:45:34  warn rootless{dev/urandom} creating empty file in place of device 1:9
2021/07/28 17:45:34  warn rootless{dev/zero} creating empty file in place of device 1:5
2021/07/28 17:45:38  info unpack layer: sha256:d18d76a881a47e51f4210b97ebeda458767aa6a493b244b4b40bfe0b1ddd2c42
2021/07/28 17:45:38  info unpack layer: sha256:119c7358fbfc2897ed63529451df83614c694a8abbd9e960045c1b0b2dc8a4a1
2021/07/28 17:45:38  info unpack layer: sha256:2aaf13f3eff07aa25f73813096bd588e6408b514288651402aa3d0357509be7a
2021/07/28 17:45:38  info unpack layer: sha256:d344a036449c9288b1818105ae4d2abce625657573f79aa9ded51403e1a51224
2021/07/28 17:45:38  info unpack layer: sha256:77f405472a1eed27adbd29123f96da0632455b6559d2c48323f6120712d77d0d
2021/07/28 17:45:40  warn rootless{usr/bin/systemd-detect-virt} ignoring (usually) harmless EPERM on setxattr "security.capability"
2021/07/28 17:46:03  info unpack layer: sha256:07459e7bed97fc825203f1587163b670b47b9fd23f4f5946559eff752f13d184
2021/07/28 17:46:08  info unpack layer: sha256:4b62c33c231b3f2bce5d7e298d3ee0d7cbb69eed00b0943a88331cce6d092615
2021/07/28 17:46:08  info unpack layer: sha256:d504e55e444cbfaeaada3d9c8d33779ee7ac3c77d45d2a4613ec9d7e9aed603a
2021/07/28 17:46:08  info unpack layer: sha256:ab19a5691df732652cb199a12c6b8a790e2b5ec24d0fff49ec029f22fa9b3b7b
2021/07/28 17:46:08  info unpack layer: sha256:8740c4399f9a911064779ce2c4142e75d8657bc41bda574f0c984a67f225a7c2
2021/07/28 17:46:25  info unpack layer: sha256:374d024f102a3cd206a88f95a6d8920dde8566d4a48ca797c030d64d96339f87
2021/07/28 17:46:25  info unpack layer: sha256:b45a352333c1891e7bdc39efa49e50f647fdc0c8d6e5548d582f5624dd27189d
2021/07/28 17:46:25  info unpack layer: sha256:1e8c2c56d6adede53b6b8075f05bc45ba83127c8b525eba55c12bd69c126f958
2021/07/28 17:46:25  info unpack layer: sha256:b4ebad9eaf90c8ad49afcfde76da1dbea1582f2ef574c2032f91df0779683f98
2021/07/28 17:46:25  warn rootless{opt/conda/bin/python} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:25  warn rootless{opt/conda/bin/python2} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:25  warn rootless{opt/conda/lib/libffi.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:25  warn rootless{opt/conda/lib/libffi.so.6} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:25  warn rootless{opt/conda/lib/libpython2.7.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:25  warn rootless{opt/conda/lib/libsqlite3.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:25  warn rootless{opt/conda/lib/libsqlite3.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:25  warn rootless{opt/conda/lib/pkgconfig/python.pc} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:25  warn rootless{opt/conda/lib/pkgconfig/python2.pc} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:30  warn rootless{opt/conda/pkgs/libffi-3.2.1-1/lib/libffi.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:30  warn rootless{opt/conda/pkgs/libffi-3.2.1-1/lib/libffi.so.6} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:30  warn rootless{opt/conda/pkgs/python-2.7.13-0/bin/python} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:30  warn rootless{opt/conda/pkgs/python-2.7.13-0/bin/python2} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:30  warn rootless{opt/conda/pkgs/python-2.7.13-0/lib/libpython2.7.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:30  warn rootless{opt/conda/pkgs/python-2.7.13-0/lib/pkgconfig/python.pc} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:30  warn rootless{opt/conda/pkgs/python-2.7.13-0/lib/pkgconfig/python2.pc} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:31  warn rootless{opt/conda/pkgs/python-2.7.13-0/share/man/man1/python.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:31  warn rootless{opt/conda/pkgs/python-2.7.13-0/share/man/man1/python2.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:31  warn rootless{opt/conda/pkgs/sqlite-3.13.0-0/lib/libsqlite3.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:31  warn rootless{opt/conda/pkgs/sqlite-3.13.0-0/lib/libsqlite3.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:31  warn rootless{opt/conda/share/man/man1/python.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:31  warn rootless{opt/conda/share/man/man1/python2.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:31  info unpack layer: sha256:62db1a3bbafaa3d4657da60a914659aa7bfc5a579314c7059f5ce57a1698de7e
2021/07/28 17:46:31  warn rootless{opt/conda/bin/bamtools} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:31  warn rootless{opt/conda/lib/libasan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:31  warn rootless{opt/conda/lib/libasan.so.5} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libatomic.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libatomic.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libbamtools.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libgfortran.so.3} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libgomp.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libgomp.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libitm.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libitm.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/liblsan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/liblsan.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libquadmath.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libquadmath.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libstdc++.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libstdc++.so.6} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libstdc++.so.6.0.21} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libtsan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libtsan.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libubsan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:32  warn rootless{opt/conda/lib/libubsan.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:34  warn rootless{opt/conda/pkgs/bamtools-2.4.0-3/bin/bamtools} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:34  warn rootless{opt/conda/pkgs/bamtools-2.4.0-3/lib/libbamtools.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-7.2.0-h69d50b8_2/lib/libgfortran.so.3} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-7.2.0-h69d50b8_2/lib/libstdc++.so.6.0.21} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libasan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libasan.so.5} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libatomic.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libatomic.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libgomp.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libgomp.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libitm.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libitm.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/liblsan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/liblsan.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libquadmath.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libquadmath.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libtsan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libtsan.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libubsan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/lib/libubsan.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libasan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libasan.so.5} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libasan.so.5.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libatomic.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libatomic.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libatomic.so.1.2.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libgomp.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libgomp.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libgomp.so.1.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libitm.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libitm.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libitm.so.1.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/liblsan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/liblsan.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/liblsan.so.0.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libquadmath.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libquadmath.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libquadmath.so.0.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libtsan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libtsan.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libtsan.so.0.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libubsan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libubsan.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libgcc-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libubsan.so.1.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libstdcxx-ng-8.2.0-hdf63c60_1/lib/libstdc++.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libstdcxx-ng-8.2.0-hdf63c60_1/lib/libstdc++.so.6} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libstdcxx-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libstdc++.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libstdcxx-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libstdc++.so.6} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/pkgs/libstdcxx-ng-8.2.0-hdf63c60_1/x86_64-conda_cos6-linux-gnu/sysroot/lib/libstdc++.so.6.0.25} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libasan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libasan.so.5} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libasan.so.5.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libatomic.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libatomic.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libatomic.so.1.2.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libgomp.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libgomp.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libgomp.so.1.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libitm.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libitm.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libitm.so.1.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/liblsan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/liblsan.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/liblsan.so.0.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libquadmath.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libquadmath.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libquadmath.so.0.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libstdc++.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libstdc++.so.6} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libstdc++.so.6.0.25} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libtsan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libtsan.so.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libtsan.so.0.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libubsan.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libubsan.so.1} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/07/28 17:46:36  warn rootless{opt/conda/x86_64-conda_cos6-linux-gnu/sysroot/lib/libubsan.so.1.0.0} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
INFO:    Creating SIF file...

Shelling into a container: Now, what do we do with these containers we pulled? First, one thing you can do with a container is interact with it

***SECURITY WARNING: We have been pulling containers someone else has built. How do we know these containers are safe to use and no malicious content? Answer: you don’t. You can assume based on the number of downloads (from Docker Hub) if a container is legitimate or not, but you can’t truly know at face value if malicious code is embedded. Fortunately, Singularity allows rootless containers to run so you can’t cause harm to the underlying system if you run as a non-admin user.

First, let’s see what OS our Lab VM is running: cat /etc/*release . It looks like we are running Centos 8.4.

$ cat /etc/*release
CentOS Linux release 8.4.2105
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
CentOS Linux release 8.4.2105
CentOS Linux release 8.4.2105

Now let’s open a shell into our container

$ singularity shell lolcow_latest.sif
Singularity>

If you notice our command prompt has changed and we are now in an interactive command prompt inside the container

Rerun cat /etc/*release. If you notice, we are actually in an Ubuntu 16 based container.

Singularity> cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS"
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Run ls and notice how our /home directory and PWD is available inside the container. The lines are blurred, even though we are in a different environment and OS, we still access some files outside the container and of course the container is sharing the host VM’s kernel/hardware.

Singularity> ls
bamtools_v2.4.0_cv4.sif  lolcow_latest.sif
Singularity> pwd
/home/labuser

Run whoami and notice I am the same “labuser” as I was outside the container. Remember, with singularity who I am outside the container is who I am inside the container. If I want to be root in the container, I must be root outside the container.

Singularity> whoami 
labuser

Run cowsay hello and notice what happens. Cowsay is a program inside the container. Run cowsay Welcome to this course!

Singularity> cowsay hello
 _______
< hello >
 -------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Run which cowsay and show how this is an application in the container and the path of it.

Singularity> which cowsay
/usr/games/cowsay

Run fortune then run fortune | cowsay. Then run fortune | cowsay | lolcat to show how you can pipe applications together

Singularity> fortune 
You're definitely on their list.  The question to ask next is what list it is.

Singularity> fortune | cowsay | lolcat
 ________________________________
/ All generalizations are false, \
| including this one.            |
|                                |
\ -- Mark Twain                  /
 --------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Type exit to leave the container environment

Executing Containerized Commands with Exec

$ singularity exec lolcow_latest.sif cowsay ‘How did you get out of the container?’
 _______________________________________
< How did you get out of the container? >
 ---------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Now we can bind a path from the host OS into our container. What does our host system have in the /data path and what does our container show?

# ls -al /data/
total 398544
drwxr-xr-x. 3 root root      4096 Jul 28 18:31 .
drwxr-xr-x. 3 root root        22 Jul  9 14:40 ..
-rwxr-xr-x. 1 root root 408084480 Jul 28 18:31 bamtools_v2.4.0_cv4.sif
-rw-r--r--. 1 root root       366 Jul 28 16:04 DATALOSS_WARNING_README.txt
drwx------. 2 root root     16384 Jul 28 16:04 lost+found
# singularity exec lolcow_latest.sif ls /data/
/bin/ls: cannot access '/data': No such file or directory

Using the –bind option allows us to make external paths available within our container.

# singularity exec --bind /data lolcow_latest.sif ls -al /data/
total 398544
drwxr-xr-x. 3 root    root         4096 Jul 28 18:31 .
drwxr-xr-x. 1 labuser labuser       100 Jul 28 18:36 ..
-rw-r--r--. 1 root    root          366 Jul 28 16:04 DATALOSS_WARNING_README.txt
-rwxr-xr-x. 1 root    root    408084480 Jul 28 18:31 bamtools_v2.4.0_cv4.sif
drwx------. 2 root    root        16384 Jul 28 16:04 lost+found

Key Points

  • First key point. Brief Answer to questions. (FIXME)


Scientific Containers

Overview

Teaching: 15 min
Exercises: 0 min
Questions
  • Getting started with a recipe file

Objectives
  • Understand the anatomy of a recipe file

FIXME

Scientific Containers

There are many advantages in using containers. In the realm of scientific computation containers facilitate scalable architectures that are flexible.

Containers support many scientific disciplines and have established repositories that can be utilized.

Docker/Singularity Hub

https://hub.docker.com/ https://singularityhub.github.io/

Vast collection of containers that will provide solutions across:

1) Biology 2) Chemistry 3) Math/Statistics 4) Programming Languages. …

Build a container to suit your needs. Include only the software and data that is needed. Componenets in the OS the system is built upon won’t change.

Install specific software versions. Version of releases are important for reproducibility of results. Stability of pipelines.

It provides integration to cloud pipelines. It is used as the core of the compuational backend of the pipeline, providning the operating system and specific alorithims to impliment. In either cloud or other HPC enviroements workflow managers are employed to describer and control entire workflows of data and computations

Nextflow example

The integration for Singularity follows the same execution model implemented for Docker. You won’t need to modify your Nextflow script in order to run it with Singularity. Simply specify the Singularity image file from where the containers are started by using the -with-singularity command line option. For example:

nextflow run <your script> -with-singularity [singularity image file]  

The container will scale to the enviroment it is running in. # of cpu(s) available can be used without additional parameters being passed. Same would be true of RAM. Storage will be the login space with additional storage made usable via bind.

Getting started with a application recipe file.


Bootstrap:docker
From:ubuntu

%labels

       MAINTAINER Richard Yanicky

%files

       # Files for Build Go Here
%post
       # DEBCONF Configuration
    export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true
    echo "tzdata tzdata/Areas select US" >> /opt/preseed.txt
    echo "tzdata tzdata/Zones/US select Chicago" >> /opt/preseed.txt
    debconf-set-selections /opt/preseed.txt

    rm /opt/preseed.txt

    # Update System

       apt update && apt upgrade -y

    apt install -y wget make locales build-essential bzip2 libncurses5-dev libbz2-dev liblzma-dev zlib1g-dev

    locale-gen en_US.UTF-8

   cd /opt

    wget https://github.com/samtools/samtools/releases/download/1.12/samtools-1.12.tar.bz2

    bzip2 -d samtools-1.12.tar.bz2

    tar -xf samtools-1.12.tar

    cd samtools-1.12

    ./configure --prefix=/opt/samtools

    make

    make install

    cd /opt

    rm samtools-1.12.tar

    rm -rf samtools-1.12
%runscript

    /opt/samtools/bin/samtools

    echo "/opt/samtools/bin/samtools/"

Key Points

  • First key point. Brief Answer to questions. (FIXME)


Break

Overview

Teaching: 10 min
Exercises: 0 min
Questions
Objectives

FIXME

Break

Key Points


Building and Modifying Containers

Overview

Teaching: 35 min
Exercises: 0 min
Questions
  • Building Containers

Objectives
  • Build a Container

FIXME

Building and Modifying Containers

The following is an example of a singularity recipe file. I will install the base OS and then configure the tools it needs to install applications.

There are sections in the recipe file to control the build process.

The first lines define the OS its version and packages. In the code here its docker layers for ubuntu

There are no files included with this recipe.

The post section is where applications are configured and loaded.

The application we will start with here will be Samtools.

Bootstrap:docker
From:ubuntu
%labels
       MAINTAINER Richard Yanicky
%files
       # Files for Build Go Here
%post
       # DEBCONF Configuration
    export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true
    echo "tzdata tzdata/Areas select US" >> /opt/preseed.txt
    echo "tzdata tzdata/Zones/US select Chicago" >> /opt/preseed.txt
    debconf-set-selections /opt/preseed.txt
    rm /opt/preseed.txt
    # Update System
       apt update && apt upgrade -y
    apt install -y wget make locales build-essential bzip2 libncurses5-dev libbz2-dev liblzma-dev zlib1g-dev
    locale-gen en_US.UTF-8
   cd /opt
    wget https://github.com/samtools/samtools/releases/download/1.12/samtools-1.12.tar.bz2
    bzip2 -d samtools-1.12.tar.bz2
    tar -xf samtools-1.12.tar
    cd samtools-1.12
    ./configure --prefix=/opt/samtools
    make
    make install
    cd /opt
    rm samtools-1.12.tar
    rm -rf samtools-1.12
%runscript
    /opt/samtools/bin/samtools
    echo "/opt/samtools/bin/samtools/"

This will install the OS and samtools By default it will run samtools.

To modify the recipe just add the application install instructions.

In this case we are adding bcftools.

It should be added at the end of %post before the runscript section.

wget https://github.com/samtools/bcftools/releases/download/1.12/bcftools-1.12.tar.bz2
    bzip2 -d bcftools-1.12.tar.bz2
    tar -xf bcftools-1.12.tar
    cd bcftools-1.12
    ./configure --prefix=/opt/bcftools
    make
    make install
    cd /opt
    rm bcftools-1.12.tar
    rm -rf bcftools-1.12

The %runscript section will need to be modified. We now can run either samtools or bcftools and the container will need to allow us to pass the application name.

the following changes need to be made to the PATH and runscript.

The $@ allows us to pass the command to run into the container.

%environment
        export PATH=$PATH:/opt/samtools/bin:/opt/bcftools/bin

%runscript
    echo $PATH
    exec "$@"

singularity run samtoolsbcf.sif samtools

singularity shell samtoolsbcf.sif

Key Points

  • First key point. Brief Answer to questions. (FIXME)


Using Containers & Questions

Overview

Teaching: 35 min
Exercises: 0 min
Questions
  • Use a container

Objectives
  • Build a BLAST container

  • Use the built BLAST container

Using Containers & Questions

Building on our previous sections, in this unit, we’re going to build a container for BLAST and show how to use that container to construct a BLAST database and search sequences against that database.

Containerizing BLAST

BLAST stands for Basic Local Alignment Search Tool, and is a sophisticated software package for rapid searching of protein and nucleotide databases. BLAST was developed by Steven Altschul in 1989, and has continually been refined, updated, and modified throughout the years to meet the increasing needs of the scientific community.

To cite BLAST, please refer to the following: Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ. Basic local alignment search tool. Journal of Molecular Biology. Volume 215(3), pages 403-410. 1990. PMID: 2231712 DOI: 10.1016/S0022-2836(05)80360-2.

To start, let’s create an empty file to use as our recipe file.

$ touch Singularity.BLAST

The touch command allows modification of file timestamps, or in the case of this usage, where the file does not already exist, creates an empty file.

Now we’ll use nano to build out our recipe file.

$ nano Singularity.BLAST

This should open the basic nano text editor for you to access through your terminal.

Let’s type the following into our Singularity.BLAST file:

Bootstrap:docker
From:ubuntu

%labels
        MAINTAINER Shane Sanders

%post
        apt update && apt upgrade -y
        apt install -y wget gzip zip unzip ncbi-blast+ locales
        LANG=C perl -e exit
        locale-gen en_US.UTF-8

%runscript
        echo "Hello from BLAST!"

Let’s hit CTRL-O to save our modifications and then CTRL-X to exit the nano editor.

Ok, now to build the container:

$ sudo singularity build BLAST.sif Singularity.BLAST

Time it

This should take ~3 minutes to build the container and install BLAST. If you’d like to time it on your system, you can prefix the command with the time command, as follows:

$ time sudo singularity build BLAST.sif Singularity.BLAST

Result

And if you time this, at the end of the process, you should see output like:

real    1m25.582s
user    1m22.355s
sys     0m6.772s

Ok, let’s test that our container is built properly.

$ ./BLAST.sif
Hello from BLAST!

Ok, now we can go into the container’s environment to verify things using the singularity shell command.

$ singularity shell BLAST.sif

Notice the change in prompt from “$” to ”Singularity>”, this is because we are inside the container.

Singularity> blastp -h
USAGE
  blastp [-h] [-help] [-import_search_strategy filename]
    [-export_search_strategy filename] [-task task_name] [-db database_name]
    [-dbsize num_letters] [-gilist filename] [-seqidlist filename]
    [-negative_gilist filename] [-negative_seqidlist filename]
    [-taxids taxids] [-negative_taxids taxids] [-taxidlist filename]
    [-negative_taxidlist filename] [-ipglist filename]
    [-negative_ipglist filename] [-entrez_query entrez_query]
    [-db_soft_mask filtering_algorithm] [-db_hard_mask filtering_algorithm]
    [-subject subject_input_file] [-subject_loc range] [-query input_file]
    [-out output_file] [-evalue evalue] [-word_size int_value]
    [-gapopen open_penalty] [-gapextend extend_penalty]
    [-qcov_hsp_perc float_value] [-max_hsps int_value]
    [-xdrop_ungap float_value] [-xdrop_gap float_value]
    [-xdrop_gap_final float_value] [-searchsp int_value] [-seg SEG_options]
    [-soft_masking soft_masking] [-matrix matrix_name]
    [-threshold float_value] [-culling_limit int_value]
    [-best_hit_overhang float_value] [-best_hit_score_edge float_value]
    [-subject_besthit] [-window_size int_value] [-lcase_masking]
    [-query_loc range] [-parse_deflines] [-outfmt format] [-show_gis]
    [-num_descriptions int_value] [-num_alignments int_value]
    [-line_length line_length] [-html] [-sorthits sort_hits]
    [-sorthsps sort_hsps] [-max_target_seqs num_sequences]
    [-num_threads int_value] [-ungapped] [-remote] [-comp_based_stats compo]
    [-use_sw_tback] [-version]

DESCRIPTION
   Protein-Protein BLAST 2.9.0+

Use '-help' to print detailed descriptions of command line arguments

The blastp command is for Protein BLASTs, where a protein sequence is searched against a protein database.

Let’s exit the container environment.

Singularity> exit

Now let’s try the containerized command from the server’s environment.

$ singularity exec BLAST.sif blastp -h
USAGE
  blastp [-h] [-help] [-import_search_strategy filename]
    [-export_search_strategy filename] [-task task_name] [-db database_name]
    [-dbsize num_letters] [-gilist filename] [-seqidlist filename]
    [-negative_gilist filename] [-negative_seqidlist filename]
    [-taxids taxids] [-negative_taxids taxids] [-taxidlist filename]
    [-negative_taxidlist filename] [-ipglist filename]
    [-negative_ipglist filename] [-entrez_query entrez_query]
    [-db_soft_mask filtering_algorithm] [-db_hard_mask filtering_algorithm]
    [-subject subject_input_file] [-subject_loc range] [-query input_file]
    [-out output_file] [-evalue evalue] [-word_size int_value]
    [-gapopen open_penalty] [-gapextend extend_penalty]
    [-qcov_hsp_perc float_value] [-max_hsps int_value]
    [-xdrop_ungap float_value] [-xdrop_gap float_value]
    [-xdrop_gap_final float_value] [-searchsp int_value] [-seg SEG_options]
    [-soft_masking soft_masking] [-matrix matrix_name]
    [-threshold float_value] [-culling_limit int_value]
    [-best_hit_overhang float_value] [-best_hit_score_edge float_value]
    [-subject_besthit] [-window_size int_value] [-lcase_masking]
    [-query_loc range] [-parse_deflines] [-outfmt format] [-show_gis]
    [-num_descriptions int_value] [-num_alignments int_value]
    [-line_length line_length] [-html] [-sorthits sort_hits]
    [-sorthsps sort_hsps] [-max_target_seqs num_sequences]
    [-num_threads int_value] [-ungapped] [-remote] [-comp_based_stats compo]
    [-use_sw_tback] [-version]

DESCRIPTION
   Protein-Protein BLAST 2.9.0+

Use '-help' to print detailed descriptions of command line arguments

Same output, so now let’s put our new BLAST container to use!

Acquiring Protein Data

To start, we are going to need some data to serve as our database to search against. For this exercise, we will use C. elegans proteome. Let’s download and uncompress this file.

$ wget https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/002/985/GCF_000002985.6_WBcel235/GCF_000002985.6_WBcel235_protein.faa.gz

This shouldn’t take long, and produces the following output:

Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 165.112.9.229, 130.14.250.13, 2607:f220:41f:250::228, ...
Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|165.112.9.229|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6989933 (6.7M) [application/x-gzip]
Saving to: ‘GCF_000002985.6_WBcel235_protein.faa.gz’

100%[=======================================================================>] 6,989,933   41.1MB/s   in 0.2s

2021-12-02 19:13:56 (41.1 MB/s) - ‘GCF_000002985.6_WBcel235_protein.faa.gz’ saved [6989933/6989933]

Now we will unzip the data file.

$ gunzip GCF_000002985.6_WBcel235_protein.faa.gz

Now we will convert the protein FASTA file we downloaded into a BLAST database to search against.

$ time singularity exec BLAST.sif makeblastdb -in GCF_000002985.6_WBcel235_protein.faa -dbtype prot -out c_elegans

This gives us the following output:


Building a new DB, current time: 12/02/2021 19:14:39
New DB name:   /home/student/c_elegans
New DB title:  GCF_000002985.6_WBcel235_protein.faa
Sequence type: Protein
Keep MBits: T
Maximum file size: 1000000000B
Adding sequences from FASTA; added 28350 sequences in 0.727009 seconds.

real    0m1.248s
user    0m0.828s
sys     0m0.421s

Now we need some sequences of interest to search against the RefSeq database we just constructed. Let’s download all the RefSeq proteins for Human Chromosome 1:

$ wget ftp://ftp.ncbi.nih.gov/refseq/H_sapiens/mRNA_Prot/human.1.protein.faa.gz
--2021-12-02 19:15:08--  ftp://ftp.ncbi.nih.gov/refseq/H_sapiens/mRNA_Prot/human.1.protein.faa.gz
           => ‘human.1.protein.faa.gz’
Resolving ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)... 130.14.250.10, 130.14.250.11, 2607:f220:41f:250::230, ...
Connecting to ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)|130.14.250.10|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /refseq/H_sapiens/mRNA_Prot ... done.
==> SIZE human.1.protein.faa.gz ... 1836521
==> PASV ... done.    ==> RETR human.1.protein.faa.gz ... done.
Length: 1836521 (1.8M) (unauthoritative)

100%[=======================================================================>] 1,836,521   8.92MB/s   in 0.2s

2021-12-02 19:15:08 (8.92 MB/s) - ‘human.1.protein.faa.gz’ saved [1836521]
$ gunzip human.1.protein.faa.gz

We’ve downloaded a multi-line FASTA file, but for ease of use, we will now convert this into a single-fasta.

Convert multi-line FASTA to single-line FASTA

$ sed -e 's/\(^>.*$\)/#\1#/' human.1.protein.faa | tr -d "\r" | tr -d "\n" | sed -e 's/$/#/' | tr "#" "\n" | sed -e '/^$/d' > human.1.protein.faa.cleaned.fasta

Now, we’ll BLAST the proteins of Human chromosome 1 against the c_elegans blast database. Normally, you would want to run your entire query sequence against the database and interpret results, but because these cloud systems we are connected to are small, in the interest of time we are going to reduce our number of query sequences.

head -n 20 human.1.protein.faa.cleaned.fasta > search_query.fasta

This pulls the first 10 protein sequences listed in the human chromosome 1 file into a new file, search_query.fasta. So instead of searching all 8,067 sequences, we will only search 10 (0.1%) against the c_elegans database we’ve constructed.

Here We will run blast program using our input and constructed blast database.

$ time singularity exec BLAST.sif blastp -num_threads 2 -db c_elegans -query search_query.fasta -outfmt 6 -out BLASTP_Results.txt -max_target_seqs 1

This gives us the following output:

Warning: [blastp] Examining 5 or more matches is recommended

real    0m5.957s
user    0m10.364s
sys     0m0.500s

Checking the output file:

$ head BLASTP_Results.txt

Gives the following:

NP_001355814.1  NP_001255859.1  41.697  542     273     10      122     646     77      592     1.46e-58        208
NP_001355814.1  NP_001255859.1  41.199  517     259     9       125     626     154     640     2.69e-57        205
NP_001355814.1  NP_001255859.1  38.609  575     266     9       116     646     109     640     3.11e-54        196
NP_001355814.1  NP_001255859.1  39.962  533     256     10      112     618     156     650     8.53e-53        192
NP_001355814.1  NP_001255859.1  40.393  458     211     9       109     565     243     639     1.33e-49        183
NP_001355814.1  NP_001255859.1  38.647  207     105     5       108     314     455     639     2.77e-15        79.3
NP_001355815.1  NP_001255859.1  40.937  491     220     10      1       472     153     592     4.73e-56        197
NP_001355815.1  NP_001255859.1  39.038  520     232     13      2       472     80      563     1.16e-48        177
NP_001355815.1  NP_001255859.1  42.958  426     196     9       76      459     75      495     9.38e-47        171
NP_001355815.1  NP_001255859.1  40.086  464     218     11      2       444     220     644     9.89e-44        163

Any Questions?

Please ask the presenters any questions you may have!

Key Points

  • ‘singularity shell ' provides a prompt from within the container.

  • ‘singularity exec ' runs program from within the container.