#249: All of Linux as a Python API

Watch the live stream:

Play on YouTube

Watch the live stream replay

About the show

Sponsored by us:

• Check out the courses over at Talk Python
• And Brian’s book too!

Special guest: Erik Christiansen

Michael #1: Fickling

• via Oli
• A Python pickling decompiler and static analyzer
• Pickled ML models are becoming the data exchange and workflow of ML
• Analyses pickle files for security risks - It can also remove or insert [malicious] code into pickle files...
• Created by a security firm, it can be a useful defensive or offensive tool.
• Perhaps it is time to screen all pickles?

      >>> import ast
      >>> import pickle
      >>> from fickling.pickle import Pickled
      >>> print(ast.dump(Pickled.load(pickle.dumps([1, 2, 3, 4])).ast, indent=4))
      Module(
          body=[
              Assign(
                  targets=[
                      Name(id='result', ctx=Store())],
                  value=List(
                      elts=[
                          Constant(value=1),
                          Constant(value=2),
                          Constant(value=3),
                          Constant(value=4)],
                      ctx=Load()))])
  

• You can test for common patterns of malicious pickle files with the --check-safety option
• You can also safely trace the execution of the Pickle virtual machine without exercising any malicious code with the --trace option.
• Finally, you can inject arbitrary Python code that will be run on unpickling into an existing pickle file with the --inject option.
• See Risky Biz's episode for more details.

Brian #2: Python Project-Local Virtualenv Management

• Hynek Schlawack
• Only works on UNIX-like systems. MacOS, for example.
• Instructions
  • Install direnv. (ex: brew install direnv)
  • Put this into a .envrc file in your project root:
  • layout python python3.9
• Now
  • when you cd into that directory or a subdirectory, your virtual environment is loaded.
  • when you cd out of it, the venv is unloaded
• Notes:
  • Michael covered direnv on Episode 185. But it wasn’t until Hynek spelled it out for me how to use it with venv that I understood the simplicity and power.
  • Not really faster than creating a venv, but when flipping between several projects, it’s way faster than deactivating/activating.
  • You can also set env variables per directory (kinda the point of direnv)

Erik #3: Testcontainers

“Python port for testcontainers-java that allows using docker containers for functional and integration testing. Testcontainers-python provides capabilities to spin up docker containers (such as a database, Selenium web browser, or any other container) for testing. “ (pypi description).

• Provides cloud native services, many databases and the like (e.g. Google Cloud Pub/Sub, Kafka..)
• Originally a java project, still a way to go for us python programmers to implement all services
• Provides an example for use in CI/CD by leveraging Docker in Docker

      import sqlalchemy
      from testcontainers.mysql import MySqlContainer
  
      with MySqlContainer('mysql:5.7.17') as mysql:
          engine = sqlalchemy.create_engine(mysql.get_connection_url())
          version, = engine.execute("select version()").fetchone()
          print(version)  # 5.7.17
  

Michael #4: jc

• via Garett
• CLI tool and python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts.
• Run it as COMMAND ARGS | jc --COMMAND
• Commands include: systemctl, passwd, ls, jobs, hosts, du, and cksum.

Brian #5: What is Python's Ellipsis Object?

• Florian Dahlitz
• Ellipsis or … is a constant defined in Python
  • “Ellipsis: The same as the ellipsis literal “...”. Special value used mostly in conjunction with extended slicing syntax for user-defined container data types.”
• Can be used in type hinting
  • Func returns two int tuple

        def return_tuple() -> tuple[int, int]:
        pass
    

• Func returns one or more integer:

      def return_tuple() -> tuple[int, ...]:
          pass
  

• Replacement for pass:

      def my_function():
          ...
  

• Ellipsis in the wild, “if you want to implement a certain feature where you need a non-used literal, you can use the ellipsis object.”
  • FastAPI : Ellipsis used to make parameters required
  • Typer: Same

Erik #6: PyTorch Forecasting PyTorch Forecasting aims to ease state-of-the-art timeseries forecasting with neural networks for both real-world cases and research alike. The goal is to provide a high-level API with maximum flexibility for professionals and reasonable defaults for beginners.

• basically tries to achieve for time series what fast.ai has achieved for computer vision and natural language processing
• The package is built on PyTorch Lightning to allow training on CPUs, single and multiple GPUs out-of-the-box.
• Implements of Temporal Fusion Transformers
  • interpretable - can calculate feature importance
• Hyperparameter tuning with optuna

Extras

Brian

• Python 3.10rc2 available. 3.10 is about a month away

Michael

• GoAccess follow up
• Caffinate more - via Nathan Henrie: you mentioned the MacOS /usr/bin/caffeinate tool on "https://pythonbytes.fm/episodes/show/247/do-you-dare-to-press-.". Follow caffeinate with long-running command to keep awake until done (caffeinate python -c 'import time; time.sleep(10)'), or caffeinate -w "$PID" for an already running task.
• Python Keyboard (via Sean Tabor)
• Open source is booming (via Mark Little)
• FFMPEG.WASM ffmpeg.wasm is a pure WebAssembly via Jim Anderson
• Everything is fine: PyPI packages
• Python 3.10 RC 2 is out

Joke: 200 == 400