Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book

Episode #249: All of Linux as a Python API

Published Thu, Sep 9, 2021, recorded Thu, Sep 9, 2021.



Watch the live stream:

About the show

Sponsored by us:

Special guest: Erik Christiansen

Michael #1: Fickling

  • via Oli
  • A Python pickling decompiler and static analyzer
  • Pickled ML models are becoming the data exchange and workflow of ML
  • Analyses pickle files for security risks - It can also remove or insert [malicious] code into pickle files...
  • Created by a security firm, it can be a useful defensive or offensive tool.
  • Perhaps it is time to screen all pickles?
    >>> import ast
    >>> import pickle
    >>> from fickling.pickle import Pickled
    >>> print(ast.dump(Pickled.load(pickle.dumps([1, 2, 3, 4])).ast, indent=4))
    Module(
        body=[
            Assign(
                targets=[
                    Name(id='result', ctx=Store())],
                value=List(
                    elts=[
                        Constant(value=1),
                        Constant(value=2),
                        Constant(value=3),
                        Constant(value=4)],
                    ctx=Load()))])
  • You can test for common patterns of malicious pickle files with the --check-safety option
  • You can also safely trace the execution of the Pickle virtual machine without exercising any malicious code with the --trace option.
  • Finally, you can inject arbitrary Python code that will be run on unpickling into an existing pickle file with the --inject option.
  • See Risky Biz's episode for more details.

Brian #2: Python Project-Local Virtualenv Management

  • Hynek Schlawack
  • Only works on UNIX-like systems. MacOS, for example.
  • Instructions
    • Install direnv. (ex: brew install direnv)
    • Put this into a .envrc file in your project root:
    • layout python python3.9
  • Now
    • when you cd into that directory or a subdirectory, your virtual environment is loaded.
    • when you cd out of it, the venv is unloaded
  • Notes:
    • Michael covered direnv on Episode 185. But it wasn’t until Hynek spelled it out for me how to use it with venv that I understood the simplicity and power.
    • Not really faster than creating a venv, but when flipping between several projects, it’s way faster than deactivating/activating.
    • You can also set env variables per directory (kinda the point of direnv)

Erik #3: Testcontainers

“Python port for testcontainers-java that allows using docker containers for functional and integration testing. Testcontainers-python provides capabilities to spin up docker containers (such as a database, Selenium web browser, or any other container) for testing. “ (pypi description).

  • Provides cloud native services, many databases and the like (e.g. Google Cloud Pub/Sub, Kafka..)
  • Originally a java project, still a way to go for us python programmers to implement all services
  • Provides an example for use in CI/CD by leveraging Docker in Docker
    import sqlalchemy
    from testcontainers.mysql import MySqlContainer

    with MySqlContainer('mysql:5.7.17') as mysql:
        engine = sqlalchemy.create_engine(mysql.get_connection_url())
        version, = engine.execute("select version()").fetchone()
        print(version)  # 5.7.17

Michael #4: jc

  • via Garett
  • CLI tool and python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts.
  • Run it as COMMAND ARGS | jc --COMMAND
  • Commands include: systemctl, passwd, ls, jobs, hosts, du, and cksum.

Brian #5: What is Python's Ellipsis Object?

  • Florian Dahlitz
  • Ellipsis or is a constant defined in Python
    • “Ellipsis: The same as the ellipsis literal “...”. Special value used mostly in conjunction with extended slicing syntax for user-defined container data types.”
  • Can be used in type hinting
    • Func returns two int tuple
    def return_tuple() -> tuple[int, int]:
    pass
  • Func returns one or more integer:
    def return_tuple() -> tuple[int, ...]:
        pass
  • Replacement for pass:
    def my_function():
        ...

Erik #6: PyTorch Forecasting PyTorch Forecasting aims to ease state-of-the-art timeseries forecasting with neural networks for both real-world cases and research alike. The goal is to provide a high-level API with maximum flexibility for professionals and reasonable defaults for beginners.

  • basically tries to achieve for time series what fast.ai has achieved for computer vision and natural language processing
  • The package is built on PyTorch Lightning to allow training on CPUs, single and multiple GPUs out-of-the-box.
  • Implements of Temporal Fusion Transformers
    • interpretable - can calculate feature importance
  • Hyperparameter tuning with optuna

Extras

Brian

Michael

Joke: 200 == 400