Episode #249: All of Linux as a Python API
Watch the live stream:
About the show
Sponsored by us:
Special guest: Erik Christiansen
Michael #1: Fickling
- via Oli
- A Python pickling decompiler and static analyzer
- Pickled ML models are becoming the data exchange and workflow of ML
- Analyses pickle files for security risks - It can also remove or insert [malicious] code into pickle files...
- Created by a security firm, it can be a useful defensive or offensive tool.
- Perhaps it is time to screen all pickles?
>>> import ast >>> import pickle >>> from fickling.pickle import Pickled >>> print(ast.dump(Pickled.load(pickle.dumps([1, 2, 3, 4])).ast, indent=4)) Module( body=[ Assign( targets=[ Name(id='result', ctx=Store())], value=List( elts=[ Constant(value=1), Constant(value=2), Constant(value=3), Constant(value=4)], ctx=Load()))])
- You can test for common patterns of malicious pickle files with the
- You can also safely trace the execution of the Pickle virtual machine without exercising any malicious code with the
- Finally, you can inject arbitrary Python code that will be run on unpickling into an existing pickle file with the
- See Risky Biz's episode for more details.
- Hynek Schlawack
- Only works on UNIX-like systems. MacOS, for example.
- Install direnv. (ex: brew install direnv)
- Put this into a
.envrcfile in your project root:
layout python python3.9
- when you
cdinto that directory or a subdirectory, your virtual environment is loaded.
- when you cd out of it, the venv is unloaded
- when you
- Michael covered direnv on Episode 185. But it wasn’t until Hynek spelled it out for me how to use it with venv that I understood the simplicity and power.
- Not really faster than creating a venv, but when flipping between several projects, it’s way faster than deactivating/activating.
- You can also set env variables per directory (kinda the point of direnv)
Erik #3: Testcontainers
“Python port for testcontainers-java that allows using docker containers for functional and integration testing. Testcontainers-python provides capabilities to spin up docker containers (such as a database, Selenium web browser, or any other container) for testing. “ (pypi description).
- Provides cloud native services, many databases and the like (e.g. Google Cloud Pub/Sub, Kafka..)
- Originally a java project, still a way to go for us python programmers to implement all services
- Provides an example for use in CI/CD by leveraging Docker in Docker
import sqlalchemy from testcontainers.mysql import MySqlContainer with MySqlContainer('mysql:5.7.17') as mysql: engine = sqlalchemy.create_engine(mysql.get_connection_url()) version, = engine.execute("select version()").fetchone() print(version) # 5.7.17
Michael #4: jc
- via Garett
- CLI tool and python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts.
- Run it as
COMMAND ARGS | jc --COMMAND
- Commands include:
Brian #5: What is Python's Ellipsis Object?
- Florian Dahlitz
…is a constant defined in Python
- “Ellipsis: The same as the ellipsis literal “...”. Special value used mostly in conjunction with extended slicing syntax for user-defined container data types.”
- Can be used in type hinting
- Func returns two int tuple
def return_tuple() -> tuple[int, int]: pass
- Func returns one or more integer:
def return_tuple() -> tuple[int, ...]: pass
- Replacement for
def my_function(): ...
- Ellipsis in the wild, “if you want to implement a certain feature where you need a non-used literal, you can use the ellipsis object.”
Erik #6: PyTorch Forecasting PyTorch Forecasting aims to ease state-of-the-art timeseries forecasting with neural networks for both real-world cases and research alike. The goal is to provide a high-level API with maximum flexibility for professionals and reasonable defaults for beginners.
- basically tries to achieve for time series what fast.ai has achieved for computer vision and natural language processing
- The package is built on PyTorch Lightning to allow training on CPUs, single and multiple GPUs out-of-the-box.
- Implements of Temporal Fusion Transformers
- interpretable - can calculate feature importance
- Hyperparameter tuning with optuna
- Python 3.10rc2 available. 3.10 is about a month away
- GoAccess follow up
- Caffinate more - via Nathan Henrie: you mentioned the MacOS /usr/bin/caffeinate tool on "https://pythonbytes.fm/episodes/show/247/do-you-dare-to-press-.". Follow caffeinate with long-running command to keep awake until done (
caffeinate python -c 'import time; time.sleep(10)'), or
caffeinate -w "$PID"for an already running task.
- Python Keyboard (via Sean Tabor)
- Open source is booming (via Mark Little)
- FFMPEG.WASM ffmpeg.wasm is a pure WebAssembly via Jim Anderson
- Everything is fine: PyPI packages
- Python 3.10 RC 2 is out
Joke: 200 == 400