Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book

#208: Dependencies out of control? Just pip chill.

Published Thu, Nov 19, 2020, recorded Wed, Nov 11, 2020

Sponsored by us! Support our work through:

Brian #1: pip-chill - Make requirements with only the packages you need

  • Ricardo Bánffy
  • Like pip freeze but lists only the packages that are not dependencies of installed packages.
  • Will be great for creating requirements.txt files that look like the ones you would write by hand.
  • I wish it had an option to not list itself, but pip-chill | grep -v pip-chill works.

  • What do I have installed?

        (foo) $ pip freeze
        appdirs==1.4.4
        black==20.8b1
        click==7.1.2
        mypy-extensions==0.4.3
        ...
    
  • No really, what did I myself install?

        (foo) $ pip-chill 
        black==20.8b1
        pip-chill==1.0.0
    
  • Without versions?

        (foo) $ pip-chill --no-version
        black
        pip-chill
    
  • What did those things install as dependencies?

        (foo) $ pip-chill -v --no-version
        black
        pip-chill
        # appdirs # Installed as dependency for black
        # click # Installed as dependency for black
        ...
    

Michael #2: Windows update broke NumPy

  • Sent in by Daniel Mulkey
  • A recent Windows update broke some behavior that I think OpenBLAS (used by NumPy) relied on.
  • There's a Developer Community thread here.
  • I am a NumPy developer. We have been trying to track down a strange issue where after updating to windows 10 2004, suddenly code that worked no longer works. Here is the NumPy issue and here is the corresponding issue in OpenBLAS. The problem can be summarized: when calling fmod, something is changed so that much later calling an OpenBLAS assembly routine fails. The only difference I can see in the registers that visual studio exposes is that after the call to fmod, register ST(0) is set to NAN.
  • Steve Dower and other Microsoft people have commented.
  • The fix is slated to take until January 2021 to be released, though there are workarounds for some scenarios.
  • Matt P. posted a workaround:
    • For all those at home following along and looking for a quick fix, NumPy has released a bugfix 1.19.3 to work around this issue. The bugfix broke something else on Linux, so we had to revert the fix in release 1.19.4, but you can still install the 1.19.3 via
    • pip install numpy==1.19.3.
    • Note this is only works around the way this bug crashes NumPy (technically, in OpenBLAS which is shipped with NumPy), and may not fix all your problems related to this bug, Microsoft’s help is needed to do that.

Brian #3: Build Plugins with Pluggy

  • kracekumar
  • Blog post related to talks given at PyGotham and PyCon India
  • Pluggy is the plugin library used by pytest
  • Article
    • starts with a CLI application that has one output format.
    • Need is for more formats, implemented as plugins.
    • Quick look at pluggy architecture of host/caller/core system and plugin/hook.
    • Also plugin manager, hook specs, and hook implementations.
    • Walks through the changes to the application needed to support plugins.
  • I’ve been waiting for an article on pluggy, and this is nice.
  • But I admit I’m still a little lost. I guess I need to watch one of the presentations and try to build something with pluggy.

Michael #4: LINQ in Python

  • via Adam: I seem to recall that Michael had a C# background, so this might be of interest:
  • Bringing LINQ-like expressions to Python with linqit
  • Example:
        last_hot_pizza_slice = programmers.where(lambda e:e.experience > 15)
                              .except_for(elon_musk)
                              .of_type(Avi)
                              .take(3) # [[HTML_REMOVED], [HTML_REMOVED], [HTML_REMOVED]]
                              .select(lambda avi:avi.lunch) # [[HTML_REMOVED], [HTML_REMOVED], [HTML_REMOVED]]
                              .where(lambda p:p.is_hot() and p.origin != 'Pizza Hut').
                              .last() # [HTML_REMOVED]
                              .slices.last() # [HTML_REMOVED]
    
  • Also interesting asq: https://github.com/sixty-north/asq

Brian #5: Klio : a framework for processing audio files or any binary files, at large scale

  • Recently open sourced by Spotify
  • An article about it
  • Klio is based on Apache Beam and allows
    • integration with cloud processing engines
    • open graph of job dependencies
    • batch and streaming pipelines
  • goals:
    • large-file input/output
    • scalability, reproducibility, efficiency
    • closer collaboration between researchers and engineers
  • uses Python
  • Obviously useful for Spotify, but they are hoping it will help with other audio research and applications.

Michael #6: Collapsing code cells in Jupyter Notebooks

  • via Marco Gorelli
  • You mentioned in that episode that you'd like to have a way of collapsing code cells in Jupyter Notebooks so you can export them as reports - incidentally, I wrote a little blog post about how to do that - in case it's useful/of interest to you, here it is!
  • Basically get a static HTML file that is the static notebook output but can start with the code cells collapsed and can toggle their visibility.

Extras

Michael:

Joke:

By Richard Cairns

Q: Why did the data scientist get in trouble with Animal Welfare? A: She was caught trying to import pandas.

“10e engineeeeeeeeers are the future.” - detahq


Want to go deeper? Check our projects