Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book

#260: It's brutally simple: made just from pickle and zip

Published Tue, Nov 23, 2021, recorded Fri, Nov 19, 2021

Watch the live stream:

Watch this episode on YouTube
Play on YouTube
Watch the live stream replay

About the show

Sponsored by Shortcut - Get started at shortcut.com/pythonbytes

Special guest: Chris Patti

Brian #1: Using cog to update --help in a Markdown README file

  • Simon Willison
  • I’ve wanted to have a use case for Ned Batchelder’s cog

    • Cog is a utility that looks for specially blocks
          [[[cog
          some code
          ]]]
      
      and
          [[[end]]]
      
  • These block can be in comments, [HTML_REMOVED] for markdown.

  • When you run cog on a file, it runs the “some code” and puts the output after the middle ]]] and before the [[[end]]].
  • Simon has come up with an excellent use, running --help and capturing the output for a README.md file for a CLI project.
  • He even wrote a test, pytest of course, to check if the README.md needs updated.

Michael #2: An oral history of Bank Python

  • Bank Python implementations are effectively proprietary forks of the entire Python ecosystem which are in use at many (but not all) of the biggest investment banks.
  • The first thing to know about Minerva is that it is built on a global database of Python objects.
    • Barbara is a simple key value store with a hierarchical key space. It's brutally simple: made just from pickle and zip.
  • Applications also commonly store their internal state in Barbara - writing dataclasses straight in and out with only very simple locking and transactions (if any).
  • There is no filesystem available to Minerva scripts and the little bits of data that scripts pick up has to be put into Barbara.
  • Barbara also has some "overlay" features:

        # connect to multiple rings: keys are 'overlaid' in order of
        # the provided ring names
        db = barbara.open("middleoffice;ficc;default")
    
        # get /Etc/Something from the 'middleoffice' ring if it exists there,
        # otherwise try 'ficc' and finally the default ring
        some_obj = db["/Etc/Something"]
    
  • Lots of info about modeling with classes (instruments, books, etc)

  • If you understand excel you will be starting to recognize similarities.
  • In Excel, spreadsheets cells are also updated based on their dependencies, also as a directed acyclic graph. Dagger allows people to put their Excel-style modelling calculations into Python, write tests for them, control their versioning without having to mess around with files like CDS-OF-CDS EURO DESK 20180103 Final (final) (2).xlsx.
  • Dagger is a key technology to get financial models out of Excel, into a programming language and under tests and version control.
  • Time to drop a bit of a bombshell: the source code is in Barbara too, not on disk. Remain composed. It's kept in a special Barbara ring called sourcecode.
  • Interesting table structures, like Pandas, but closer to a DB (MnTable)
  • Over time the divergence between Bank Python and Open Source Python grows. Technology churns on both sides, much faster outside than in of course, but they do not get closer.
  • Minerva has its own IDE - no other IDEs work if you keep your source files in a giant global database.
  • What I can't understand is why it contains its own web framework. Investment banks have a one-way approach to open source software: (some of) it can come in, but none of it can go out
  • BTW, I “read” this with naturalreaders app

Chris #3: Pyxel

  • Pyxel is a ‘retro gaming console’ written in Python!
  • This might seem old and un-shiny, but the restrictions imposed by the environment gift simplicity
    • Vastly decreased learning time and effort compared to something like Unity or even Pygame
    • Straight forward simple commands, just like it was for micro-computers in the 80s
      • cls(), line(), rect(), circ() etc.
  • Pyxel is somewhat more Python and less console than others like PICO-8 or TIC-80 but this is a feature! Use your regular development environment to build.

Brian #4: How to Ditch Codecov for Python Projects

  • Hynek Schlawack
  • Codecov is a third party service that checks your coverage output and fails a build if coverage dropped.
  • It’s not without issues.
  • Hynek is using coverage.py --fail-under flag in place of this in GitHub actions.
  • It’s not as simple as just adding a flag if you are using --parallel to combine coverage for multiple test runs into one report.
  • Hynek is utilizing the coverage output as an artifact for each test, then pulling them all together in a coverage stage combine and check coverage.
  • He provides the snippet of GH Action, and even links to a working workflow file using this process. Nice!

Michael #5: tiptop (like glances)

  • via Zach Villers
  • tiptop is a command-line system monitoring tool in the spirit of top. It displays various interesting system stats, graphs it, and works on all operating systems.
  • Really nice visualization for your servers
  • Good candidate for pipx install tiptop

Chris #6: pyc64

  • A Commodore 64 emulator written in pure Python!
  • Not 100% complete - screen drawing is PETSCII character mode only
    • This still allows for a lot of interesting apps & exploration
  • Actual machine emulation using py65 - a pure Python 6502 chip emulator!
  • You can pop to a Python REPL from inside the emulator and examine data structures like memory, registers, etc!
  • An incredible example of what Python is capable of
  • 0.6 Mhz with CPython and over 2Mhz with pypy!

Extras

Michael:

Chris:

Joke: I hate how the screens get bright so early this time of year


Want to go deeper? Check our projects