python % string formatting #padding

Many details I tend to forget:

  • the actual data can be either a tuple or dict. The dict version is powerful but less known
    • if there’s just one item in the tuple, you don’t need to parenthesis —  myStr = “%d” % var1
  • for a tuple, the format specifier count must match the tuple length
  • for a dict, each format specifier must name a valid key value.
myStr = "%(var1)d" % locals()) # locals() returns a dict including var1 
  • There are at least two q(%) in the above expression
  • extra parentheses after the 2nd % are required:

“#%03d” % (id(node)%1000)  # return last 3 digit of object id, with 0-padding

hacking imported module #homemade trick

Background: Suppose in a big python application your main script imports a few packages and modules. One of them is mod2.py, which in turn imports mod2a.py.

Now You need to add invesgative logging/instrumentation to mod2a.py but this file is loaded from a readonly firm-wide repository, common practice in big teams. Here’s my tested technique:

1. clone mod2a.py to your home dir and add the logging. Now we need to import this modified version.
2. clone mod2.py to your home dir and open it to locate the importation of mod2a
3. edit mod2.py to change the importation of mod2a:

sys.path.insert(0, ‘/home/dir’)
import mod2a # via /home/dir
sys.path.remove(‘/home/dir’)

All other imports should be unaffected.

4. edit main.py and update the importation of mod2.py to load it too from /home/dir

As an additional hack, Some people may rename the modified mod2a.py file. This is doable IIF the import line is

from mod2a import someSymbol

Otherwise, every mention of “mod2a” in mod2.py needs a change

python run complex external commands #subprocess

I prefer one single full-feature solution that’s enough for all my needs. The os.system() solution is limited. The subprocess module is clearly superior. One of the simplest features is

>>> subprocess.call([“ls”, “-l”])

If you need redirection and background, then try the single-string version

>>> subprocess.call(‘ls /tmp > /tmp/a.log &’, shell=True) # output goes to STDOUT, hard to capture

python to dump binary data in hex digits

Note hex() is a built-in, but I find it inconvenient. I need to print in two-digits with leading 0.

Full source is hosted in https://github.com/tiger40490/repo1/blob/py1/tcpEchoServer.py

def Hex(data): # a generator function
  i=0
  for code in map(ord,data):
    yield "%02x " % code
    i += 1
    if i%8==0: yield ' '

print ''.join(Hex("\x0a\x00")); exit(0)

## 5 idioms of python (vs other dynamic lang)

I feel elegant techniques/idioms are usually non-OO, because OO code is too verbose to write quickly.

I feel best-practice cookbooks would showcase the most useful idioms.

# I feel perl is the first language to put the “hash” on center stage, as the most versatile apparatus (or “construct”). Ditto in Python. I guess the same data structure is at the heart of py/perl/javascript OO

# list comprehension and generator expressions. Until you start thinking in terms of these constructs, you aren’t truly a python practitioner.

# list processing functions — map() reduce() filter() zip(). I feel these are very popular in real  projects.

# decorators

file/console input output ] python

Output typically uses “print”.

print >> myFileObject , arguments
print >> sys.stderr , arguments…

Quiz: so, in that case, how do you print to stdout?

Input from file is very common, so that's another quiz

Input from std input —

aLine = sys.stdin.readline()
all_the_Lines = myFileObject.readlines()

Some prefer the object-oriented way to output

myFileObject.write(arguments….)

import package5 #(python)when package5 = directory

Python import statement has many trivial or non-trivial scenarios. Here are some common ones:

Common scenario 1: we “import modu2” to execute modu2.py and also exposes all the “public” objects and functions in modu2 as modu2.something

(Simplest scenario but still not known to some programmers.)

Common scenario 2: if pack3 is a directory containing some python modules, we can
“import pack3.modu5” or “from pack3 import modu5”. Easy to understand.

Here’s a relative unusual scenario:

./pip install --verbose --no-deps --index-url=http://ficlonapd05.macbank:9090/api/ pymodels-linux==0.0.1.480 # assuming 0.0.1.480 is the pymodels version to test

./python
>>> import pymodels
>>> pymodels.__file__
'/home/tcrmg/vtan7/myenv/lib/python2.6/site-packages/pymodels/__init__.pyc'
>>> pymodels.BlackImplyVol(.2, 3,3, 1, 0, 1,1)

In this case, there’s a pymodels directory but BlackImplVol is a function in the compiled library ./pymodels/pymodels_imp.so. How is this function accessible?

A: when “import pymodels” is run, the ./pymodels/__init__.py script executes, which does  “from pymodels_imp import * “

python script to descend n edit files in-place

import re
import os
import sys
from os import walk
import xml.dom.minidom as md

pretty_print = lambda f: ‘n’.join([line for line in md.parse(open(f)).toprettyxml(indent=’ ‘*2).split(‘n’) if line.strip()])

dirName, baseName = os.path.split(sys.argv[1])
print sys.argv
print dirName
print baseName

for (path, dirs, files) in walk(sys.argv[1]) :
        for oFileName in files :
                print oFileName; #raw_input(“…”)
                if not re.search(“vol.*.xml$”,oFileName): continue
                fullpath = path+”\”+oFileName;
                try:
                        write_file=open(fullpath+”_normalized.txt”,’w’)
                except Exception as e:
                        print e; raw_input(“…”)                      

                totalsubs = 0
                for line in open (fullpath): #path+”\tmp.txt”) :
                        # $2 not supported!
                        # \b same as in perl
                        newStr, subsMade = re.subn(‘\b(Tenor=”.*?”)s+(Date=”.*?”)’ , “\2 \1”, line)
                        if (subsMade > 0):
                                print newStr, # comma to suppress n
                                totalsubs += subsMade
                        write_file.write(newStr)
                write_file.close()
                print str(totalsubs) + ” total substitutions made — ” + fullpath

   
raw_input(“…”)
”’
                tmpFile=open(path+”\tmp.txt”, ‘w’)
                tmpFile.write(pretty_print (fullpath))
                tmpFile.close
”’

add a python script to context menu (windows explorer)

based on http://zephyrfalcon.org/weblog/arch_d7_2003_08_09.html#e306

Look for HKEY_CLASSES_ROOT.Folder to add to folder-level right-click context menu. (HKEY_CLASSES_ROOT.* i.e. the _asterisk_ for file-level context menu)

In your python script,

       dirName, baseName = os.path.split(sys.argv[1])
       print sys.argv
       print dirName
       print baseName

simple script to count classes defined in a python project

Any time you have a sizeable python project with many *.py source files, you can use this script to count how many classes defined.

import re, sys
from os import walk

printed={}
for (path, dirs, files) in walk(“c:\py”) :
       for filename in files :
               if not re.search(“.py$”,filename) : continue
               if not printed.has_key(path):
                       print ” path = ” + path
                       printed[path] = True

               for line in open (path+'\'+filename) :
                       #if re.search('^s*def ', line) : print line,
                       #if re.search('^s*classs', line) : print filename + ':t' + line,
                       if re.search('^s*trys', line) : print filename + ':t' + line,
                       #if re.search('@', line) : print filename + ':t' + line,