python wiki

How can I inspect/print pickle data?

Fri, 17 May 2013 20:36:38 GMT

I thought it'd be cool to have some sort of "pickle inspector" window, where you could see the contents of saved pickle data. The below module is the result: You can drag&drop the pickle file onto this module's icon, and it'll print out the pickled data. That being said, it's pretty easy to break: If the pickled data requires any imports from modules that aren't part of your default Python path, it will fail. But I thought it was a good place to start.

# pickleInspector.py
import sys
import pickle
from pprint import pprint

def main(args):
    """
    Print the values from the pickled data.
    """
    if len(args) != 2:
        return
    with open(args[1]) as gerken:
        loadedData = pickle.load(gerken)
    pprint(loadedData)

if __name__ == "__main__":
    try:
        main(sys.argv)
    except Exception, e:
        print "EXCEPTION", e
    finally:
        raw_input("Press Enter to exit...")

Plant algorithms

Fri, 17 May 2013 17:26:00 GMT

Not Python related persay, but nice selection of papers here:
http://algorithmicbotany.org/papers/

How can I make sounds in Python?

Fri, 17 May 2013 16:27:00 GMT

On Windows, it's pretty easy to make some nice beeping sounds with winsound.Beep:
http://docs.python.org/library/winsound.html

import winsound
for i in range(50, 5000, 10):
    winsound.Beep(i, 25)

Blog post dealing with PyAudio:
http://davywybiral.blogspot.com/2010/09/procedural-music-with-pyaudio-and-numpy.html

Blog post about other Python implementations:
http://diagrammes-modernes.blogspot.com/2007/08/music-control-tools-python-based.html

Pyknon
"Pyknon is a simple music library for Python hackers. With Pyknon you can generate Midi files quickly and reason about musical
proprieties."

There's also the builtin wave library that can read and write wav data, but I'm not sure if it can actually play it.

How can I play a wav file?

Fri, 17 May 2013 16:27:00 GMT

On Windows:
http://docs.python.org/library/winsound.html

import winsound
winsound.PlaySound('c:/temp/myWav.wav', winsound.SND_ALIAS)

There's also the builtin wave library that can read and write wav data, but I'm not sure if it can actually play it.

How can I kill/close a window by name?

Mon, 13 May 2013 22:28:00 GMT

It's easy to make a Tkinter window, but I was often stumped on how to close them, if one is already open. Meaning: A window is open, the user re-clicks on the icon to create it: now there are two windows. Instead, I want the first one deleted, before the second one is created.

The below code will delete any application if it can match the given 'title' string in the data that the dos tasklist returns. Windows only, obviously. The idea is, before the window is created, this code is first ran to see if there is a pre-existing one, and if so, it kills it.

import subprocess
def killWindow(title):
    """
    Windows only:  Uses the tasklist & taskkill dos cmds.
    Kill the window with the given title.
    """
    output = subprocess.Popen(['tasklist', '/FO', 'TABLE', '/V', '/FI', 'IMAGENAME eq python*'],
                                stdout=subprocess.PIPE).stdout
    for line in output:
        strip = line.strip()
        if title in strip:
            # PID is the 2nd item in the list:
            pid = strip.split()[1]
            subprocess.Popen(['taskkill', '/PID', pid])

title = "My Tkinter Window"
killWindow(title)

How can I get a list of all running applications?

Mon, 13 May 2013 20:48:00 GMT

On Windows, you can easily acces this via the tasklist system command:

import subprocess
output = subprocess.Popen(['tasklist'], stdout=subprocess.PIPE).stdout
for line in output:
    print line.strip()

Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
System Idle Process              0 Services                   0         24 K
System                           4 Services                   0     38,204 K
smss.exe                       312 Services                   0      1,356 K
csrss.exe                      412 Services                   0      6,076 K
etc...

You can also filter the result by passing args to the tasklist command:

import subprocess
output = subprocess.Popen(['tasklist', '/FI', 'IMAGENAME eq svcHost.exe'], stdout=subprocess.PIPE).stdout
for line in output:
    print line.strip()

Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
svchost.exe                    692 Services                   0     12,284 K
svchost.exe                    840 Services                   0     13,968 K
svchost.exe                    940 Services                   0     20,796 K
etc...

Here's a more robust solution that turns all this data into a list of Python dicts, with each dict representing a specific process:

import subprocess
def getTaskData(searchStr=None):
    """
    Windows only:  Uses the tasklist dos cmd.
    Based on all running processes, returns a list of dicts: Each dict has these keys:
    "Image Name", "PID", "Session Name", "Session#", "Mem Usage", "Status",
    "User Name", "CPU Time", "Window Title".

    Optionally the user can pass in a search string, which will be used as a filter
    on the "Image Name" (executable name).
    """
    output = None
    if searchStr:
        output = subprocess.Popen(['tasklist', '/FO', 'LIST', '/V', '/FI', 'IMAGENAME eq python*'],
                                  stdout=subprocess.PIPE).stdout
    else:
        output = subprocess.Popen(['tasklist', '/FO', 'LIST'],
                                  stdout=subprocess.PIPE).stdout
    data = []
    appender = {}
    for line in output:
        stripped = line.strip()
        if len(stripped) == 0 and len(appender):
            data.append(appender)
            appender = {}
            continue
        kv = [item.strip() for item in line.split(": ")]
        if kv > 2:
            # Some of our 'values' have the ": " string in them, so we need to
            # rebuild that data after being split:
            k = kv.pop(0)
            v = ': '.join(kv)
            kv = [k,v]
        if kv[1] != '':
            appender[kv[0]] = kv[1]
    if not len(data):
        data.append(appender)

    return data

If you're looking for a way more directly authored in Python, check out this blog post:
http://www.blog.pythonlibrary.org/2010/10/03/how-to-find-and-list-all-running-processes-with-python/

How can I sort a list of files based on modification date?

Tue, 07 May 2013 18:46:00 GMT

You can create a custom sorter function that compares the modification date of the passed in list of files:

import os
import glob

def sortByDate(a,b):
    """
    Custom sort function designed to sort a list of files based on their
    modification date.
    """
    aTime = os.path.getmtime(a)
    bTime = os.path.getmtime(b)
    if aTime < bTime:
        return -1
    elif aTime == bTime:
        return 0
    elif aTime > bTime:
        return 1

files = glob.glob("c:/temp/*.txt")
dateSortedFiles = sorted(files, cmp=sortByDate)

How can I get a previous revision of a file from Perforce?

Mon, 06 May 2013 20:54:00 GMT

In the below example, we get a string representing the previous version of a given depot file from Perforce.

An interesting thing to note: p4.run('print') executes much faster than p4.run_print().

from P4 import P4

p4 = P4()
p4.connect()

depotFile = "//myDepot/path/to/my/file.txt"
depotInfo = p4.run("fstat", depotFile)
headRev = int(depotInfo[0]["headRev"])
prevRev = headRev-1
oldRevData = p4.run('print', "-q", ['%s#%s'%(depotFile,prevRev)])
oldRevStr = oldRevData[1]

Perforce access via Python

Mon, 06 May 2013 20:53:00 GMT

http://www.perforce.com/
Perforce is version control software that I use in games development. Having my animation package Maya (which supports Python scripting) interface with it is really handy. What resources exist to let Python talk to it?

They release their installers by version, by os:
Windows Installer (Python 2.5):

ftp://ftp.perforce.com/perforce/r07.3/bin.ntx86/p4python25.exe

Source:

ftp://ftp.perforce.com/perforce/r07.3/tools/p4python.tgz

Here's the root installer download page, to grab the latest version:

ftp://ftp.perforce.com/perforce/

Documentation:

Main Page: http://www.perforce.com/perforce/technical.html
Python specific HTML: http://www.perforce.com/perforce/doc.current/manuals/p4script/03_python.html#1116373
Multi-language .pdf: http://www.perforce.com/perforce/doc.current/manuals/p4script/p4script.pdf

Informative Blog Posts:

The install appears to stick the modules here:

C:\Python25\Lib\site-packages\

P4.py
p4.pyc
p4.pyo
P4API.pyd
P4Python-2007.3-py2.5.egg-info

Some example code modified from their manual (they had... mistakes...)

from P4 import P4, P4Exception

p4 = P4()
p4.port = "myPort:1666"
p4.user = "myName"
p4.client = "myClient"
try:
	p4.connect()
	info = p4.run("info") # returns a list
	d = info[0]  # extract the dictionary
	for key in d.keys():   # print our 'info'
		print key + "  :  " + str(d[key])
	p4.disconnect()
except P4Exception:
	for e in p4.errors:
		print e

Notes:

It appears that most of P4's commands are wrappered in methods that start with run_, or can be called to via the run() method (as in the example above):

info = p4.run("info")
info = p4.run_info()
# Both appear to do the same thing.

# Both do the same thing, but sometimes run takes extra args:
files = p4.run_edit([r"c:\myFile.txt", r"c:\myFileB.txt])
files = p4.run("edit", [r"c:\myFile.txt", r"c:\myFileB.txt])

I've also ran into issues that some commands run much faster than others based on the run() syntax.
For example, p4.run('print') executes much faster than p4.run_print().

How can I sync to a Perforce changelist?

Mon, 06 May 2013 20:49:00 GMT

Came up with two different ways, one via the API, and one via the system. The API is nice, but I can't find any way to extract the results. Could just be my ignorance. Calling to P4 at the system level makes it easy to get the results:

# Via the P4 API:
import P4
changelist = 1234
p4 = P4.P4()
p4.connect()
# Only raise exceptions, not warnings
p4.exception_level = 1
print "Starting P4 sync for changelist %s"%changelist
p4.run_sync('@=%s'%changelist)
p4.disconnect()

# Via calls to the system:
import subprocess
changelist = 1234
print "Starting P4 sync for changelist %s"%changelist
result  = subprocess.Popen(['p4', 'sync', '@=%s'%changelist], 
                           stdout=subprocess.PIPE).communicate()
foo = [f.strip() for f in result[0].split("\r\n") if f != ""]
if len(foo):
    for f in foo:
        print "\t", f
else:
    print "\tEverything is up to date."
print "Finished P4 changelist sync, see results above:"

http://www.perforce.com/perforce/doc.current/manuals/p4script/03_python.html#1116373

How can I query if a file is in Perforce?

Mon, 06 May 2013 20:49:00 GMT

If a file lives in Perforce, the fstat command will return a bunch of info on it. If it doesn't live in P4, it returns an empty list.
Via the Perforce Python API:

import P4
p4 = P4.P4()
# only raise excptions on errors, not warnings:
p4.exception_level = 1
p4.connect()
result = p4.run_fstat(pathToMyFile)
p4.disconnect()
if result:
    print "Lives in P4!", pathToMyFile
else:
    print "Not in P4!", pathToMyFile

MainMenu

Mon, 06 May 2013 20:48:00 GMT

pickle, protocol, and perforce

Mon, 06 May 2013 20:48:00 GMT

http://docs.python.org/library/pickle.html
I recently had an issue where I'd pickle some data to disk, and add the data to Perforce (P4) so others could access. But when the other team members would sync to the data, they couldn't use it. They'd get an exception with this info:

No module named copy_reg

Looking online the common fix was to make sure your data was pickled in binary format. But this didn't help.
Doing some research, I found that the size of the file would get bigger after checking into P4, although when I'd diff the file it would say there were no differences. The plot thickens....

Talking with our IT department, they came to the idea that maybe P4 was changing the line ending of the files: If you open your Clientspec in P4, there is a 'LineEnd' drop-down, mine was set to 'local'. Changing this to 'unix' and resubmitting the file suddenly fixed the problem. But we couldn't have our whole team make this change, especially not knowing what other repercussions it may have, so I set it back.

So we took a look at the pickle docs. Come to find out, introduced in Python 2.3, there was a new 'protocol' arg provided (protocol '2') which according to the docs: "...provides much more efficient pickling of new-style classes.".

The original code was this:

outf = open(dataFile, 'wb')
pickle.dump(data, outf)
outf.close()

The new code is this:

outf = open(dataFile, 'wb')
pickle.dump(data, outf, 2)
outf.close()

Setting that new protocol value fixed the problem: The data could be added to P4, other team-members could access it no problem.

How can I make custom changelists in Perforce, and have Python know what number they are?

Mon, 06 May 2013 20:48:00 GMT

If you're wondering: http://www.perforce.com/

Via the Perforce API:

P4 API Docs
This will both create a changelist, and optionally add files to it. The file add system is terribly limited: It will only work on files already existing in P4, that aren't already open for edit. You'd need to write a more robust set of conditions to work with files that are to be added, files that live in another changelist, files that should be deleted, or files that someone else has checked out. But it gives you an idea for where to start...
Thanks to an email from Ross Kameny got me this code snippet:

import P4

def makeChangelist(description, files=[]):
    """
    description : string : The changelist description
    files : list : The files to edit.  Default is an empty list.
    """
    p4 = P4.P4()
    p4.connect()
    if files:
        p4.run_edit(files)
    changeList = p4.fetch_change()
    changeList["Description"] = description
    if files:
        changeList["Files"] = files
    results = p4.save_change(changeList)
    changelistNum = int(results[0].split(' ')[1])
    p4.disconnect()

    return changelistNum

Via calls to the system:

This just makes an empty changelist, doesn't add any files to it.
First, you need to generate a text file with the change info in it, and save it. Like c:/temp/myChangelist.txt:

Change:  new
Description:
      Enter description here, one line

Then in Python, call to P4 to create the changelist, then find the changelist number (to be used to add things to the list):

import os
p4changeReturn = os.popen("P4 change -i < c:/temp/myChangelist.txt").read().strip()
buffer = p4changeReturn.split(" ")
changelistNumber = buffer[1]

FYI: In Python 2.6 and newer, popen is depricated. Check notes on subprocess here

Delete all empty changelists in Perforce for current client

Mon, 06 May 2013 20:48:00 GMT

I'm always surprised that the Perfoce ui doesn't give the user the option to delete all empty changelists. Here's a simple function that will do it via the P4 API. See Perforce access via Python.

import P4

def deleteEmptyChangelists():
    p4 = P4.P4()
    # User needs to handle p4.errors & p4.warnings (not done in this code):
    p4.exception_level = 0
    p4.connect()
    # Get all current changes for this client:
    data = p4.run("changes", "-s", "pending", "-c", p4.client)

    emptyChange = []
    for d in data:
        if 'shelved' in d:
            # Archived\shelved, skip
            continue

        # Find files in this change:
        files = p4.run("describe", d['change'])
        if 'depotFile' not in files[0]:
            emptyChange.append(d['change'])

    if emptyChange:
        for empty in emptyChange:
            # Need to loop, can't delete multiple changes at once:
            p4.run("change", "-d", empty)
        print "Deleted empty changelists:", emptyChange
    else:
        print "No empty changelists to delete."

    p4.disconnect()
    return emptyChange

Perforce Python API docs

Accessing user temp directory and files

Tue, 09 Apr 2013 00:05:00 GMT

(On the Windows system at least...) this directory is stored in the system variable TMP or TEMP.

Create a name for a file in the users temp directory:

import os

# TMPDIR on mac
tempfile = os.path.join(os.getenv("TMP"), "tempfile.txt")
print tempfile

On Vista:

C:\Users\<USERNAME>\AppData\Local\Temp\tempfile.txt

On Mac:

/var/folders/kv/dk8t43d535s3pvw9z_8zhh2c0000gn/T/tempfile.ma

http://docs.python.org/library/tempfile.html
http://blog.doughellmann.com/2008/02/pymotw-tempfile.html

How can I get a list of Environment Variables?

Tue, 19 Mar 2013 16:30:00 GMT

os.environ returns a Dictionary

import os
for key in os.environ:
    print key + " : " + os.environ[key]

Also see:

How can I set an environment variable?

Tue, 19 Mar 2013 16:30:00 GMT

If you can query an environment variable via os.getenv like so:

import os
myvar = os.getenv("MYVAR")

You'd think you'd be able to able to set one like the same way, via os.putenv:

os.putenv("MYVAR", "my var value")

On Windows however, putenv doesn't seem to do anything.... :-S

To actually update the var, you need to update via os.environ

os.environ["MYVAR"] = "my var value")

Also see:

How can I get a list of Environment Variables?

How can I query the value of an Environment Variable?

Tue, 19 Mar 2013 16:29:00 GMT

os.environ returns a Dictionary

import os
var = os.getenv("PATH")

Also see:

Pre-packaged Python distributions

Mon, 04 Mar 2013 19:36:00 GMT

There are a number of "Precompiled Python Distributions" available. There is a complete list here. Two of the more popular ones are:

Enthought Python Distribution (free version)
ActiveState ActivePython
Portable Python
- A list of what packages it has can be found for each of its downloads: 2.x, 3.x (those links are bound to stale, navigate from their home page for latest).
- It's Windows only :(