Brian C. Lane

Requires python-tiny package on Fedora. Cleans up the HTML, preventing broken elements from spilling over into adjacent postings. Code was lifted from feedparser.py and dropped into a plugin for rawdog since I couldn’t find an easy way to get mx.Tiny installed.

# rawdog plugin to tidy up html output using python-tidy module
# Brian C. Lane <bcl@brianlane.com>
#
from tidy import parseString
import rawdoglib.plugins, re

def tidy_html(config, box, baseurl, inline):
    data = box.value
    utf8 = type(data) == type(u'')
    if utf8:
        data = data.encode('utf-8')

    data = str(parseString(data, output_xhtml=1, numeric_entities=1, wrap=0))

    if utf8:
        data = unicode(data, 'utf-8')
    if data.count('<body'):
        data = data.split('<body', 1)[1]
        if data.count('>'):
            data = data.split('>', 1)[1]
    if data.count('</body'):
        data = data.split('</body', 1)[0]

    box.value = data.strip()

rawdoglib.plugins.attach_hook("clean_html", tidy_html)

Tonight I finished adding delete handlers and a delete confirmation dialog to hms. This is probably the state that the code will be in for my LFNW presentation next Saturday.

Newseum archives the front pages of of over 500 newspapers from all around the world. If you know the ID of the papers you want to see you can use this simple Python program to download the jpg of the papers’ front page to your local system.

Edit the CITIES list to set the IDs of the papers to be grabbed.

#!/usr/bin/env python
"""
    Quick Newseum Frontpage Grabber script
    Copyright 2009 by Brian C. Lane
    Imp Software
    All Rights Reserved

    Modify CITIES list below to add the city designators (as seen in the
    URLS at http://www.newseum.org/todaysfrontpages/default.asp)
"""
import urllib2
import re
import os
import urlparse

# Add more cities here
CITIES = [ "AL_AS", "AL_MA",   ]

NEWSEUM_URL="http://www.newseum.org/todaysfrontpages/hr.asp?fpVname=%s"
NEWSEUM_IMG="http://www.newseum.org"

def fetchNewseumImage(city):
    """
    Fetch the image for a city
    """
    print "Parsing the page for %s" % (city)
    page = urllib2.urlopen(NEWSEUM_URL % city).read()

    # Quick and dirty grep for the image name
    match = re.search('<img class="tfp_lrg_img" src="(.*)" alt=', page)
    if match:
        img_url = NEWSEUM_IMG + os.path.abspath(match.group(1))
        print "Saving the image for %s" % (city)
        image = urllib2.urlopen(img_url).read()
        open(os.path.basename(match.group(1)), "wb").write(image)

def main():
    """
    Main code goes here
    """
    for city in CITIES:
        fetchNewseumImage(city)

if __name__ == '__main__':
    main()

The source is also hosted here at github.