MoonPoint Support Logo


Shop Amazon Warehouse Deals - Deep Discounts on Open-box and Used ProductsAmazon Warehouse Deals

Advanced Search
Sun Mon Tue Wed Thu Fri Sat
24 25 26
27 28 29 30 31    
Oct Nov Dec

Wed, Aug 23, 2017 11:36 pm

Determining a file's type from within a Python script

I needed a way to determine a file's type within a Python script when I can't rely on the file's extension to determine the file format. I'll be running the script on a MacBook Pro laptop running the OS X El Capitan operating system. OS X/macOS, like Linux, comes with the file command, so I could run that command at a shell prompt to have the utility check the magic number in the files I'm interested in, but I want to do some additional processing of the files within the Python script, so I want to perform the format check within Python. Python provides the subprocess module that provides the capability to "spawn new processes, connect to their input/output/error pipes, and obtain their return codes." So I can call the file utility from within Python using that module. To get the results from running a shell command, you use suprocess.Popen(). You can then set a variable to hold the results of .communicate() and print the contents of that variable as shown below. The script expects the name of the file to be checked to be provided as an argument on the command line.


import subprocess as sub, sys

except IndexError:
   print "Error - missing input file name! Usage ./ infile"
   fileName = sys.argv[1]

p = sub.Popen(['file',fileName],stdout=sub.PIPE,stderr=sub.PIPE)
output, errors = p.communicate()
print output

[ More Info ]

[/languages/python] permanent link

Mon, Aug 21, 2017 11:15 pm

Find all occurrences of a string in a file using Python

I need to download and view Excel workbooks, Portable Document Format (PDF), and sometimes other types of documents related to work requests from a website. When I view a webpage for a particular work request, some of the documents may have been posted weeks or months ago while some are more recent, but when I download them they all get the timestamp of the time I downloaded them, but I would like to change the timestamps to match the timestamps on the website. On the webpage for a particular work request, I see the following information for files:

Type Name Site Modified Modified By
xlsm icon CRQ000000884164_PDSO -None- 3/6/2017 4:53 PM Smith, Gary
pdf icon CRQ000000884164_PDSO -None- 3/6/2017 4:52 PM Smith, Gary
pdf icon CRQ000000884164_DCS -None- 6/12/2017 9:29 AM Doe, Mike
xlsm icon CRQ 884164_SDP -None- 6/12/2017 9:30 AM Doe, Mike

[ More Info ]

[/languages/python] permanent link

Fri, Aug 18, 2017 10:41 pm

Importing data from a text file into an SQLite database with Python

I need to track work requests that are associated with particular projects. The work requests are submitted through a website, but I want to be able to run queries on the data that aren't available to me from the website. So I decided to add a projects table to an existing SQLite database I have on my MacBook Pro laptop. I didn't want to manually enter a list of about 200 projects into the table, however, so I copied the projects list from the website into a text file. The webpage that lists all of the projects includes projects that are no longer valid and I want to include those in my database table, but note that they are longer valid. When I copy the data from the web page, I have lines like the following ones:

450 SN/GN Tech  Edit Delete
ACE     Edit Delete
ADO     Edit Undelete
AGO     Edit Delete
AGS     Edit Delete
AIM     Edit Delete
Artemis         Edit Undelete
ASF     Edit Delete
ATSC    Edit Undelete
AXAF    Edit Undelete
BATSE/Gro       Edit Undelete
CANDOS  Edit Undelete
CARA    Edit Delete
CD Manager      Edit Delete
CMF     Edit Undelete
CMOC    Edit Delete
CMS     Edit Undelete

The entries that are no longer valid have "Undelete" at the end of the line indicating the projects were deleted, but they can be undeleted.

I created a table in the database with the following columns all of which contain text data:

NameDescription AddedRemoved DeletedNotes

[ More Info ]

[/languages/python] permanent link

Thu, Jul 20, 2017 10:49 pm

Passing a parameter to a Python script from a web page

I have a Python script that I use to process a copy of a webpage downloaded from a website and stored on my MacBook Pro laptop's hard drive to produce a CSV file from the data within that file. I was running the script from a command-line interface (CLI), i.e., a Terminal window, on the system by issuing a command like ./ inputfile outputfile where inputfile and outputfile were the file names and locations of the file holding the data and the output CSV file, respectively. I wanted to execute that script from a link on a web page, instead, so I needed a way to pass the arguments I had been passing on the command line to the Python script in the URL that I'd specify as the link on the web page. One way that you can do that for Python is explained at [Tutor] Passing Arguments to a Remote CGI Script where the following sample Python script is shown:

"""test.cgi  --- a small CGI program to show how one can pass parameters.

import cgi
fs = cgi.FieldStorage()

print "Content-type: text/plain\n"
for key in fs.keys():
    print "%s = %s" % (key, fs[key].value)

[ More Info ]

[/languages/python] permanent link

Fri, Jul 07, 2017 10:46 pm

Printing the error encountered with Python

In a Python script I was calling from a webpage residing on an Apache webserver, I was unable to copy a file whose location and file name were stored in the variable countfile to another file whose name and location were stored in the variable backupfile with any of the following lines of code:

shutil.copy(countfile, backupfile)
shutil.copy2(countfile, backupfile)
shutil.copyfile(countfile, backupfile)

The backup file should be stored in the same directory as countfile using the same name, but with ".bak" appended to the file name. I wasn't able to identify the cause of the problem when I just used except to print my own error message with the code below, since all I would see in the output was "Error! Unable to make a backup copy of the input file."

import shutil

backupfile = countfile + ".bak"

   # Make a backup copy of the prior file
   shutil.copy(countfile, backupfile)
except e:
   print "Error! Unable to make a backup copy of the input file."

But the explicit Python error message can be obtained by using except Exception, e as shown below:

   shutil.copy(countfile, backupfile)
except Exception,e:
   print str(e)
   print "Error! Unable to make a backup copy of the input file."

When I used that code, instead, I saw the following output for the error, which showed that the problem was with the creation of the backup file:

[Errno 13] Permission denied: '/Users/jasmith1/Documents/www/SGRS/data/SGRS_Count.csv.bak'
Error! Unable to make a backup copy of the input file.

[ More Info ]

[/languages/python] permanent link

Tue, Apr 11, 2017 8:37 pm

Writing to a CSV file with Python

I need to track the status of firewall rule requests and provide a status report every Monday. The requests are managed through a website where I can see the list of uncompleted requests and their status, but the site doesn't give me a count of the number of requests in each of the states that a request can be in, which can be "Pending Approval", "Modified", "Pending Removal" "On Hold", "Clarification Required", "Waiting Implementation", or "Waiting Removal". To obtain the count of the number of requests in each state, I download the webpage showing the requests and their status to my laptop and then run a Python script to count the queued items. But I also want to see how the numbers are changing over time, so I modified the script to write the counts to a comma-separated values (CSV) file that I can open in a spreadsheet program, such as Microsoft Excel.

To work with comma-separated value (CSV) files in a Python script, include the command import csv in the script. I also include the os.path module to use for testing whether a file already exists that will hold the data, the re module to perform regular expression parsing of lines in the downloaded webpage, the sys module for checking the command line arguments to the script, and the datetime module for determining the current date and formatting it in YYY-MM-DD format, where "YYYY" is the year, "MM" the month, and "DD" the day of the month, as explained in Python - Checking times and dates, for writing the date to the output file.

[ More Info ]

[/languages/python] permanent link

Fri, Apr 07, 2017 3:14 pm

Counting SQLite records based on a specified date

I track work requests in an SQlite database. For each request in a table named "Tasks" in the database, there is a column holding the date the request was approved. Every week I need to include the number of requests approved that week in a weekly report. I can do that with the Structured Query Language (SQL) command SELECT COUNT(*) FROM Tasks WHERE Approved >= date(CURRENT_DATE,"-7 day"). I have a Python script that queries the SQL database to count the requests approved within the last week, but sometimes I want to determine the number of requests approved since a particular date. So I modified the script to accept a date provided as an argument on the command line and, if a date is specified, to determine the number of entries where the approval date matches the specified date or is later than the specified date.

[ More Info ]

[/languages/python] permanent link

Wed, Apr 05, 2017 10:47 pm

Counting queued items with a Python script

I need to review and approve firewall rule requests. I can get a list of those pending approval, ones requesting a modification to existing rules, those pending removal because they've expired (all rules must be reviewed on at least a yearly basis), ones on hold, ones that are in a "clarification required" state due to a question about the rules being requested, those approved for implementation, but not yet implemented, and those awaiting removal. But the page that displays the requests in each category doesn't give me a count of the number in each category, so I wrote a Python script that will read the webpage containing that information that I've downloaded, parse the HTML code for the page and give me a count of the number of requests in each category.

The script is named . The name of the HTML file that was downloaded should be provided on the command line. E.g.:

$ ./ ~/Documents/Work/queued/Request.html
Request Status

Pending Approval:       69
Modified:               36
Pending Removal         43
On Hold:                0
Clarification Required: 28
Waiting Implementation: 26
Waiting Removal:        12

Total requiring review: 176

[ More Info ]

[/languages/python] permanent link

Tue, Feb 21, 2017 9:09 pm

Viewing the list of IP address ranges for AmazonAWS

If you need to create firewall rules for outgoing connectivity to Amazon Web Services (AWS) addresses, you can obtain a complete listing of the IP address ranges that Amazon uses in JavaScript Object Notation (JSON) format from AWS IP Ranges Are Now Available in JSON Format - the download URL for the file is

I downloaded the IP address range file from using cURL, which is present on Apple OS X/macOS systems and is available for Linux and many other operating systems, including Microsoft Windows - see Releases and Downloads.

$ curl -o ip-ranges.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   99k  100   99k    0     0   191k      0 --:--:-- --:--:-- --:--:--  191k

The following Python script can be used to extract just the Classless Inter-Domain Routing (CIDR) address blocks, i.e., the ip_prefix information from the JSON data stored in the ip-ranges.json file downloaded from the Amazon website and display it.

import json

data = json.load(open('ip-ranges.json'))
for d in data["prefixes"]:
    print d["ip_prefix"]

[ More Info ]

[/languages/python] permanent link

Sat, Feb 11, 2017 10:27 pm

Using Python to query an SQLite database and return a count of records

I have an SQLite database that I use to track work requests on my MacBook Pro running OS X El Capitan (10.11.6) . Every week I need to determine the number of change requests I've reviewed and approved in the last week. I have been running SQL commands from a text file using sqlite3, but I wanted to start using a Python script, instead, since I want to everntually perform some additional queries as well and display the results as a web page. Initially, I had some problems displaying the record count, but found I could use cursor.fetchone() to display the count. The script I currently have consists of the following code:


import sqlite3
conn = sqlite3.connect('/Users/jasmith1/Documents/Work/CRQ/CRQ.db')
cursor = conn.cursor()

sql = 'SELECT COUNT(*) FROM Tasks WHERE Approved >= date(CURRENT_DATE,"-7 day")'
result = cursor.fetchone()
numApproved = result[0]
print numApproved

The first step I need to take is to import the sqlite3 module for use in Python. I then need to create a connection object for the database which I assign to conn. After that I need to create a cursor instance that I will use to execute an SQL statement. I assign the SQL statement to the variable sql (the name is arbitrary) and then execute the statement with cursor.execute(sql). The variable result is then assigned the results of cursor.fechone(). If I printed the contents of the result variable, I would see (5,), if the count was 5, since the result is a tuple with one element, but I want to see just the number, so I set numApproved to be result[0]. Instead of setting numApproved to result[0] and printing numApproved, I could skip a step and use the following two lines after sursor.execute(sql):

(numApproved,) = cursor.fetchone()
print numApproved

If I make the script excecutable with chmod 755 and then run the script, I will then see just the number of reccords approved within the last week:

$ chmod 755
$ ./


  1. sqlite3 — DB-API 2.0 interface for SQLite databases
    Python documentation

[/languages/python] permanent link

Once You Know, You Newegg AliExpress by

Shop Amazon Local - Subscribe to Deals in Your Neighborhood

Valid HTML 4.01 Transitional

Privacy Policy   Contact

Blosxom logo