Rapid Python Programming

Chapter 4

Files Working with files made simple. The Basics simple ↓open example:

>>> blah=open("data.txt", ’w’)
>>> blah.write("I am learning python\nI love python\nPython is great")
>>> blah.close()
>>>

4.1.2 Variations of Reading Data

File ’data.txt’ contains the following:

I am learning python
I love python
Python is great

’file1.py’ contains the following:

#!/usr/bin/python
# file1.py
blah = open("data.txt", "r")
print blah.readlines()
blah.close()

Results:

[’I am learning python\n’, ’I love python\n’, ’python is great\n’, ’\n’]

Let’s make changes to ’file2.py’ with the following:

#!/usr/bin/python
# file2.py
blah = open("data.txt", "r")
list = blah.readline()
for i in list:
	print i
blah.close()

results: I a m l e a r n ...

Here is something to remember: ’list = blah.↓read()’ does the same except output is on separate lines. Try it out with the script above and review the differences.

#!/usr/bin/python
# file2.py
blah = open("data.txt", "r")
list = blah.readlines()
for i in list:
	print i
blah.close()

results:

I am learning python
I love python
python is great

Adjust ’file3.py’ again:

#!/usr/bin/python
# file3.py
blah = open("data.txt", "r")
blah.seek(5)
print blah.readline()
blah.close()

results:

learning python

4.1.3 Parsing a file

In the previous chapters you learned to parse strings on a very limited bases. This section will provide a more detailed example on how to parse information in a very simple way. Pay ATTENTION to indentation!

#!/usr/bin/python
# parse.py
blah = open("data.txt", "r")
list=blah.readlines()
for i in list:
	if ’love’ in i:
	new=i
print "print specific line ’love’ in file"
print new
change=new.split()
print "print specific list in file as a list"
print change
print "print specific field in list ’love’"
print change[1]
for i in change:
	if ’python’ in i:
	wow=i
print "search for specific field in list ’python’ to print wow"
blah.close()

Results:

print specific line ’love’ in file
I love python
print specific list in file as a list
[’I’, ’love’, ’python’]
print specific field in list ’love’
love
search for specific field in list ’python’ to print
python

The above example demonstrates the ability to only print a specific instance on a specific line of a file even if there might be more than one instance of the word ’python’ in the file.

4.1.4 Binary Files

simple open binary file:

>>> blahbin = open ( ’binary.txt’, ’wb’ )
>>> blahbin.write ( ’write binary data.’ )
>>> blahbin.close()
>>>
>>> blahbin = open ( ’binary.txt’, ’rb’ )
>>> print blahbin.read()
>>> blahbin.close()
>>>

4.1.5 Pickle

help(’pickle’)

Now that we can read and write string, what about lists, tuples, and dictionaries?

4.1.6 Let’s Pickle

’Pickle’ is used with databases and a few other python frameworks to assist with persistent data manipulation.

#!/usr/bin/python
#apickle.py
import pickle
filePickle = open ( ’pickleFile.txt’, ’w’ )
list = [ ’I’, ’love’, ’python’, 1,2,3, ’ok’ ]
list2 = { ’firstname’: ’john’, ’lastname’: ’doe’, 1: ’one’}
pickle.dump ( list, filePickle)
pickle.dump ( list2, filePickle)
filePickle.close()

Now look at the contents of the ’pickleFile.txt’. Open it up and take a look inside the file with your editor:

(lp0
S’I’
p1
aS’love’
p2
aS’python’
p3
aI1
aI2
aI3
aS’ok’
...

Create ’lpickle.py’ and add the following: Pay ATTENTION to indentation!

#!/usr/bin/python
#lpickle.py
import pickle
filePickle = open ( ’pickleFile.txt’ )
list = pickle.load ( filePickle)
list # used while in idle screen
for i in list:
	print i
dict = pickle.load (filePickle)
dict # used while in idle screen
for i in dict:
	print i
dict[’firstname’]
for i, j in dict.iteritems():
	print i, j
filePickle.close()

results:

I
love
python
1
2
3
ok
...

4.1.7 cPickle

A faster version of pickle is ’cPickle’ which is a ’C’ compiled version of pickle. This can be easily imported as follows:

import cPickle as pickle

Here is an example its essentially the same but while using idle you get a different effect using ’test1’ or ’print test1’:

>>>import cPickle as pickle
>>>
>>> test1 = (’I love python’, 1, [1, 2, 3] )
>>> test1
(’I love python’, 1, [1, 2, 3])
>>>
>>> part1 = pickle.dumps(test1)
>>> part1
"(S’I love python’\nI1\n(lp1\nI1\naI2\naI3\natp2\n."
>>>
>>> print part1
(S’I love python’
I1
(lp1
I1
aI2
aI3
atp2
.
>>>

4.1.8 Python ↓with Statement

Using the ’with’ statement cuts down on exception handling for file stream by avoiding the need for ’try/finally’ blocks of code. According to python documentation, the ’with’ statement clarifies code previously by ’try...finally’ blocks to ensure clean-up code is executed. Syntax is as follows:

with expression [as variable]: with-block

The expression is evaluated, and it should result in an object that supports the context management protocol ’enter()’ and ’exit()’ methods. Let’s write data to a file:

with open(’output.txt’, ’w’) as f:
f.write(“Usual Hello World!”)

The above ’with’ statement will automatically close the file after nested block of code.

4.1.8.1 File Handling Modes

Table 4.1 ↓File Modes Modes Description

r Opens file for reading only
rb Opens file for reading binary only
r+ Opens file for both reading and writing
rb+ Opens binary file for reading and writing
w Opens file for writing only
wb Opens binary file for writing only
a Opens a file for appending
ab Opens a binary file for appending
a+ Opens a file for both appending and reading
ab+ Opens a file for both appending and reading in binary

4.1.8.2 File A↓ttributes

Table 4.2 File Attributes Attribute Description file.closed Returns true if closed file.mode Returns access mode of opened file file.name Returns the name of the file file.softspace Returns false if space explicitly required with print, otherwise true. Example below:

#!/usr/bin/python
#filetest.py
f = open("dat.txt", "wb")
print "Name the file: ", f.name
print "Closed or open: ", f.closed
print "mode: ", f.mode
print "Softspace: ", f.softspace

Also note you did not need ’dat.txt’ to run this script. Results:

Name the file: dat.txt
Closed or open: False
mode: wb
Softspace: 0

4.1.9 Deep Binary Manipulation

Python has a limitation when working with binary files. The smallest unit you can work with is a byte. This can prove challenging when working with a data stream. Working with binary is beyond scope in this book and requires a firm foundation in computer software

But you can use ↓bitwise operators to evaluate if a value is True or False. For example:

>>> x=3
>>> x&1!=0
True
>>> x&2!=0
True
>>> x&3!=0
True
>>> x&4!=0
False

4.1.9.1 bit string manipulation

This example converts string to binary using ↓base2

>>>print int(’001101’,2)
13
>>>print int(’0000011001’,2)
25
>>>print “0x%x” % int(’0011001’,2)
0x25

Please note you didn’t need to use 8 bits. My example above has 7 bits. However it took 1001 for the first value of 9 in the 1’s place and the next 001 for 10’s place making 19. Conversion to Character requires an 8 bit maximum string.

>>> print chr(int(’1111001’,2))
y
>>> print chr(int(’1111011’,2))
{

Conversion of characters to integers.

>>> print int(’1111011’,2)
123
>>> ord(’{’)
123

Another way to convert individual bits.

>>> 1<<0
1
>>> 1<<1
2
>>> 1<<3
8

Conversion of ↓Hex string to Integer

>>> int(’0xff’,16)
255

python 3 supports ↓binary literal by adding b’01011’ before the string to denote binary string.

The following is an example of bin(),oct(),and hex() conversion

>>> x=123
>>> bin(x)
’0b1111011’

Notice the literal ’b’ encoded in the string representation. Literal data types are available in python3.0 or by importing future library

>>> oct(x)
’0173’
>>> hex(x)
’0x7b’
>>> format(x,’b’)
’1111011’
>>> format(x,’o’)
’173’
>>> format(x,’x’)
’7b’

As you can see above ’int()’ object handles a lot. help(’int’)

class int(object)
| int(x=0) -> int or long
| int(x, base=10) -> int or long
|
| Convert a number or string to an integer, or return 0 if no arguments
| are given. If x is floating point, the conversion truncates towards zero.
| If x is outside the integer range, the function returns a long instead.
|
| If x is not a number or if base is given, then x must be a string or
| Unicode object representing an integer literal in the given base. The
| literal can be preceded by ’+’ or ’-’ and be surrounded by whitespace.
| The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to
| interpret the base from the string as an integer literal.
| >>> int(’0b100’, base=0)
| 4

4.1.10 Binary Struct

The structure of ’struct’ to pack and unpack. ↓pack() and ↓unpack() require a string to define how binary data is structured.

#mystruct.py
import struct
# i represents integer
# c represents character
bdata=struct.pack("icc", 42,b’A’, b’Z’)
print(bdata)
# unpack yields a tuple of data
tdata=struct.unpack("icc",bdata)
print(tdata)

Results:

*AZ
(42, ’A’, ’Z’)

The struct module performs conversions between Python values and ’C’ struct’s represented as a Python string. ↓Pack and ↓Unpack require ↓character formats to determine structure:

Format Type
x no values
c char
b signed int
B unsigned int
? bool
h int short
H unsigned short
i int
I unsigned int
l long
L unsigned long
q long long
Q unsigned long long
f float
d double
s char[]
p char[]
P void *

Below is an example of reading parsing a jpeg file and extracting size

#readjpg.py
import sys
import ↓binascii
import struct
signature=[
binascii.unhexlify(b’FFD8FFD8’),
binascii.unhexlify(b’FFD8FFE0’),
binascii.unhexlify(b’FFD8FFE1’) ]
with open(sys.argv[1],’rb’) as file:
	first_four_bytes=file.read(4)
if first_four_bytes in signature:
	print("JPEG found.")
else: print("This is not a JPEG file.")
	print first_four_bytes
	print binascii.hexlify(first_four_bytes)
second=file.read(1)
while (ord(second) !=0xDA):
	while (ord(second) !=0xFF): second=file.read(1)
		while (ord(second) ==0xFF): second=file.read(1)
			if (ord(second) >=0xC0 and ord(second) <=0xC3):
				print binascii.hexlify(file.read(3))
				print struct.unpack(’>HH’,file.read(4))

4.1.11 Summary

At this point, you have the ability to program just about anything using a structured top down method. Python works best as a higher level programming language and is not a practical language to work at a low level with large binary data. The binary example in this section will only be used as a reference and will not be discussed or used later in this book due to the complexity.

4.1.12 Exercises

Write two files. A library function to open and parse your file searching for the quantity of python in the sample file below. Create another file to import that library and print out lines (3-6), and replace every instance of keyword ’python’.

Sample file:

I love python
I will master python
python is the greatest
python pays big money
python will set you free
I will make a python script
python will be my future
All my programs will be replaced with python
only python
python python

To post a comment you need to login first.