DISCLAIMER: The presented material relies heavily on Python Advance course carried out at CERN. The material is also available freely at the website: https://www.python-course.eu
In 1999, Guido Van Rossum submitted a funding proposal to DARPA called "Computer Programming for Everybody", in which he further defined his goals for Python:
print('Hello world!')
import this
Variable in python is always a reference to an object as in python everything, even a function, is an object.
x = 3
y = x
y, x
x = 2
y, x
Conditional statement to assign a value
x = -5
if x > 0:
label = 'Pos'
else:
label = 'Neg'
print(label)
x = -5
label = 'Pos' if x > 0 else 'Neg'
print(label)
print('Pos' if x > 0 else 'Neg')
string = 'My string'
string[0] = 'T'
string.replace('M', 'T')
string
String is iterable
for s in 'My string':
print(s)
Formating of strings
from datetime import date
'Today is ' + str(date.today()) + '.'
'Today is {} and number {}.'.format(date.today(), [1, 2, 3])
f-strings have been introduced in Python 3.6
print(f'Today is {date.today()}')
Check if a substring is in a string
if 'sub' in 'substring':
print('True')
There are already many built-in functions for handling strings in Python
dir(list)
dir(str)
'my first sentence'.upper()
Enum is a data type which links a name to an index. They are useful to represent a closed set of options
from enum import Enum
class QhBrowserAction(Enum):
QUERY_BUTTON_CLICKED = 1
SAVE_BUTTON_CLICKED = 2
DATE_CHANGED = 3
QH_NAME_CHANGED = 4
SLIDER_MOVED = 5
a = QhBrowserAction.DATE_CHANGED
a.name, a.value
a_next = QhBrowserAction(a.value+1)
a_next
if a_next == QhBrowserAction.QH_NAME_CHANGED:
print('In state {}'.format(a_next.value))
Container data types in Python are dedicated to store multiple variables of a various type. The basic container types are: lists, tuples, sets, dictionaries.
my_list = [1, 'b', True]
my_list
Lists are 0-indexed and elements are accessed by a square bracket
my_list[0]
Lists are mutable
my_list[1] = 0
my_list
In order to extend a list one can either append...
my_list.append(3)
my_list
Or simply
my_list + [1, 'b']
...or append elements
my_list += [3]
my_list
my_list = my_list + [3] # One shall not do that
my_list
Be careful with the last assignment, this creates a new list, so a need to perfom a copy - very inefficient for large lists.
How to append a list at the end?
my_list.append([1, 'a'])
my_list
This adds a list as an element, which is not quite what we wanted.
my_list.extend([5])
my_list
import itertools
list2d = [[1,2,3], [4,5,6], [7], [8,9]]
merged = list(itertools.chain(*list2d))
merged
Which one to choose in order to add elements efficiently?
Old-fashioned way
my_list = []
for i in range(10):
my_list.append(i)
my_list
One-line list comprehension
abs(0.1 - (1.1-1)) < 1e-16
my_list = [1/(i+1) for i in range(10)]
my_list
my_list = [i for i in range(10) if i > 4]
my_list
Generator comprehension
x = (x**2 for x in range(10))
print(x)
next(x)
import datetime
str(datetime.datetime.now())
print(datetime.datetime.now())
for x in ((x+1)**2 for x in range(int(1e7))):
x**(-1/2)
print(datetime.datetime.now())
print(datetime.datetime.now())
lst = [(x+1)**2 for x in range(int(1e7))]
for x in lst:
x**(-1/2)
print(datetime.datetime.now())
Generator returns values on demand - no need to create a table and than iterate over it
x = iter(range(10))
next(x)
x = (x**2 for x in range(10))
list(x)
my_list = [-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5]
filter(lambda x: x>0, my_list)
Filter returns an iterable generator. Generator is a very important concept in Python!
for el in filter(lambda x: x>0,my_list):
print(el)
list(filter(lambda x: x>0, my_list))
Map
print(my_list)
list(map(lambda x: abs(x), my_list))
Map can be applied to many lists
lst1 = [0,1,2,3,4]
lst2 = [5,6,7,8]
list(map(lambda x, y: x+y, lst1, lst2))
Reduce
sum([0,1,2,3,4,5,6,7,8,9,10])
from functools import reduce
reduce(lambda x, y: x+y, [0,1,2,3,4,5,6,7,8,9,10])
$0+1+...+n = \frac{n(n+1)}{2}$
i = 0
for el in [-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5]:
print(i, el)
i += 1
Iterating with index
for index, el in enumerate([-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5]):
print(index, el)
Iterating over two (many) lists
letters = ['a', 'b', 'c', 'd']
numbers = [1, 2, 3, 4, 5]
for l, n in zip(letters, numbers):
print(l, n)
list(zip(letters, numbers))
dict(zip(letters, numbers))
help(zip)
x = [1, 2, 3, 4]
y = x
y[0] = 'a'
print(x, y)
x.copy()
x = [1, 2, 3, 4]
y = x.copy()
y[0] = 'a'
print(x, y)
x = [[1, 'a'], 2, 3, 4]
y = x.copy() # equivalent to x[:]
y[0] = 'a'
print(x, y)
x = [[1, 'a'], 2, 3, 4]
y = x.copy()
y[0][0] = 'b'
print(x, y)
The reason for this behavior is that Python performs a shallow copy.
from copy import deepcopy
x = [[1, 'a'], 2, 3, 4]
y = deepcopy(x)
y[0][0] = 'b'
print(x, y)
x = [1, 10, 2, 9, 3, 8, 4, 6, 5]
x = x.sort()
print(x)
list.sort() is an inplace operation. In general, inplace operations are efficient as they do not create a new copy in memory
x = [1, 10, 2, 9, 3, 8, 4, 6, 5]
x.sort()
print(x)
list.sorted does create a new variable
x = [1, 10, 2, 9, 3, 8, 4, 6, 5]
sorted(x)
print(x)
x = [1, 10, 2, 9, 3, 8, 4, 6, 5]
x = sorted(x)
print(x)
x = [1, 10, 2, 9, 3, 8, 4, 6, 5]
x is sorted(x)
How to sort in a reverted order
x = [1, 10, 2, 9, 3, 8, 4, 6, 5]
x.sort(reverse=True)
print(x)
Sort nested lists
employees = [(111, 'John'), (123, 'Emily'), (232, 'David'), (100, 'Mark'), (1, 'Andrew')]
employees.sort(key=lambda x: x[0])
employees
employees = [(111, 'John'), (123, 'Emily'), (232, 'David'), (100, 'Mark'), (1, 'Andrew')]
employees.sort(key=lambda x: x[1])
employees
Also with reversed order
employees = [(111, 'John'), (123, 'Emily'), (232, 'David'), (100, 'Mark'), (1, 'Andrew')]
employees.sort(key=lambda x: x[0], reverse=True)
employees
my_list = 5*['a']
my_list
3 in [1,2,3,4,5]
x = ['a']
y = ['a']
x == y
x = ('a')
y = ('a')
x is y
Tuples, similarly to lists can stores elements of different types.
my_tuple = (1,2,3)
my_tuple
my_tuple[0]
Unlike the lists, tuples are immutable.
my_tuple[0]=0
tuple([1,2,3])
Sets are immutable and contain only unique elements
{1,2,3,4}
{1,2,3,4,4}
So this is a neat way for obtaining unique elements in a list
my_list = [1, 2, 3, 4, 4, 5, 5, 5]
set(my_list)
or a tuple
my_tuple = (1, 2, 3, 4, 4, 5, 5, 5)
set(my_tuple)
One can perform set operations on sets ;-)
A = {1,2,3}
B = {3,4,5}
print(f'A+B={A.union(B)}')
print(f'A-B={A-B}')
print(f'A*B={A.intersection(B)}')
print(f'A*0={A.intersection({})}')
pm = {'system', 'source', 'I_MEAS', 'I_REF'}
signals = pm - {'system', 'source'}
signals
for s in signals:
print(s)
help(set)
signals[0]
next(iter(signals))
list(signals)[0]
first, second = [1, 2]
print(first, second)
first, second = (1, 2)
print(first, second)
first, second = {1, 2}
print(first, second)
employees = [(111, 'John'), (123, 'Emily'), (232, 'David'), (100, 'Mark'), (1, 'Andrew')]
for employee_id, employee_name in employees:
print(employee_id, employee_name)
empty_set = {}
type(empty_set)
empty_set = set()
type(empty_set)
my_dict = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
my_dict
my_dict['a']
for key in my_dict:
print(key)
for key, value in my_dict.items():
print(key, value)
# lambda functions
f = lambda x: x**2
f(2)
def f(x):
return x**2
f(2)
def f(a, b, *, c):
return a+b+c
f(1,2,3)
f(1,2,c=3)
def f(*args):
return args[0]+args[1]+args[2]
f(1, 2, 3)
def f(**kwargs):
return kwargs['a'] + kwargs['b']
f(a=1, b=2, c=3)
A function passed as an argument
def f(x):
return x**2
def g(func, x):
return func(x)
g(f,2)
A function can return multiple values, in fact it returns a tuple
def f():
return 'a', 'b'
f()
first, second = f()
print(first)
print(second)
def factorial(n):
if n == 1:
return 1
else:
return n*factorial(n-1)
factorial(3)
factorial(-1)
def factorial(n):
if type(n) is not int or n <= 0:
raise Exception("Argument is not an integer")
if n == 1:
return 1
else:
return n*factorial(n-1)
factorial(5)
factorial(-1)
# Fibonacci
def fib(n):
if n == 0:
return 0
elif n == 1:
return 1
else:
return fib(n-1) + fib(n-2)
[fib(i) for i in range(6)]
How many times do we calculate fib(3)?
arguments = []
def fib(n):
arguments.append(n)
if n == 0:
return 0
elif n == 1:
return 1
else:
return fib(n-1) + fib(n-2)
x = [fib(i) for i in range(6)]
print(x)
counts = {i: arguments.count(i) for i in range(max(arguments)+1)}
counts
sum(counts.values())
In computing, memoization or memoisation is an optimization technique used primarily to speed up computer programs by storing the results of expensive function calls and returning the cached result when the same inputs occur again.
# Memoization for Fibonacci
# Fibonacci
memo = {0:0, 1:1}
arguments = []
def fib(n):
arguments.append(n)
if n not in memo:
memo[n] = fib(n-1) + fib(n-2)
return memo[n]
[fib(i) for i in range(6)]
counts = {i: arguments.count(i) for i in range(max(arguments)+1)}
counts
sum(counts.values())
Decorators are functions dedicated to enhance functionality of a given function, e.g., check parameter inputs, format input
def argument_test_natural_number(f):
def helper(x):
if type(x) is int and x > 0:
return f(x)
else:
raise Exception("Argument is not an integer")
return helper
def factorial(n):
if n == 1:
return 1
else:
return n*factorial(n-1)
factorial = argument_test_natural_number(factorial)
factorial(3)
factorial(-1)
def argument_test_natural_number(f):
def helper(x):
if type(x) is int and x > 0:
return f(x)
else:
raise Exception("Argument is not an integer")
return helper
@argument_test_natural_number
def factorial(n):
if n == 1:
return 1
else:
return n*factorial(n-1)
factorial(3)
factorial(-1)
def sum_aritmetic_series(n):
return n*(n+1)/2
sum_aritmetic_series(2)
sum_aritmetic_series(1.5)
@argument_test_natural_number
def sum_aritmetic_series(n):
return n*(n-1)/2
sum_aritmetic_series(2)
sum_aritmetic_series(1.5)
Fixing the Fibonacci series
def memoize(f):
memo = {}
def helper(n):
if n not in memo:
memo[n] = f(n)
return memo[n]
return helper
arguments = []
@memoize
def fib(n):
arguments.append(n)
if n == 0:
return 0
elif n == 1:
return 1
else:
return fib(n-1) + fib(n-2)
[fib(i) for i in range(6)]
counts = {i: arguments.count(i) for i in range(max(arguments)+1)}
counts
sum(counts.values())
There is a built-in cache decorator
# built-in least-recently used cache decorator
import functools
@functools.lru_cache(maxsize=128, typed=False)
def fib(n):
if n < 2:
return
else:
return fib(n-1) + fib(n-2)
# Exercise
# - write a decorator counting the number of times a function was called
# - the same but for a varying number of parameters and keyword-arguments
def counter(func):
# first define function
def helper(x, *args, **kwargs):
helper.count += 1
return func(x, *args, **kwargs) # return function as it is
# then, define an attribute to be incremented with every call
# this attribute behaves like a static variable
# helper exist only after the function definition. Once defined, then we can attach an attribute
helper.count = 0
return helper
@counter
def fun(x):
return x
fun(1)
fun(2)
fun(3)
fun.count
s = "Python"
itero = iter(s)
itero
# what I write is:
# for char in s:
# what python does:
# for char in iter(s)
# in fact it is a while loop until stop is reached
next(itero)
next(itero)
next(itero)
next(itero)
next(itero)
next(itero)
next(itero)
Own generator
def abc_generator():
yield "a"
yield "b"
yield "c"
x = abc_generator() # we call like a function. A function returns an object
for i in x:
print(i)
# print(next(x)) <-- yield "a"
# print(next(x)) <-- yield "b"
# print(next(x)) <-- yield "c"
# this is a co-process. This function creates a code waiting to be executed, when we assign x = abc_generator()
# after it reaches a yield, it returns value and stops. Then next is positioned fter the yield.x
x = abc_generator()
print(next(x))
print(next(x))
print(next(x))
print(next(x))
A function is also a single-value generator
def abc_generator():
return "a"
x = abc_generator()
for i in x:
print(i)
# works, because the returned value is iterable
type(abc_generator())
def abc_generator():
for char in ["a", "b", "c"]:
yield char
for i in abc_generator():
print(i)
type(abc_generator())
# Generate a pi value
# pi/4 = 1 - 1/3 + 1/5 - 1/7
def pi_series():
sum = 0
i = 1.0
j = 1
while True:
sum = sum + j/i
yield 4*sum
i = i + 2
j = j * -1
# runs forever
# we can break with a counter, but it is not a good idea
for i in pi_series():
print(i)
def firstn(g, n):
for i in range(n):
yield next(g)
print(list(firstn(pi_series(), 8)))
Is used to allocate and release some sort of resource when we need it.
Which means that before we start a block we open e.g. a file, and when we are going out, the file is automatically released.
If we don't close, it remains open in a file system. Closing a program, it would close. A good practice is to always close.
With context managers, the benefit is no need to close.
The issue is with the exceptions. With with, the exception is caught and handled.
Context manager is a general concept. The concept is as follows.
with device():
before:
1. check device
2. start device
we enter the block:
1. we do something
after:
1. we execute stop block
in case of exceptions we are sure that the after part will be executed.import csv
with open('example.txt', 'w') as out:
csv_out = csv.writer(out)
csv_out.writerow(['date', '# events'])
from contextlib import contextmanager
@contextmanager
def device():
print("Check device")
device_state = True
print("Start device")
yield device_state # the block after with is executed
print("Stop device")
with device() as state:
print("State is ", state)
print("Device is running!")
Exception handling
It is easier to ask for forgiveness than for permission
E.g.
if fileexisits(file_name):
txt = open(file_name).read()We first check if the file exists, then in the next step we fetch the file - two operations (asking for permission)
We can try to read, if it is there we are good, otherwise it raises an exception - single operation (asking for forgiveness)
try:
txt = open(file_name)
except Exception as e:
txt = ""while True:
try:
x = int(input("Please enter a number: "))
break
except ValueError as err:
print("Error message: ", err)
print("No valid number. Try again")
try:
some code
except ZeroDivisionError:
some code
there could be a raise here
except FooError:
some code
except BarError:
some code
finally:
some code executed always# Finally is executed always
try:
x = float(input("Your number: "))
inverse = 10/x
except ValueError as err:
print("Error message: ", err)
print("No valid number. Try again")
finally:
print("There may or may not have been an exception.")
print("The inverse: ", inverse)
# assert
x = 5
y = 6
assert x < y, "x has to be smaller than y"