1 Structuring Your Project
2 ========================
4 When we say structure we mean the decisions you make concerning
5 how your project best meets its objective. We need to consider how to
6 best leverage Python's features to create clean, effective code.
7 In practical terms structure means the actual files and folders that
8 comprise your project as well as the way dependencies are handled within
11 How does data flow through the project? What features and functions
12 can be grouped together and isolated? By answering questions like these
13 you can begin to plan, in a broad sense, what your finished product will
16 In this section we take a closer look at Python's module and import
17 systems as they are the central element to enforcing structure in your
24 Thanks to the way imports and modules are handled in Python, it is
25 relatively easy to structure a Python project. Easy, here, means
26 that you do not have many constraints and that the module
27 importing model is easy to grasp. Therefore, you are left with the
28 pure architectural task of crafting the different parts of your
29 project and their interactions.
31 Easy structuring of a project means it is also easy
32 to do it poorly. Some signs of a poorly structured project
35 - Multiple and messy circular dependencies: if your classes
36 Table and Chair in furn.py need to import Carpenter from workers.py
37 to answer a question such as table.isdoneby(),
38 and if conversely the class Carpenter needs to import Table and Chair,
39 to answer the question carpenter.whatdo(), then you
40 have a circular dependency. In this case you will have to resort to
41 fragile hacks such has using import statements inside
44 - Hidden coupling: each and every change in Table's implementation
45 breaks 20 tests in unrelated test cases because it breaks Carpenter's code,
46 which requires very careful surgery to adapt the change. This means
47 you have too many assumptions about Table in Carpenter's code or the
50 - Heavy usage of global state or context: instead of explicitly
51 passing ``(height, width, type, wood)`` to each other, Table
52 and Carpenter rely on global variables that can be modified
53 and are modified on the fly by different agents. You need to
54 scrutinize all access to these global variables to understand why
55 a rectangular table became a square, and discover that remote
56 template code is also modifying this context, messing with
59 - Spaghetti code: multiple pages of nested if clauses and for loops
60 with a lot of copy-pasted procedural code and no
61 proper segmentation are known as spaghetti code. Python's
62 meaningful indentation (one of its most controversial features) make
63 it very hard to maintain this kind of code. So the good news is that
64 you might not see too much of it.
66 - Ravioli code is more likely in Python: it consists of hundreds of
67 similar little pieces of logic, often classes or objects, without
68 proper structure. If you never can remember if you have to use
69 FurnitureTable, AssetTable or Table, or even TableNew for your
70 task at hand, you might be swimming in ravioli code.
76 Python modules are one of the main abstraction layers available and probably the
77 most natural one. Abstraction layers allow separating code into parts holding
78 related data and functionality.
80 For example, a layer of a project can handle interfacing with user actions,
81 while another would handle low-level manipulation of data. The most natural way
82 to separate these two layers is to regroup all interfacing functionality
83 in one file, and all low-level operations in another file. In this case,
84 the interface file needs to import the low-level file. This is done with the
85 `import` and `from ... import` statements.
87 As soon as you use `import` statements you use modules. These can be either built-in
88 modules such as `os` and `sys`, third-party modules you have installed in your
89 environment, or your project's internal modules.
91 To keep in line with the style guide, keep module names short, lowercase, and
92 be sure to avoid using special symbols like the dot (.) or question mark (?).
93 So a file name like `my.spam.py` is one you should try to avoid! Naming this way
94 will interfere with the way python looks for modules.
96 In this example python expects to find a "spam.py" file in a folder named "my"
97 which is not the case. There is an
98 `example <http://docs.python.org/tutorial/modules.html#packages>`_
99 of how the dot should be used available in the python docs.
101 If you'd like you could name it as `my_spam.py` but even our friend the
102 underscore should not be seen often in module names.
104 Aside for some naming restrictions, nothing special is required for a Python file
105 to be a module, but the import mechanism needs to be understood in order to use
106 this concept properly and avoid some issues.
108 Concretely, the `import modu` statement will look for the proper file, which is
109 `modu.py` in the same directory as the caller if it exists. If it is not
110 found, the Python interpreter will search for `modu.py` in the "path"
111 recursively and raise an ImportError exception if it is not found.
113 Once `modu.py` is found, the Python interpreter will execute the module in an
114 isolated scope. Any top-level statement in `modu.py` will be executed,
115 including other imports if any. Function and class definitions are stored in
116 the module's dictionary.
118 Then, the module's variables, functions, and classes will be available to the caller
119 through the module's namespace, a central concept in programming that is
120 particularly helpful and powerful in Python.
122 In many languages, an `include file` directive is used by the preprocessor to
123 take all code found in the file and 'copy' it into the caller's code. It is
124 different in Python: the included code is isolated in a module namespace, which
125 means that you generally don't have to worry that the included code could have
126 unwanted effects, e.g. override an existing function with the same name.
128 It is possible to simulate the more standard behavior by using a special syntax
129 of the import statement: `from modu import *`. This is generally considered bad
130 practice. **Using `import *` makes code harder to read and makes dependencies less
133 Using `from modu import func` is a way to pinpoint the function you want to
134 import and put it in the global namespace. While much less harmful than `import
135 *` because it shows explicitly what is imported in the global namespace, its
136 advantage over a simpler `import modu` is only that it will save some typing.
140 .. code-block:: python
145 x = sqrt(4) # Is sqrt part of modu? A builtin? Defined above?
149 .. code-block:: python
151 from modu import sqrt
153 x = sqrt(4) # sqrt may be part of modu, if not redefined in between
157 .. code-block:: python
161 x = modu.sqrt(4) # sqrt is visibly part of modu's namespace
163 As said in the section about style, readability is one of the main features of
164 Python. Readability means to avoid useless boilerplate text and clutter,
165 therefore some efforts are spent trying to achieve a certain level of brevity.
166 But terseness and obscurity are the limits where brevity should stop. Being
167 able to tell immediately where a class or function comes from, as in the
168 `modu.func` idiom, greatly improves code readability and understandability in
169 all but the simplest single file projects.
175 Python provides a very straightforward packaging system, which is simply an
176 extension of the module mechanism to a directory.
178 Any directory with an __init__.py file is considered a Python package. The
179 different modules in the package are imported in a similar manner as plain
180 modules, but with a special behavior for the __init__.py file, which is used to
181 gather all package-wide definitions.
183 A file modu.py in the directory pack/ is imported with the statement `import
184 pack.modu`. This statement will look for an __init__.py file in `pack`, execute
185 all of its top-level statements. Then it will look for a file `pack/modu.py` and
186 execute all of its top-level statements. After these operations, any variable,
187 function, or class defined in modu.py is available in the pack.modu namespace.
189 A commonly seen issue is to add too much code to __init__.py
190 files. When the project complexity grows, there may be sub-packages and
191 sub-sub-packages in a deep directory structure, and then, importing a single item
192 from a sub-sub-package will require executing all __init__.py files met while
195 Leaving an __init__.py file empty is considered normal and even a good practice,
196 if the package's modules and sub-packages do not need to share any code.
198 Lastly, a convenient syntax is available for importing deeply nested packages:
199 `import very.deep.module as mod`. This allows you to use `mod` in place of the verbose
200 repetition of `very.deep.module`.
202 Object-oriented programming
203 ---------------------------
205 Python is sometimes described as an object-oriented programming language. This
206 can be somewhat misleading and needs to be clarified.
208 In Python, everything is an object, and can be handled as such. This is what is
209 meant when we say that, for example, functions are first-class objects.
210 Functions, classes, strings, and even types are objects in Python: like any
211 objects, they have a type, they can be passed as function arguments, they may
212 have methods and properties. In this understanding, Python is an
213 object-oriented language.
215 However, unlike Java, Python does not impose object-oriented programming as the
216 main programming paradigm. It is perfectly viable for a Python project to not
217 be object-oriented, i.e. to use no or very few class definitions, class
218 inheritance, or any other mechanisms that are specific to object-oriented
221 Moreover, as seen in the modules_ section, the way Python handles modules and
222 namespaces gives the developer a natural way to ensure the
223 encapsulation and separation of abstraction layers, both being the most common
224 reasons to use object-orientation. Therefore, Python programmers have more
225 latitude to not use object-orientation, when it is not required by the business
228 There are some reasons to avoid unnecessary object-orientation. Defining
229 custom classes is useful when we want to glue together some state and some
230 functionality. The problem, as pointed out by the discussions about functional
231 programming, comes from the "state" part of the equation.
233 In some architectures, typically web applications, multiple instances of Python
234 processes are spawned to respond to external requests that can
235 happen at the same time. In this case, holding some state into instantiated
236 objects, which means keeping some static information about the world, is prone
237 to concurrency problems or race-conditions. Sometimes, between the initialization of
238 the state of an object (usually done with the __init__() method) and the actual use
239 of the object state through one of its methods, the world may have changed, and
240 the retained state may be outdated. For example, a request may load an item in
241 memory and mark it as read by a user. If another request requires the deletion
242 of this item at the same, it may happen that the deletion actually occurs after
243 the first process loaded the item, and then we have to mark as read a deleted
246 This and other issues led to the idea that using stateless functions is a
247 better programming paradigm.
249 Another way to say the same thing is to suggest using functions and procedures
250 with as few implicit contexts and side-effects as possible. A function's
251 implicit context is made up of any of the global variables or items in the persistence layer
252 that are accessed from within the function. Side-effects are the changes that a function makes
253 to it's implicit context. If a function saves or deletes data in a global variable or
254 in the persistence layer, it is said to have a side-effect.
256 Carefully isolating functions with context and side-effects from functions with
257 logic (called pure functions) allow the following benefits:
259 - Pure functions are deterministic: given a fixed input,
260 the output will always be the same.
262 - Pure functions are much easier to change or replace if they need to
263 be refactored or optimized.
265 - Pure functions are easier to test with unit-tests: There is less
266 need for complex context setup and data cleaning afterwards.
268 - Pure functions are easier to manipulate, decorate_, and pass-around.
270 In summary, pure functions, without any context or side-effects, are more
271 efficient building blocks than classes and objects for some architectures.
273 Obviously, object-orientation is useful and even necessary in many cases, for
274 example when developing graphical desktop applications or games, where the
275 things that are manipulated (windows, buttons, avatars, vehicles) have a
276 relatively long life of their own in the computer's memory.
282 The Python language provides a simple yet powerful syntax called 'decorators'.
283 A decorator is a function or a class that wraps (or decorate) a function
284 or a method. The 'decorated' function or method will replace the original
285 'undecorated' function or method. Because functions are first-class objects
286 in Python, it can be done 'manually', but using the @decorator syntax is
287 clearer and thus preferred.
289 .. code-block:: python
298 foo = decorator(foo) # Manually decorate
305 This mechanism is useful for separating concerns and avoiding
306 external un-related logic 'polluting' the core logic of the function
307 or method. A good example of a piece of functionality that is better handled
308 with decoration is memoization or caching: you want to store the results of an
309 expensive function in a table and use them directly instead of recomputing
310 them when they have already been computed. This is clearly not part
311 of the function logic.
316 Python is said to be dynamically typed, which means that variables
317 do not have a fixed type. In fact, in Python, variables are very
318 different from what they are in many other languages, specifically
319 strongly-typed languages. Variables are not a segment of the computer's
320 memory where some value is written, they are 'tags' or 'names' pointing
321 to objects. It is therefore possible for the variable 'a' to be set to
322 the value 1, then to the value 'a string', then to a function.
324 The dynamic typing of Python is often considered to be a weakness, and indeed
325 it can lead to complexities and hard-to-debug code. Something
326 named 'a' can be set to many different things, and the developer or the
327 maintainer needs to track this name in the code to make sure it has not
328 been set to a completely unrelated object.
330 Some guidelines help to avoid this issue:
332 - Avoid using variables for different things.
336 .. code-block:: python
345 .. code-block:: python
352 Using short functions or methods helps reduce the risk
353 of using the same name for two unrelated things.
355 It is better to use different names even for things that are related,
356 when they have a different type:
360 .. code-block:: python
362 items = 'a b c d' # This is a string...
363 items = items.split(' ') # ...becoming a list
364 items = set(items) # ...and then a set
366 There is no efficiency gain when reusing names: the assignments
367 will have to create new objects anyway. However, when the complexity
368 grows and each assignment is separated by other lines of code, including
369 'if' branches and loops, it becomes harder to ascertain what a given
372 Some coding practices, like functional programming, recommend never reassigning a variable.
373 In Java this is done with the `final` keyword. Python does not have a `final` keyword
374 and it would be against its philosophy anyway. However, it may be a good
375 discipline to avoid assigning to a variable more than once, and it helps
376 in grasping the concept of mutable and immutable types.
378 Mutable and immutable types
379 ---------------------------
381 Python has two kinds of built-in or user-defined types.
383 Mutable types are those that allow in-place modification
384 of the content. Typical mutables are lists and dictionaries:
385 All lists have mutating methods, like append() or pop(), and
386 can be modified in place. The same goes for dictionaries.
388 Immutable types provide no method for changing their content.
389 For instance, the variable x set to the integer 6 has no "increment" method. If you
390 want to compute x + 1, you have to create another integer and give it
393 .. code-block:: python
397 print my_list # [4, 2, 3] <- The same list as changed
400 x = x + 1 # The new x is another object
402 One consequence of this difference in behavior is that mutable
403 types are not "stable", and therefore cannot be used as dictionary
406 Using properly mutable types for things that are mutable in nature
407 and immutable types for things that are fixed in nature
408 helps to clarify the intent of the code.
410 For example, the immutable equivalent of a list is the tuple, created
411 with ``(1, 2)``. This tuple is a pair that cannot be changed in-place,
412 and can be used as a key for a dictionary.
414 One peculiarity of Python that can surprise beginners is that
415 strings are immutable. This means that when constructing a string from
416 its parts, it is much more efficient to accumulate the parts in a list,
417 which is mutable, and then glue ('join') the parts together when the
418 full string is needed. One thing to notice, however, is that list
419 comprehensions are better and faster than constructing a list in a loop
420 with calls to append().
424 .. code-block:: python
426 # create a concatenated string from 0 to 19 (e.g. "012..1819")
429 nums += str(n) # slow and inefficient
434 .. code-block:: python
436 # create a concatenated string from 0 to 19 (e.g. "012..1819")
440 print "".join(nums) # much more efficient
444 .. code-block:: python
446 # create a concatenated string from 0 to 19 (e.g. "012..1819")
447 print "".join([str(n) for n in range(20)])
449 One final thing to mention about strings is that using join() is not always
450 best. In the instances where you are creating a new string from a pre-determined
451 number of strings, using the addition operator is actually faster, but in cases
452 like above or in cases where you are adding to an existing string, using join()
453 should be your preferred method.
455 .. code-block:: python
460 foobar = foo + bar # This is good
461 foo += 'ooo' # This is bad, instead you should do:
462 foo = ''.join([foo, 'ooo'])
464 Vendorizing Dependencies
465 ------------------------