operator[] i.e. slice returns a shallow copy

https://docs.python.org/2/library/copy.html shows that slicing a list creates a shallow copy.

For a string slice, you can’t modify the object anyway, so you can think of it as a shallow copy or a reference.

Advertisements

c++dtor^python finalizer^java finalizer

This blog has enough posts on c# finalizers. See also AutoCloseable^Closeable #java

python finalizer is a special class method object.__del__(self), invoked when reference count drops to zero, and garbage-collected. As such, it’s not useful for resource management, which is better done with context manager, a popular python idiom and the best-known “protocol” in python.

— Java finalizer is an important QQ topic

https://stackoverflow.com/questions/2506488/when-is-the-finalize-method-called-in-java has some highly voted summaries.

The finalize() method can be at any time after it has become eligible for garbage collection, possibly never.

The finalize() method should only be written for cleanup of (usually non-Java) resources like closing files. [[effJava]] says avoid it.

https://www.infoq.com/articles/Finalize-Exiting-Java compares

  1. c++RAII
  2. java finalize()
  3. java 7 try-with-resources

at run time, function must be predefined ] python #c++

As shown in https://github.com/tiger40490/repo1/blob/py1/py/str/testAbbr_ez.py, you can randomly shuffle your functions bodies (i.e. definitions) as long as … at run time, each function call can resolve to a function already seen by the interpreter, not a function to be defined later in the script.

Easiest way is to call main() after all function bodies. By the time main() runs and uses any of those functions, the interpreter runtime has already parsed the function body.

If you move func8() body to after the call to main() (I didn’t say “the body” of main()), then python will complain that func8 is undefined. It’s as bad as:

print var3; var3=3

(Note there’s no declaration vs definition for python functions.)

C is more strict — at Compile time (not run time), when the compiler encounters a name like func5(), it insists func5() declaration (or definition) is already seen.

To satisfy the compiler, people put function declarations in include files. Function definition obeys ODR.

python: value@ECT+syntax > deep insight

c++ interviews value deep insight more than any language. Java and c# interviews also value them highly, but not python interviews.

Reminder — zoom in and dig deep in c++, java and c# only. Don’t do that in python too much.

Instead of deep insight, accumulate ECT syntax … highly valued in TIMED coding tests.

Use brief blog posts with catchy titles

python has limited treeMap, array

Relevant to some coding questions (like Facebook) that emphasize big-O.

— vector — The default list is a vector, with constant time random access.

Just like std::vector, deleting from beginning is slow, according to https://stackoverflow.com/questions/19441488/efficiency-of-len-and-pop-in-python

— fixed array — https://stackoverflow.com/questions/176011/python-list-vs-array-when-to-use shows array type is rarely needed.

— tree map — not popular

https://github.com/tiger40490/repo1/blob/py1/py/tree/simpleBST.py is my own BST without self-balancing.

python class^module(singleton)

Both modu5 and class2 are based on a idic. Both can contain “system-wide” i.e. global variables, effectively singleton objects.

A module is like a singleton class, without constructor or inheritance.

Global variables and singletons — I figured these out because 2nd time you import a module, the module-level objects aren’t created again! You can safely construct objects at the module level and all of them become global variables. Is this similar to c++ namespace variables?

Calling a method on a class goes through __getattr__ hook. Probably no such thing in a module?

Importing a regular module actually executes the *.py file – no such feature with a class.

Modules (not classes) are hooked into the (non-trivial) import mechanism.

Module functions vs Methods have differences. All module-level methods are like classmethods, so a module can be simpler to use if you want a simple singleton class.

See also
https://learnpythonthehardway.org/book/ex40.html
http://stackoverflow.com/questions/600190/python-choosing-between-modules-and-classes

python RW global var hosted in a module

Context: a module defines a top-level global var VAR1, to be modified by my script. Reading it is relatively easy:

from mod3 import *
print VAR1

Writing is a bit tricky. I’m still looking for best practices.

Solution 1: mod3 to expose a setter setVAR1(value)

Solution 2:
import mod3
mod3.VAR1 = ‘new_value’

Note “from mod3 import * ” doesn’t propagate the new value back to the module. See example below.

::::::::::::::
globalrw.py
::::::::::::::
#!/usr/bin/python -u
from mod3 import *

def main():
  mod3func()
  ''' Line below is required to propagate new value back to mod3
      Also note the semicolon -- to put two statements on one line '''
  import mod3; mod3.VAR1 = 'new value'
  mod3func()
main()
::::::::::::::
mod3.py
::::::::::::::
VAR1='initial value'
def mod3func():
  print 'VAR1 =', VAR1

[11]python mutable^immutable type variables

Page numbers refer to [[automate the boring stuff with python]]
Note this kind of knowledge is interview relevant only in java/c++. low-level knowledge is not appreciated in python interviews… as stated in other blog posts.
List, Dict and user defined classes are mutable. They provide “mutation” methods (and operators) to edit the content in-place.
As explained in https://medium.com/@meghamohan/mutable-and-immutable-side-of-python-c2145cf72747, numbers, strings, and tuples are immutable, in the bitwise sense not the java sense — If a tuple’s 2nd element is a dict, then yes the dict is still mutable. Why? the reference thing..
– A python variable of immutable type contains the object, not a reference (P99). However, I think the immutability removes the difference — the variable might contain a reference and it makes no difference.
– A python variable of   mutable type contains a reference to the object.
Two common use cases:
  1. When we reassign to an existing/loaded variable, we never overwrite the object in-place. Instead, the reference gets reseated, like a java reference variable reassignment. This applies to strings, integers, tuples and all mutable types. In c++ lingo, this is pointer reseat rather than operator=
  2. When we pass a variable into a function, we copy the reference. I think this applies to all types, including integers.
Our intuition is usually good enough, so most working developers don’t bother to understand these /nuances/.

[13]integer immutability,refcount #python (deep)lesson#1

See gc.get_referrers() and sys.getrefcount() …
Do you know — list Objects are mutable; integer Objects are Immutable? In java, c#, c, int object is Mutable.
—-Now look at integer objects, which are Immutable
i=1
id(i)
i=2 # rebinding
i=1 # rebinding back to original object!
id(i)

j=i # ref count = 2 on the shared object

id(1)
id(2)

—-Now look at lists. Do lists hold items by value or reference? More specifically,
– Add() by pbref? yes according to many posts in this blog and also [[python essential ref]]
– Read() by pbref? Yes. pbclone would make myList[0].edit() meaningless.
– qq(mylist[2] = ….) is by reference? Probably yes, otherwise counter-intuitive.

How do you verify? How about id(mylist[0]). Note

i=1.9
id( i )
arr=[]
arr.append(i) # pbref. Ref count = 2
id( arr[0] )
i=2 # Rebinding! not “content edit”. i now points to a new object, so ref count = 1 on the original object! Counter-intuitive to me.
arr[0] = 2.1 # Rebinding, not “content edit”. arr[0] now points to a new object, so ref count = 0 on the original object “1.9”
id( arr[0] )

Q: is it possible to change the value of arr[0]?
A: arr[0] currently points to an int object. We can rebind this pointer to a “9.89” object. However, if you want to change pointee object content at arr[0], then it depends on the pointee object type. For integer objects, content is immutable, so answer is NOWAY.

variable assignment in python, c#, c++ #key difference

Java has the cleanest design —
V) java _primitive_ type assignment is like Spreadsheet cell editing in-place. Saves new value into memory cell directly. Not following-the-pointer. (no pointer at all)
R) java _reference_ type assignment re-seats/rebinds a pointer with reference-counting — one increment + one decrement

Python assignment is always(?) R ? . I have not seen a “primitive” or “nonref” variable in python.

C# value type assignment is always V, even if the object is a big struct — eg 64 [[c#precisely]]

C++ is similar to java —
– Nonref assignment follows V, though classes often customize assignment operator.
– Pointer assignment follows R.
– However, For a reference type variable, initialization is R, but subsequent assignment is V, since a reference never rebinds.

[13] mutable data types: python ilt java #dictKey

As illustrated in python object^variable, Type, Value, immutable, initialize.. and P29 [[python essential ref]], python simple int Age variable is implicitly a pointer to a reference-counted, copy-on-write, IMmutable pointee object. That begs the question ..

Q: so, how do 2 variables share a Mutable object?
A: Use instance methods like mutablePerson.setAge()???
A: Lists and dictionaries also offer versatile “shared mutable objects”. listofMutables[0] would return a reference to the first element.

I feel these 2 answers cover 80% of the use cases.

In summary,
– “scalar” or primitive type Variables point to immutable Objects. Example — Everyday strings and numbers
– most “composite” object are mutable, such as dict, list and user-defined objects
– between the 2 well-understood categories, there exist some special data types
** tuples are composite but immutable
** method objects?
** class objects?
** modules?

Incidentally, (in a bold departure from java/c#[2]) only immutables (string, tuple,”myInt” variables) can be dictionary keys[1]. Lists, dictionaries and most user-defined objects are Mutable therefore disqualified. A “fake” immutable tuple of list also disqualifies — just try it. In real project, we only use strings and ints as keys.

[1] the underlying Object must be immutable, even though the variables can re-bind.
[2] c# expects but does not require keys to be immutable

[13]immutability in python – fundamentals

Rule: immutability is on objects, not on variables. Actually, same as other languages.

Rule: An object’s mutability is determined by its type; for instance, numbers, strings and tuples are immutable, while dictionaries and lists are mutable.

Examples of containers are tuples, lists and dictionaries. The value of an immutable container object (like a tuple instance) that contains a reference to a mutable object can change when the latter’s value is changed; however the container is still considered immutable, because the collection of objects it contains cannot be changed. So, immutability is not strictly the same as having an unchangeable content, it is more subtle. See http://docs.python.org/2/reference/datamodel.html

In most cases, when we talk about the value of a container, we imply the values, not the identities of the contained objects; however, when we talk about the mutability of a container, only the identities of the immediately contained objects are implied.

I think for a custom MyClass that HAS-A object field myField, MyClass immutability means the content of myField is unchangeable. That’s the java concept of immutability.

python object^variable, Type, Value, immutable, initialize..

[[Python essential reference]] points out that an Object [1] has an address (immutable), a value and a Type, which is the so-called run-time type. An object is always a run-time thing and doesn’t have any “declared type” per-se.

Now contrast a variable, which is 1st) a compile-time thing, and 2nd) a run-time thing when bound to an Object. Think of a variable as a nickname for an object otherwise identified by address. Same nickname can later to given to a different object — variable re-assigning/re-binding. See also post on rooted vs re-bindable variables.

C makes a distinction between ptr variables holding address of an object vs a nonref variable.

In most languages, a variable has a name (immutable), Type (immutable), and after initialization a Value (See footnote [3] for a concrete example.) —

* The type of a variable is the declared type. No such thing as a run-time type because a variable exists in source code only and often disappears after compilation.
* (The type of an object is the run-time type. No such thing as a declared type, because that is a compile-time concept and objects exist only at run-time.)

The “type” concept is non-trivial. Hardware knows bits and not types. Types are created by compilers. Run-time type comes into existence when the run-time instantiates a known type — int, Account etc.

All programming languages rely on types, even dynamic ones. Python/perl variables (not “Objects”) are typeless because the same nickname can re-bind to vastly different objects at run-time. This is one of the fundamental defining features of dynamic languages vs strong-typing languages.

#### Value means ….
# Value (or state) of an object is usually mutable.
# Value or “Binding” of a variable is usually changeable, but there are important exceptions [2]
## when we say a variable myAge is immutable, we mean the object behind the “nickname” is immutable. Mutability traditionally refers to object state.

++++ initialization means …
+ object initialization means changing the bits in the storage-location from some random uninitialized values
+ variable initialization means Assigning a “nickname” to some object. Until initialized, the nickname is unbound but languages often null-initialize by default.
++ in c++ however, declaring a variable of MyType implicitly instantiates MyType on the stack or (if the variable is a field) inside a host object’s real estate.

—————–
Now we realize these most basic “nouns” are a minefield of confusion. Why bother? Well, they matter when you compare languages and read technical documentations.

====Some concrete illustrations using pythyon:
= python tuple variables are re-assignable, but immutable
= python string variables are re-assignable, but immutable (=java). All string methods and operators are readonly.
= python simple numeric variables are re-assignable but probably immutable. “a=2; id(a); a=3; id(a)” shows the variable re-bound — bold departure from C nonref variables. Effectively, such a numeric Object is an immutable copy-on-write object with a reference count.

[1] defined as a storage-location in [[ARM]], be it 1 byte or 1000 bytes. I don’t like the jave redefinition of Object.
[2] such as c++ references and primitive/value-type/nonref variables like myInt. I’d say the population of everyday variables are 50% re-bindable and 50% rooted, just like male/female populations. This is actually the worst minefield of confusion. In C, all non-pointer variables are rooted and all pointer variables can re-bind. C++ reference variables muddy the water. java/c# followed (java — cleanest rules). Python deviates further.
[3] In the simplest traditional scenario, a C myInt variable is permanently bound (rooted) to a 32-bit mutable object — immutable binding, address, run-time type, declared-type, name, but mutable state.

python remote-debugging, introspection/reflection,

Python — One of python’s top 3 specialties is introspection. Dynamic scripting languages usually excel in this arena.

I believe python/perl offer richer (more comprehensive) introspection and live-debugging than c++/c#/java. The more experienced a programmer gets, the more she is likely to appreciate this feature. However, interviews generally don’t touch on this. Why?

(I guess these may not be well-documented or well known as other features?)

(I guess some interviewers aren’t hardcore.)

Python supports remote debugging too.

what things are #include’d – c++, java, python

Not just utility functions, but also

* global variables like cin.
** think of them as fully initialized OBJECTS (or services), usable out of the box

* classes – as cookie cutters, very different from cookies the OBJECTS above
** and Families of classes

Java (and c#?), in contrast, only put TYPES in included libraries. To include (and then use) utility _OBJECTS_, they have to be presented as public static fields like System.out. These static member objects get created when the TYPE is loaded by the class loader. In contrast, c++ library objects (like cout) are typically part of a  namespace.

Why the difference? I think reason is the so-called “free(standing)” variable outside any type. Java forbids it. Any language permitting it can package it along with types and free functions into a “unit” (package, module, namespace…) to be #included.

Python (and perl) is more like c++.

rooted vs re-bindable variables – c#, c++, java, python

Q: What kinds of variables can re-bind (reseated) to a different object at run-time and what kinds can’t? This understanding is not academic but helps programmers remember ground rules.

—-Python moves further towards rebinding. Even a simple myInt variable can rebind. I feel the fundamental distinction in python world is between immutable vs mutable “Objects” (defined as storage-locations).
* Python Immutables are reference-counted, probably copy-on-write. Therefore variables bound to immutable Objects are reseat-able.
* What python variables are rooted? Well I believe the first element (other elements too) in a tuple is, though the tuple variable itself can rebind.

—-In java, all primitive variables are “rooted”. All reference variables are reseat-able.
+ Assigning to a primitive variable writes into “the ultimate” memory location;
+ Assigning to a reference variable reseats the pointer, without cloning any object.
– There’s a Separate 32-bit storage for every reference variable, distinct from pointee’s storage.[1]
– There’s no separate storage for a primitive variable. Variable name is a nickname of the storage address. Compiler translates variable name into storage address. Run-time access to variable is one-hit. In contrast, Reference variables’ access is 2-hit – following the pointer.

[1] Evidence? See memory layout of any MyClass having a non-primitive field. How much memory (like sizeof(MyClass)) is allocated by new MyClass()?
—-In c#, all Value variables (including structs) are rooted. Assignment clones, including pass-by-value into a method.
All reference variables can be rebound.
—-C is simple and clean
All pointer variables can be re-seated but non-pointer variables are rooted. When a variable is on the LHS, it either rebinds or the Object is “edited”. See post on “Immutable, initialize..” to see the difference between Object vs Variable.
—-C++ feels more complicated.
In C++, all nonref and reference variables are rooted. Assignment writes directly into the object’s “stomach”. Pointers are reseat-able (unless const …)

However, a C++ reference variable (like pointer variable) has a separate 32-bit storage (address hidden) distinct from pointee/referent storage. Some writers say “referent” but I find “pointee” more distinct and less ambiguous.

what kind of entity is "str" in a python script

(Warning — for a novice introspection is overly powerful and complex in dynamic languages like python. We won’t get to the bottom any time soon.)

Everything (except literals and operators) you see in a python script is usually one of

– a variable name
– a reserved keyword like for/while, import, in,
– module name like re, string, os. I think these are actually dictionaries
– a type-friendly builtin function — you can put the identifier into type() and dir() — like round, type.
– a type-unfriendly builtin function — print, ..

* “str” is a very strange animal. It’s a function, but type(str) says it’s a typename
* “print” is a strange animal. You can’t examine it using dir(), type()

Python docs list all builtin functions, but a few won’t “type” as builtin function — str, print, type, tuple…

For this sort of investigation, you need a few instrumentation tools — str(), repr(), type(), dir(). I call these meta-functions. They operate on other functions, variables ….

try
print dir(type)
print dir(str)
print dir(dir)

python tuples aren’t waterproof immutable

–Based on http://www.velocityreviews.com/forums/t339699-are-tuple-really-immutable.html
t = ([1],[2])
# apply the id() function to each item in t
map(id,t)   
[47259925375816, 47259925376392]

t[0].append(0)
t
([1, 0], [2])
map(id,t) # unchanged
[47259925375816, 47259925376392]

So tuple deviates from java immutability, which mandates t[0] returning a clone — essentially copy-on-write.

A tuple is like an ordered club roster written with indelible ink. The members of the club may change jobs, age, salary etc but the roster remains the same: same members, same SSN like python id(), same ranking.

In C++ lingo, the “t” tuple has 2 pointers on its real estate. It qualifies as immutable since the two 32-bit fields remain _bit_wise_constant_. The pointees live outside the tuple’s real estate and are Editable.

python=an object-based lang !! OO lang

By java/c++/c# definitions, Python is not strictly object-oriented (“—–“), but in a special sense, python is more object-based (“++++”) than the big 3.

– a class is a dictionary/hash, just like perl
– no declared type for each variable
– no type checking, less type safety
– no strict ctor
– no private/protected ie no access control
– java and c# disallows python’s freestanding, standalone functions.
– no static members
– unclear separation of instance field and class field.
– subclass ctor doesn’t always call base class ctor to initialize the inherited fields
– a regular object instance can have any undeclared field/method injected into it in broad daylight, bypassing any type check.

I think python is object-based more than object-oriented. Further,
I guess python is more dict-based than object-based. I guess many OO features are implemented using dict. I guess a lot of meta-programming uses the dict.

+ a namespace is accessed as an object (and also a dict)
+ a module is an object (and also a dict)
+ a function is an object (and also a dict with attributes)
+ a method is an object (and also a dict with attributes)
+ a built-in function is an object
+ a type is an object, and perhaps is used more than java reflection is
+ an int, float etc is an object (and probably a dict)

python class !! based@C struct

Java, c# and c++ classes are based on C struct, but Python class is more of a dictionary, or rather … A python class instance is more of a dictionary instance. (Cf javascript)

Telltale sign — You can instantiate a python class and subsequently add fields to it.

Even bolder, an instance myObj of class MyClass can introduce a new method m() that’s completely missing in MyClass. As a result, python run-time dispatch algo will search
1) the instance’s private registry
2) the class’s registry
3) base class’s registry

I think the registry is an internal dictionary (“idic”). As stated in P302 [[Programming Python]], classes are just dictionaries that inherit from other dictionaries; each instance is a new dictionary. All these run-time searches hit the dictionaries.

I feel python introspection is more sophisticated and more comprehensive than c++/c#/java.

Note this applies to method resolution only. Fields? Probably similar but i’m not sure yet.

user-dict^idict ] python #speculations

Never tested in IV and never needed in projects…

There are various terms for the same concept
– registry — generic concept across languages. A class has a registry and an instance has a registry too.
– namespace — has special meaning in python, but some veterans say the various registries are implemented same as a namespace
– idic i.e. internal dict, but exposed for runtime introspection

2 types of dictionaries in python —

A) user dictionary are the well-understood, simple dictionaries you create
B) “idic” meaning “internal dict” is a building block of python objects, (possibly) including user dictionary objects.

A user dict is usually taught as an object with methods and fields. I hope that’s not over-simplification.

A namespace for a module (or class or anything) is implemented as an idic. Accessed as whatever.__dict__. In a twist for consistency and coherence, this idic can be *used* as a regular user dictionary. You can do whatever.__dict__.keys(). You may rightly feel that a __dict__ is implemented the same as a user dictionary, but I doubt it. It could be implemented as a complicated symbol table but exposed as a simple dict.

Even if a __dict__ (i.e. an idic) is implemented the same way as a user dict, I strongly believe beginners better treat them as 2 distinct types unrelated to each other, though the syntax looks similar.

Internal dict is the basis of python objects[3].  I guess a idic is more like a small expandable hash table [2]. Must be small because a *concrete* idic instance exists in EVERY parent class, subclass, and EVERY instance[1] of EVERY class.

[1] It’s not too wrong to say it resembles the C struct.
[2] a perl object IS nothing but a hashtable or associative array. See the camel book. In php, an object is implemented as an associative array (or simply an “array”).
[3] at least user defined objects

python operator are wrapper over method

Java/c# operators are the basis of (and faster than) methods, but Python turns the stack upside down — some of the most common, workhorse operators are based on special methods —

dot —- __getattr__()
[key] i.e. key access —- __getitem__()
for-loop —- depends on more than one special method
the plus sign (“+”) —— __add__()
assignment? See other post

In terms of run-time performance, these efficient-looking operators are just as slow as a method call.

Q: I guess in python even the basic “+” cannot be a plain vanilla addition for simple types like integer. Why?

%%A: the basic addition operator should/ought-to change object state, like editing a spreadsheet cell in-place, but python integer addition creates a new Integer object. Therefore, I guess the Integer class must customize __add__() method

%%A: variable of a java primitive int type, c# simple int type or c++ nonref int type (not a ptr or reference) can have the basic addition operator, but python has no nonref type variable. All python int variables are pointers to Integer instances.

python introspection – speculation+observations

Background — Python’s 2 fundamental features, introspection and polymorphism, appear to be integrated (more than in other languages). You could add to the mix the “module” feature, but I won’t.

The __dict__ thingy seems to be the common theme (heart of everything python), but I found it hard to prove in code. For now, let’s take [[Programming python]] at face value.

Python introspection offers perhaps the best window into python internals. I guess introspection reveals
– polymorphism, inheritance,
– function call resolution (always run time)
– instance-method/class-method
– bound methods
– user defined functions and classes
– modules and namespaces

Here’s a limited sample of the numerous introspection features in python —
– __dict__
– __methods__, __members__
– dir()
– type()
– isinstance()
– __mro__