q[internal] keyword in C#

Q: is it related to anything in C++?
A: not really, but look at “extern”

Q: C++ supports “protected” as a middle ground between private/public, and C# added internal?
A: also “protected internal”

Internal basically means “internal to this dll”

Advertisements

simple diff class – in pure java, easily embeddable (src)

<![CDATA[ //Based on work by other authors import java.io.DataInputStream; import java.io.FileInputStream; import java.io.IOException; import java.util.Formatter; /** * This single-file self-contained utility consists of 1 public class and 2 * non-public classes (non-CamelCase names). *

* USAGE: diff oldfile newfile *

* This program assumes that “oldfile” and “newfile” are text files. The program * writes to stdout a description of the changes which would transform “oldfile” * into “newfile”. *

* The printout is in the form of commands, each followed by a block of text. * The text is delimited by the commands, which are: *

* DELETE AT n ..deleted lines *

* INSERT BEFORE n ..inserted lines *

* n MOVED TO BEFORE n ..moved lines *

* n CHANGED FROM ..old lines CHANGED TO ..newer lines *

* The line numbers all refer to the lines of the oldfile, as they are numbered * before any commands are applied. The text lines are printed as-is, without * indentation or prefixing. The commands are printed in upper case, with a * prefix of “>>>>”, so that they will stand out. Other schemes may be * preferred. *

* Files which contain more than MAXLINECOUNT lines cannot be processed. This * can be fixed by changing “symbol” to a Vector. The algorithm is taken from * Communications of the ACM, Apr78 (21, 4, 264-), * “A Technique for Isolating Differences Between Files.” Ignoring I/O, and * ignoring the symbol table, it should take O(N) time. This implementation * takes fixed space, plus O(U) space for the symbol table (where U is the * number of unique lines). Methods exist to change the fixed space to O(N) * space. *

* Note that this is not the only interesting file-difference algorithm. In * general, different algorithms draw different conclusions about the changes * that have been made to the oldfile. This algorithm is sometimes “more right”, * particularly since it does not consider a block move to be an insertion and a * (separate) deletion. However, on some files it will be “less right”. This is * a consequence of the fact that files may contain many identical lines * (particularly if they are program source). Each algorithm resolves the * ambiguity in its own way, and the resolution is never guaranteed to be * “right”. However, it is often excellent. This program is intended to be * pedagogic. Specifically, this program was the basis of the Literate * Programming column which appeared in the Communications of the ACM (CACM), in * the June 1989 issue (32, 6, 740-755). *

* By “pedagogic”, I do not mean that the program is gracefully worded, or that * it showcases language features or its algorithm. I also do not mean that it * is highly accessible to beginners, or that it is intended to be read in full, * or in a particular order. Rather, this program is an example of one * professional’s style of keeping things organized and maintainable. *

* The program would be better if the “print” variables were wrapped into a * struct. In general, grouping related variables in this way improves * documentation, and adds the ability to pass the group in argument lists. This * program is a de-engineered version of a program which uses less memory and * less time. The article points out that the “symbol” arrays can be implemented * as arrays of pointers to arrays, with dynamic allocation of the subarrays. * (In C, macros are very useful for hiding the two-level accesses.) In Java, a * Vector would be used. This allows an extremely large value for MAXLINECOUNT, * without dedicating fixed arrays. (The “other” array can be allocated after * the input phase, when the exact sizes are known.) The only slow piece of code * is the “strcmp” in the tree descent: it can be speeded up by keeping a hash * in the tree node, and only using “strcmp” when two hashes happen to be equal. * * @author Ian F. Darwin, Java version * @author D. C. Lindsay, C version (1982-1987) * @author minor adjustment by Bin Tan */ public class Diff { /* The following are global to printout’s subsidiary routines */ // enum{ idle, delete, insert, movenew, moveold, // same, change } printstatus; public static final int idle = 0, delete = 1, insert = 2, movenew = 3, moveold = 4, same = 5, change = 6; private static final String LINE = “————————–“; /** * block len > any possible real block len */ final static int UNREAL = Integer.MAX_VALUE; /** * main – entry point when used standalone. NOTE: no routines return error * codes or throw any local exceptions. Instead, any routine may complain to * stderr and then quit with error to the system. */ public static void main(String argstrings[]) { argstrings = new String[] { “C:\\0\\Copy of c#IV_question.txt”, “C:\\0\\c#IV_question.txt” }; if (argstrings.length != 2) { throw new RuntimeException(“Usage: diff oldfile newfile”); } Diff d = new Diff(); Formatter f = d.doDiff(argstrings[0], argstrings[1]); System.out.println(f); return; } boolean anyprinted; /** * blocklen is the info about found blocks. It will be set to 0, except at * the line#s where blocks start in the old file. At these places it will be * set to the # of lines in the block. During printout , this # will be * reset to -1 if the block is printed as a MOVE block (because the printout * phase will encounter the block twice, but must only print it once.) The * array declarations are to MAXLINECOUNT+2 so that we can have two extra * lines (pseudolines) at line# 0 and line# MAXLINECOUNT+1 (or less). */ int blocklen[]; final public Formatter formatter = new Formatter(new StringBuilder()); /** * Keeps track of information about file1 and file2 */ fileInfo oldinfo, newinfo; int printstatus, printoldline, printnewline; // line numbers in old & new file public Diff() { } /** * Do one file comparison. Called with both filenames. */ public Formatter doDiff(String oldFile, String newFile) { println(“>>>> Difference of file \”” + oldFile + “\” and file \”” + newFile + “\”.\n”); oldinfo = new fileInfo(oldFile); newinfo = new fileInfo(newFile); /* we don’t process until we know both files really do exist. */ try { inputscan(oldinfo); inputscan(newinfo); } catch (IOException e) { System.err.println(“Read error: ” + e); } /* * Now that we’ve read all the lines, allocate some arrays. */ blocklen = new int[(oldinfo.maxLine > newinfo.maxLine ? oldinfo.maxLine : newinfo.maxLine) + 2]; oldinfo.alloc(); newinfo.alloc(); /* Now do the work, and print the results. */ transform(); return printout(); } /** * inputscan Reads the file specified by pinfo.file. ——— Places the * lines of that file in the symbol table. Sets pinfo.maxLine to the number * of lines found. */ @SuppressWarnings(“deprecation”) void inputscan(fileInfo pinfo) throws IOException { String linebuffer; pinfo.maxLine = 0; while ((linebuffer = pinfo.file.readLine()) != null) { storeline(linebuffer, pinfo); // System.out.println(linebuffer); } } /* * newconsume Part of printout. Have run out of old file. Print the rest of * the new file, as inserts and/or moves. */ void newconsume() { for (;;) { if (printnewline > newinfo.maxLine) break; /* end of file */ if (newinfo.other[printnewline] oldinfo.maxLine) break; /* end of file */ printnewline = oldinfo.other[printoldline]; if (printnewline < 0) showdelete(); else if (blocklen[printoldline] oldinfo.maxLine) { newconsume(); break; } if (printnewline > newinfo.maxLine) { oldconsume(); break; } if (newinfo.other[printnewline] < 0) { if (oldinfo.other[printoldline] < 0) this.showchange(); else showinsert(); } else if (oldinfo.other[printoldline] < 0) showdelete(); else if (blocklen[printoldline] >>> End of differences.”); else println(“>>>> Files are identical.”); return formatter; } /* * scanafter Expects both files in symtab, and oldinfo and newinfo valid. * Expects the “other” arrays contain positive #s to indicate lines that are * unique in both files. For each such pair of places, scans past in each * file. Contiguous groups of lines that match non-uniquely are taken to be * good-enough matches, and so marked in “other”. Assumes each other[0] is * 0. */ void scanafter() { int oldline, newline; for (newline = 0; newline = 0) { /* is unique in old & new */ for (;;) { /* scan after there in both files */ if (++oldline > oldinfo.maxLine) break; if (oldinfo.other[oldline] >= 0) break; if (++newline > newinfo.maxLine) break; if (newinfo.other[newline] >= 0) break; // oldline & newline exist, and aren’t already matched if (newinfo.symbol[newline] != oldinfo.symbol[oldline]) break; // not same newinfo.other[newline] = oldline; // record a match oldinfo.other[oldline] = newline; } } } } /** * scanbefore As scanafter, except scans towards file fronts. Assumes the * off-end lines have been marked as a match. */ void scanbefore() { int oldline, newline; for (newline = newinfo.maxLine + 1; newline > 0; newline–) { oldline = newinfo.other[newline]; if (oldline >= 0) { /* unique in each */ for (;;) { if (–oldline = 0) break; if (–newline = 0) break; // oldline and newline exist, and aren’t marked yet if (newinfo.symbol[newline] != oldinfo.symbol[oldline]) break; // not same newinfo.other[newline] = oldline; // record a match oldinfo.other[oldline] = newline; } } } } /** * scanblocks – Finds the beginnings and lengths of blocks of matches. Sets * the blocklen array (see definition). Expects oldinfo valid. */ void scanblocks() { int oldline, newline; int oldfront = 0; // line# of front of a block in old, or 0 int newlast = -1; // newline’s value during prev. iteration for (oldline = 1; oldline <= oldinfo.maxLine; oldline++) blocklen[oldline] = 0; blocklen[oldinfo.maxLine + 1] = UNREAL; // starts a mythical blk for (oldline = 1; oldline <= oldinfo.maxLine; oldline++) { newline = oldinfo.other[oldline]; if (newline < 0) oldfront = 0; /* no match: not in block */ else { /* match. */ if (oldfront == 0) oldfront = oldline; if (newline != (newlast + 1)) oldfront = oldline; ++blocklen[oldfront]; } newlast = newline; } } /* * scanunique Scans for lines which are used exactly once in each file. * Expects both files in symtab, and oldinfo and newinfo valid. The * appropriate "other" array entries are set to the line# in the other file. * Claims pseudo-lines at 0 and XXXinfo.maxLine+1 are unique. */ void scanunique() { int oldline, newline; node psymbol; for (newline = 1; newline >>> ” + printoldline + ” CHANGED FROM”); printstatus = change; oldinfo.symbol[printoldline].showSymbol(); anyprinted = true; printoldline++; } /** * showdelete Part of printout. Expects printoldline is at a deletion. */ void showdelete() { if (printstatus != delete) println(“\n>>>> DELETE AT ” + printoldline); printstatus = delete; oldinfo.symbol[printoldline].showSymbol(); anyprinted = true; printoldline++; } /* * showinsert Part of printout. Expects printnewline is at an insertion. */ void showinsert() { if (printstatus == change) println(LINE + “\n>>>> CHANGED TO”); else if (printstatus != insert) println(“\n>>>> INSERT BEFORE ” + printoldline); printstatus = insert; newinfo.symbol[printnewline].showSymbol(); anyprinted = true; printnewline++; } /** * showmove Part of printout. Expects printoldline, printnewline at start of * two different blocks ( a move was done). */ void showmove() { int oldblock = blocklen[printoldline]; int newother = newinfo.other[printnewline]; int newblock = blocklen[newother]; if (newblock = newblock) { // assume new’s blk moved. blocklen[newother] = -1; // stamp block as “printed”. println(“>>>> ” + newother + ” THRU ” + (newother + newblock – 1) + ” MOVED TO BEFORE ” + printoldline); for (; newblock > 0; newblock–, printnewline++) newinfo.symbol[printnewline].showSymbol(); anyprinted = true; printstatus = idle; } else /* assume old’s block moved */ skipold(); /* target line# not known, display later */ } /** * showsame Part of printout. Expects printnewline and printoldline at start * of two blocks that aren’t to be displayed. */ void showsame() { int count; printstatus = idle; if (newinfo.other[printnewline] != printoldline) { throw new RuntimeException(“BUG IN LINE REFERENCING”); } count = blocklen[printoldline]; printoldline += count; printnewline += count; } /** * skipnew Part of printout. Expects printnewline is at start of a new block * that has already been announced as a move. Skips over the new block. */ void skipnew() { int oldline; printstatus = idle; for (;;) { if (++printnewline > newinfo.maxLine) break; /* end of file */ oldline = newinfo.other[printnewline]; if (oldline oldinfo.maxLine) break; /* end of file */ if (oldinfo.other[printoldline] fileInfo.MAXLINECOUNT) { throw new RuntimeException(“MAXLINECOUNT exceeded, must stop.”); } pinfo.symbol[linenum] = node.addSymbol(linebuffer, pinfo == oldinfo, linenum, this.formatter); } /* * transform Analyzes the file differences and leaves its findings in the * global arrays oldinfo.other, newinfo.other, and blocklen. Expects both * files in symtab. Expects valid “maxLine” and “symbol” in oldinfo and * newinfo. */ void transform() { int oldline, newline; int oldmax = oldinfo.maxLine + 2; /* Count pseudolines at */ int newmax = newinfo.maxLine + 2; /* ..front and rear of file */ for (oldline = 0; oldline < oldmax; oldline++) oldinfo.other[oldline] = -1; for (newline = 0; newline < newmax; newline++) newinfo.other[newline] = -1; scanunique(); /* scan for lines used once in both files */ scanafter(); /* scan past sure-matches for non-unique blocks */ scanbefore(); /* scan backwards from sure-matches */ scanblocks(); /* find the fronts and lengths of blocks */ } }; // end of main class! /** * This is the info kept per-file. */ class fileInfo { static final int MAXLINECOUNT = 20000; DataInputStream file; /* File handle that is open for read. */ public int maxLine; /* After input done, # lines in file. */ int other[]; /* Map of line# to line# in other file */ node symbol[]; /* The symtab handle of each line. */ /* ( -1 means don't-know ). */ /* Allocated AFTER the lines are read. */ /** * Normal constructor with one filename; file is opened and saved. */ fileInfo(String filename) { symbol = new node[MAXLINECOUNT + 2]; other = null; // allocated later! try { file = new DataInputStream(new FileInputStream(filename)); } catch (IOException e) { System.err.println("Diff can't read file " + filename); System.err.println("Error Exception was:" + e); throw new RuntimeException(e); } } // This is done late, to be same size as # lines in input file. void alloc() { other = new int[symbol.length + 2]; } } /** * Class "node". The symbol table routines in this class all understand the * symbol table format, which is a binary tree. The methods are: addSymbol, * symbolIsUnique, showSymbol. */ class node { /* the tree is made up of these nodes */ static final int freshnode = 0, oldonce = 1, newonce = 2, bothonce = 3, other = 4; static node panchor = null; /* symtab is a tree hung from this */ /** * addSymbol(String pline) – Saves line into the symbol table. Returns a * handle to the symtab entry for that unique line. If inoldfile nonzero, * then linenum is remembered. */ static node addSymbol(String pline, boolean inoldfile, int linenum, Formatter formatter) { node pnode; pnode = matchsymbol(pline, formatter); /* find the node in the tree */ if (pnode.linestate == freshnode) { pnode.linestate = inoldfile ? oldonce : newonce; } else { if ((pnode.linestate == oldonce && !inoldfile) || (pnode.linestate == newonce && inoldfile)) pnode.linestate = bothonce; else pnode.linestate = other; } if (inoldfile) pnode.linenum = linenum; return pnode; } /** * matchsymbol Searches tree for a match to the line. * * @param pline * pline, a line of text If node's linestate == freshnode, then * created the node. * @param formatter */ static node matchsymbol(String pline, Formatter formatter) { int comparison; node pnode = panchor; if (panchor == null) return panchor = new node(pline, formatter); for (;;) { comparison = pnode.line.compareTo(pline); if (comparison == 0) return pnode; /* found */ if (comparison 0) { if (pnode.pright == null) { pnode.pright = new node(pline, formatter); return pnode.pright; } pnode = pnode.pright; } } /* NOTE: There are return stmts, so control does not get here. */ } final private Formatter formatter; String line; int linenum; int /* enum linestates */linestate; node pleft, pright; /** * Construct a new symbol table node and fill in its fields. * * @param pline * A line of the text file */ node(String pline, Formatter formatter) { this.formatter = formatter; pleft = pright = null; linestate = freshnode; /* linenum field is not always valid */ line = pline; } /** * showSymbol Prints the line to stdout. */ void showSymbol() { this.formatter.format(line+”\n”); // System.out.println(line); } /** * symbolIsUnique Arg is a ptr previously returned by addSymbol. * ————– Returns true if the line was added to the symbol table * exactly once with inoldfile true, and exactly once with inoldfile false. */ boolean symbolIsUnique() { return (linestate == bothonce); } } ]]>

a simple (tricky) sabotage on java debugger

If you rely heavily on a java debugger, beware of this insidious sabotage.

You could use finally blocks. You could surround everything with try/catch(Throwable). If all of these get skipped (rendering your debugger useless) and system silently terminates at inconsistent moments, as if by a divine intervention, then perhaps ….

perhaps you have a silent System.exit() in an obscure thread.

Let me make these pointers clear —
– System.exit() will override/ignore any finally block. What you put in finally blocks will not run in the face of System.exit()
– System.exit() will not trigger catch(Throwable) since no exception is thrown.
– System.exit() in any thread kills the entire JVM.

Q: Is JNI crash similar to System.exit()?
%%A: i think so.

Actually, in any context a silent System.exit() can be hard to track down when you look at the log.

non-dummy-type template parameters

(Note this topic is not related to template Partial specialization)

First, let’s distinguish a template parameter (like “T”) vs a template argument (like a class “Account”). For a NDTTP, the template parameter has a) concrete type and b) a parameter name, typically “size_t value” vs the template argument like “31”.

Simplest example: std::array template has 2 template parameters —

  1. a dummy type T
  2. a non-dummy type “size_t length”

std::array<Account, 31> concretizes the template with a concrete type Account and a value of 31.

Majority of class templates in practice have a dummy type (or multiple, but let’s stay focused) to signify an Unknown-type. For example, the STL vector can be “concretized” in memory to become a real CLASS when given a real type like “Account”. The real type replaces the dummy type. [[C++ primer]] is the first book I read that mentions a very powerful type of class template. I call it NDTTP i.e. non-dummy-type template parameter.

Background — What appears inside angle brackets after the keyword “template” is always a list of “tokens” or “thingies”. In most cases in practice, each thingy is a “typename T” or “class C”.

Now, the thingy can also be something like “int size”. This is known as a non-type template parameter. I call it a non-dummy-type template parameter, or NDT template parameter. a NDT is not a dummy type, but a real type with a parameter name. NDT declaration syntax is like a regular function parameter. NDT represents a dummy type pinned to a real type.  But how is NDT used when the push comes to the shove i.e. template instantiation (I prefer “template concretization”)?

Remember a real type argument like “AccountAllocator” in STL vector is a piece of information at the class level, not Instance level. When the vector is concretized in memory to become a real/concrete Class, the real class uses AccountAllocator Class. In the same vein, the NDT value is at class-level, not class-instance level nor at template level. That implies the “int size” value of 55 is a constant for the class, across all class instances.

In other words, when we convert a GENERIC “unconcretized” template matrix_double into a real class, the “int size” template parameter (NDT template parameter) is replaced by a value like 55, and treated as a class-level static constant. If we construct 9999 instances of the concrete matrix_double class, all of them share the same size=55.

A class template can have all its “tokens” be dummy types (standard practice), or all of them NDT (P861 [[c++primer]]), or a mixture.

See other posts about When the NDT arg is a specific func ptr like “revalue”. In contrast, given a functor class, you use a regular type param (more common), not a NDTTP.

heap usage: C ilt C++/Java/C#

At the application level (as opposed to libraries), I personally feel C apps tend to do most of their work on the stack whereas c++ apps uses more heap.

Q: what are the classic C usages for heap? I feel most requirements are met by “auto” and global variables, including large N-dimensional arrays of structures. A big structure can in turn hold arrays/pointers (well, array and pointers are almost indistinguishable.)

A: linked graph data structures.

C++ added a lot of support for heap — including all the new-expressions and various operator-new (not to mention deletes).

– C++ new-expression ties together heap allocation and class constructor.
– C++ delete-expression ties together heap de-allocation and class destructor.

In C++, class instances are commonly allocated either on stack OR on heap. Java/C# is even more heap-oriented. Why?

add a python script to context menu (windows explorer)

based on http://zephyrfalcon.org/weblog/arch_d7_2003_08_09.html#e306

Look for HKEY_CLASSES_ROOT.Folder to add to folder-level right-click context menu. (HKEY_CLASSES_ROOT.* i.e. the _asterisk_ for file-level context menu)

In your python script,

       dirName, baseName = os.path.split(sys.argv[1])
       print sys.argv
       print dirName
       print baseName

simple script to count classes defined in a python project

Any time you have a sizeable python project with many *.py source files, you can use this script to count how many classes defined.

import re, sys
from os import walk

printed={}
for (path, dirs, files) in walk(“c:\py”) :
       for filename in files :
               if not re.search(“.py$”,filename) : continue
               if not printed.has_key(path):
                       print ” path = ” + path
                       printed[path] = True

               for line in open (path+'\'+filename) :
                       #if re.search('^s*def ', line) : print line,
                       #if re.search('^s*classs', line) : print filename + ':t' + line,
                       if re.search('^s*trys', line) : print filename + ':t' + line,
                       #if re.search('@', line) : print filename + ':t' + line,

stack frame has a pointer to the caller stack frame

Q: what data occupy the real estate of a single stack frame? Remember a stack frame is a chunk of memory of x bytes and each byte has a purpose.

A: (obviously) any “auto” variable declared (therefore allocated) in the function

A: (obviously) any formal parameter. If a 44-byte Account parameter is passed-by-value, then the 44-bytes are allocated in the stack frame. If by-reference, then only a 4-byte pointer allocated.

A: 4-byte pointer to caller's stack frame. Note that frame also contains a pointer to its own caller. Therefore, the stack frames form a linked list. This pointer is known as a “previous stack top”.

A: 4-byte ReturnAddress. When a function f4() returns, control passes back to the caller function f3(). Now at assembly level the caller function may be a stream of 20 instructions. Our f4() may be invoked on Instruction #8 or whatever. This information is saved in f4() stack frame under “ReturnAddress”. Upon return, this information is put into the “instruction pointer” register inside the CPU.

operands to assembly instructions

I feel most operands are registers. See first operand in example below. That means we must load our 32 bits into the EAX register before the operation.

However, an operand can also refer directly to a memory location.

SUB EAX [0x10050D49]

A third type of operand is a constant. You pass that constant from source code to compiler and it is embedded in the “object file”

sybase rand() could return 0 and 1

“The rand function uses the output of a 32-bit pseudo-random integer generator. The integer is divided by the maximum 32-bit integer to give a double value between 0.0 and 1.0”

There’s a non-zero chance of getting the max integer, which gives 1 when divided by max. Same probability for 0/max which gives 0.

convert (int, rand()*3) can return 0, 1, 2 and 3, with a non-zero probability for 3 and equal distribution among the 0, 1 and 2.The non-zero chance is something like 232 or 2**(-32) in python syntax.

iterator = simple smart ptr

– Iterators are *simple* extensions of raw pointers, whereas
– smart pointers are *grand* extensions of pointers.

They serve different purposes.

If an iterator is implemented as a class (template) then it usually defines
– operator=
– operator==, >=
operator* i.e. dereference
– operator++, —
– operator+, += i.e. long jumps
….
—-However, iterator class (template) won't define any operation unsupported by raw pointers, therefore no member functions! Smart pointers do define public ctor, get() etc

iterator categories, stratification, hint..

Iterator categories are a /stratification/ of the 5 to 10 standard Capabilities of raw pointers in the C language. Stratification — like deposits in a lake.

The dereference-then-Read operation is part of Input-Iterator category, which lacks the dereference-then-Write capability.
The dereference-then-Write operation is part of output Iterator category, which lacks the dereference-then-Read.
The ++ and — operators are available in the bidirectional-iterator category
The += and -= (jumps) and greater-than etc operations belong to the topmost strata — random access iterator category

There are a total of 5 categories, arranged in layers. (Some say “hierarchy” — a bit misleading). [[STL tutorial and reference]] has detailed coverage of this technicality, which is too technical for other books.

Iterator categories are not some kind of inheritance hierarchy.

Iterator categories are not some kind of typedef.
 
Q: So how do you write code to enforce what category of iterator you require?
Answer: no-way. Compiler has absolutely no idea what ForwardIterator category means. A category name is no different from those dummy type names in a template declaration. To the compiler, template<Bidirectional_Iterator,…> is no different from template.
A: you can only “hint“, not enforce, the “kind” of iterator you require.

Q: how does an algorithm's author indicate what type of iterator he needs?
A: (This is what STL authors did) Be creative — use special names to name the dummy types in template. Remember all STL algorithms are function templates. If you look at the specification (not “declaration”) of STL algorithms, the dummy types all come with special names. Don't just rename them with the customary T or S.

You appreciate the categories After you start writing STL-style function templates.

Note constness of iterator is completely unrelated to the 5 categories. No STL algo specification mention const in the dummy type names.

pairing up array (or placement) new/delete

Item 8 of [[more eff C++]] has a good coverage. Here’s my summary.
Q: Which of these does the compiler allow you to invoke directly in your source code?
OPERATOR placement-new? Not sure. Perhaps never needed.
placement-new EXPRESSION? Of course. P40

placement delete? NO SUCH THING
dtor? yes. P42. I also saw it in some C++ FAQ.
operator new? yes P39. Rather similar to how you call malloc()
operator delete? Yes p41

delete[] EXPRESSION? Of course. That’s the standard way to de-allocate an array of “heapy thingies”
OPERATOR delete[]? Not sure.

operator new[] ? Not sure. I feel we can always avoid invoking this explicitly. We can invoke operator-new with a sizeof(targetObject)*arraySize. Both operators return a void pointer to a chunk of raw memory of the correct size.
————Here are the legitimate pairings. “Inter-faith marriage” leads to undefined behaviour ———–
malloc === free
operator new === operator delete // similar to malloc/free, but very rarely needed
new expression === delete expression
new[] expression === delete[] expression
placement-new expression === explicit dtor call followed by some special form of de-allocation. There’s no such thing as placement-delete
————
Q: which of these can we customize?
placement-new EXPRESSION? no
OPERATOR placement new? yes P40
OPERATOR new[] and operator delete[]? Yes P43

rbegin() in STL container class templates

I once told someone that not every container supports rbegin(). Now I know all standard STL container class templates define rbegin(). STL string class also supports rbegin(). By the way, If you specialize vector (or any container) class-template with your own Account class, you end up with a specialized class-template. You must re-implement rbegin() and all public methods. That’s part of the “contract”.

Now, WHO don’t support rbegin()?
– The simple array doesn’t support rbegin() or any method for that matter.
– I think ostream and istream don’t provide rbegin().
– How about hashed containers? They don’t have a rbegin()

What if someday someone creates a useful container that doesn’t support rbegin()? If your algorithm insists on a container defining rbegin(), then this algo can’t be used for that new container.

In conclusion, it’s probably more useful in practice to write utility functions accepting iterator arguments, not container arguments. Follow STL algorithm conventions.

y list iterator can’t jump (+=, -=)

Q: Given the STL list iterator supports increment (++), why not make it support jumps? This way, list iterators can be used with algorithms like sort(), right?

P64 of [[STL tutorial and reference]], written by one of the 3 STL inventors, made it clear —

– It’s not about feasibility — adding jump is feasible but not a great idea.
– It’s all about efficiency. Implementing jumps using increment is “faking it”. The sort() algorithm would become extremely inefficient when given a fake random-access iterator.

avoid notifyAll() in low latency apps

notifyAll() is unpopular in low-latency apps. For fear of lost signal, we ask jvm to do a lot of useless work. You wake up everyone when you just need one of them waken up. But How do we get away with notify()? Here are 2 common conditions that should be satisfied.

* All the threads in your target wait set must be identical, so any one can get up and finish the task.
* All are waiting on the same while() condition.

Imagine some threads are waiting for a blocking queue to become non-empty; other threads are waiting for non-full. If you notify(), the risk is that scheduler chooses one waiting thread, but the chosen one checks the while() condition to see it’s waiting for non-empty, but the signal is no-longer-full, so the signal is ignored and lost. Using jdk 5 conditionVar, you can simplify this kind of situation using multiple conditions bound to the same lock.

divide-by-zero: c++ y no excp; y java throws

https://stackoverflow.com/questions/8208546/in-java-5-0-statement-doesnt-fire-sigfpe-signal-on-my-linux-machine-why explains best.

http://stackoverflow.com/questions/6121623/catching-exception-divide-by-zero — c++ standard says division-by-zero results in undefined behavior (just like deleting Derived via a Base pointer without virtual dtor). Therefore programmer must assume the responsibility to prevent it.

A compliant c++ compiler could generate object code to throw an exception (nice:) or do something else (uh :-() like core dump.

If you are like me you wonder why no exception. Short answer — c++ is a low-level language. Stroustrup said, in “The Design and Evolution of C++” (Addison Wesley, 1994), “low-level events, such as arithmetic overflows and divide by zero, are assumed to be handled by a dedicated lower-level mechanism rather than by exceptions. This enables C++ to match the behavior of other languages when it comes to arithmetic. It also avoids the problems that occur on heavily pipelined architectures where events such as divide by zero are asynchronous.”.

C doesn’t have exceptions and handles division-by-zero with some kind of run time error (http://en.wikibooks.org/wiki/C_Programming/Error_handling). C++ probably inherited that in spirit. However, [[c++primer]] shows you can create your own divideByZero subclass of a base Exception class.

java has no “undefined behavior” and generates an exception instead.

moving a bit up the pyramid in a trading desk

In each trading desk, there are very few decision-making roles and a large number of supporting roles, including IT[1]. Pyramid — The closer to the top, the more valuable you become. For an IT guy, the route to those valuable roles is sometimes non-obvious. For this discussion, i will assume you don't want to become trader. If you also don't want to become an IT manager, then consider a quant role.

There are several factors …

The field must have sufficient profit to fund large “valuation” systems (VaR, quote pricing, marking…). If it's a small desk, then system is less likely to be complex.

In terms of inherent complexity, option > IRS > futures and cash; FI > others.

Risk is more quantitative than pre-trade pricing. Risk includes marking and real-time risk, but the most complex is simulation-based VaR. Business allocate enough time and resources to prepare an elaborate risk risk analysis.

When you first enter a field that's 60% (on an arbitrary scale) quantitative compared to an earlier 40%, you may not notice you are slowly shifting up on the pyramid. But when you develop your quantitative track record over the years you might move further away from IT and closer to business i.e. profit center. Your value-add may increases slowly or signficantly. When you are still very much in IT, you may still feel you are a replaceable supporting staff but someday, without warning, you become part of the “inner circle”.

[1] Decision-making roles including market-risk. If desk head has enough confidence in the market-risk analysis, then risk managers can significantly affect (intervene/curtail/rein-in) trader actions. In some buy-side/sell-side trading desks, the chief does pretty much nothing except watching risk numbers.

c++low latency phone IV #MS-shanghai

I think this is all QQ… i.e theoretical knowledge. Not really harder than the mvea interview. Easier than the CVA interview.

Q: why we should not throw exception from dtor? Why do you say sometimes you can break the rule? See throwing dtor: %%justified use cases
%%A: if i have a utility routine that may throw, and I want to use it in my dtor, we should assess the danger. If we know entire process should crash under this condition, whether this dtor throws exception or swallow it, then it’s fine to use that routine without try/catch. Even without double-exception situation, the default behavior is std::terminate()

Q: divide by zero — is that an exception?
%%A: yes in java but UB in c++
AA: UB in c++

Q: is there something you can do in C but can’t in c++?
%%A: I guess you mean “C code uncompilable under c++ compiler”? I don’t think so.
A: “include <stdio>” won’t compile in g++
%%A: in pure-C environment if you receive a pointer at run time and it’s known to point to heap, you can call free() but in a C++ environment, that pointer may be created by new/array-new so free() is problematic.
A: https://isocpp.org/wiki/faq/big-picture#back-compat-with-c hints that if a C source code has no function prototype, then c++ compiler will complain.

Q: at what threshold of latency requirement would you switch from java to c++?
%%A: 100 μs. but I think nowadays java can rival c++.

Q: singleton — how would you implement
%%A: no friend class; declare but not define copier/op=. Provide a static method getInstance

Q: size of an empty c++ class’s instance
%%A: 1 byte (Correct ! https://stackoverflow.com/questions/6552319/c-sizeof-of-a-class-with-functions)

Q: diff between struct and class
Q: can you implement inheritance and polymorphism in C?
%%A: yes. Most c++ features (until exception) used to be converted to C source code.
Q: what’s complete vs partial specialization of template? (Jargon question)
Q: what happens to stack when an exception is thrown?

Q: what’s your reaction when you see “delete this” in a program?
A: After that, need to be careful not to reference anything in the class, so as to avoid dereferencing a dangling pointer.

 

hide client names and address

I proposed a system to a buy-side asset manager shop. I said client names don't need to stored in the central database. Maybe the salesforce and investment advisors need the names but they don't need to save those in a shared central database for everyone else to see.

Each client is identified by account id, which might include an initial.

When client logs in to a client-facing website, they will not see their name but some kind of relatively public information such as their self-chosen nick name, investment objectives, account balance, and last login time.

Client postal address is needed only for those who opt for paper statement. And only one system needs to access it — the statement printing shop.

A veteran in a similar system told me this is feasible and proposed an enhancement — encrypt sensitive client information.

What are some of the inconveniences in practice?

real world libor curve bootstratp – facts&&figures

* first 3 months use libor deposit rates — O/N, 1w, 2w etc

* 3 month – 3 years use libor futures. For USD, CME lists 3-month futures contracts 10 years out, but liquidity is good for first 2-5 years only.

* 4 years – 50 years use swap rates from tradeweb or bloomberg

Some curves (EUR? GBP? BRL) also use zero-coupon bond prices or year-end adjustments. All 5 types of instruments provide a maturity/rate pair, i.e. one data point on the yield curve space, to be fitted.

Note in this investment bank, US treasury prices are not used. I wonder why. I guess this is strictly a Libor yield curve. Base instrument is ED loans between AAA-rated banks in London, not US treasury bonds. Lida said “risky curve vs riskless curve”

swing propChangeListener: pass info between event generator/callback Methods

background — callback/event-handling/listener is one of the more interesting or tricky part of swing development.

Callback site can use EventObject.getSource() to find the generator Object (for a semantic event), cast to a specific type, then access its specific methods or fields. This requires prior knowledge of the exact data type of the event-generator object. That leads to tight-coupling.

A more generic solution is the PropertyChangeListener.java. This and related classes are part of java.beans package, not limited to swing. PCL is a flexible and practical solution in a wide variety of contexts beyond swing.

PropertyChangeEvent.java holds (in addition to source) propertyKeyName, oldValue and newValue fields.
Any Listener class can implement the PropertyChangeListener.java interface and support a single callback method — propertyChange(PropertyChangeEvent). When the callback runs, at run-time it has access to all 4 constituents of the PropertyChangeEvent instance.

design considerations – my first homemade FIX client

In a typical sell-side / market maker, the FIX gateway is a FIX client for a liquidity venue. The liquidity venue (typically an exchange, but also ECNs, dark pools) runs a massively multi-threaded, FIX server farm, possibly with custom-made FPGA hardware. I believe the technical requirements for an exchange FIX server is different from a sell-side FIX client.

Therefore, I assume my audience are far more interested in the client code than the server code.

My mock server implementation is a mock-up, with no optimization, no resilience/robust design, no fault-tolerance, no select()-based sockets, no high-availability, no checksum validation.

— design priorities —
Since this is a from-scratch implementation of a large spec, I focused on basic functionality. Debugging and instrumentation is important in the early stage. A lot of extra code was created to help developers verify the correct operation of all those nitty gritty details.

I don’t aim to build a complete engine, but the basic functionality I build, i try to build it on solid ground, so hopefully it doesn’t become throw-away code.

At the heart of the OO design was the abstraction expressed by AbstractClient, AbstractState and ClientSession. These form the backbone of the core object graph. The fields and constructors were chosen with care. Cohesion and low-coupling were my ideals but sometimes hard to achieve.

These core classes help meet most of the fundamental requirements related to object state of a client connection, sequence number, state transition, output stream sharing, connection liveness monitoring… I wouldn’t say these are well-design classes, but they are a reasonable first attempt at modelling the problem using java classes.

Reusable components were created whenever possible, such as the message parser, message formatter.

FIX engine are highly configurable. All config parameters are extracted into a config object, which is obtained from a config factory. One config for Test, one for Production or UAT etc.

— features —
– parsing a raw FIX msg into an array, which is faster to access than a hashmap. By our design, we avoid hash lookup completely.
– Destroyer.java is a home-made solution to the potential forever-blocking of readLine(). On some platforms, socket read is not interruptible and the only way to unfreeze a socket read is to close the socket. We do that in this implementation but only after sending a final logout message.
– Various template methods to guarantee close() and guard against resource leak
– AbstractClient has most of the generic logic, while SingleOrderClient implements very small amount of custom logic.
– FIX msg formatter is generic enough to accept a map of tags and values — any tags and values
– sequence num validation
– state machines on both client and (simplified) server. State design pattern.
– server side — very few features, but ConnectionManager is reasonable encapsulation of new socket creation.

—- client-side features to be implemented —-
– heart beat — more robust. Should never fail.
– timeout waiting for ack
– re-logon if no heartbeat for 60 seconds
– checksum validation
– storing exchange order-ack into database
– Right now, given time constraint, i didn’t use nonblocking IO. All FIX system should use non-blocking IO.
———- Disclaimer ————-
I spent about 4 hours on this project, as suggested by Jonathan. I had to do some basic research on FIX since I have never programmed FIX (except some ECN connectivity modules that probably used FIX under the hood).
————— How to run the server/client —————–
Run ServerMain.java, then ClientMain.java.
To see the heart beat monitoring/intervention (Destroyer) in action, put 500 in getSilenceThreshold().

Software arch vs enterprise arch

In many companies architects (are perceived to) operate at a high level of abstraction — high-level system modules/interfaces, high-level concepts, high-level decisions. Their audiences include non-technical users, who speak business language. I guess some architects absolutely avoid low-level tech jargon almost like profanity — it’s all about image of an architect. The wrong word on the wrong occasion would creates a permanent bad impression.

To many people, “architecture” implies some network layer, networking protocol, network nodes. These architects might feel less than comfortable if their system has just one machine or just one jvm instance [1], without messaging, database, remote-procedure-call, web service, distributed caching, IPC, shared memory, shared data format (like XML) or communication protocol.

In contrast, software architects can be extremely low-level, too. Their audience are developers. (Examples below include a lot of c/c++…)

– Consider the creator of STL — an architect?
– connectivity engine team lead?
– FIX engine architect?
– Any low-latency trading engine architect?
– Creator of perl/python/boost regex engines?
– All boost libraries I know are low-level. Are the authors architects?
– Tomcat is a single-jvm system. Authors qualify as architects?
– Hibernate isn’t a big system with big modules.  so the authors are low-level architects?
– JNI component designers — single-VM
– java GC innovators
– Creators of any reusable component like a jar, a dll — Just a pluggable module — not even a standalone “server”. Do they qualify as architects?
– The pluggable look-and-feel framework in Swing is designed by someone — an architect?
– spring framework, before all the add-on modules were added
– the tcp/ip/udp software stack was designed by some architect, but that person probably worked mostly on 1 machine, occasionally on 2 machines for testing only.
– GCC creator
– apache creator
– mediawiki creator
– creators of the dotnet CLR and the crown-jewel of Sun, the JVM.

I feel majority of the really important software [2] used in the world today were created by truly powerful developers who focused on a single module in a single system, without messaging, database, remote-procedure-call, web service, distributed caching, shared data format (like XML) or communication protocol.

Now I think there’s a difference between software architect vs enterprise system architect (ESA). To the software/internet industry, the software architect adds more value, whereas to a non-tech company (like finance/ telecom/logistics…), the ESA is the master mind.

I feel some ESA may need to solve performance problems but not implementation/debugging problems.

ESA often covers network/hardware configuration and design too. Security is often part of the “network piece”. If the database is small, then the ESA often covers that too. Tunin/optimization is cross-discipline and touches all the pieces, but in most of my enterprise systems, hardware/network tuning tends to require fewer specialists than DB or application (including MOM and cache). Therefore, ESA sometimes becomes the optimization lead.

(I recently encountered some concepts, methodologies and paradigms at architect level but not sure where to place them — design patterns, TDD/BDD, closures/lambda, meta-programming, functional programming, aspect-oriented… and other programming paradigms — http://en.wikipedia.org/wiki/Programming_paradigm.)

[2] By the way, these so-called “really important” software products are predominantly in C.
[1] Many of the items listed can add real value in a single process or multiple

##some c++REAL productivity skills seldom covered by IV

pre-processor tricks
debugger — graphical or command line
remote debugging
truss
(quick) navigating manpages
gcc (troubleshooting) command line
make (troubleshooting)
linker (troubleshooting)
core dump analysis — not too tough. Just know the basics
memory leak detection
scripting to automate dev-related processes to minimize mistakes and avoid rerunning something 30 times mannually.
text processing to analyze input/output text files and logs
profilers

——
With debugging c++ or others, I think many times you don’t need in-depth knowledge. Many of these tools have the intelligence to probe the runtime/binary and extract insightful data, often presented in a mountain of ascii text (at least human readable:). You just need to search and search in it.

Often one tool is unable to extract some critical data on one platform but another tool works better. You don’t really need to know in-depth though it often pays to be a bit curious. Just keep trying different tricks until you find a break through.

shared_ptr doesn’t like a pointee on Stack or global memory

Boost documentation says shared_ptr class template stores a pointer to a dynamically allocated object (not an “auto” object), typically allocated with a C++ new-expression (or array-new). The object pointed to is guaranteed to be deleted when…
If you feed it a stack object, then the eventual deletion will probably be undefined behaviour.

Q: Can you finish your task and exit the program before the eventual deletion? Maybe you don’t care?

Anyways, it’s unwise to create a library routine accepting a pointer argument and return a share_ptr. The library routine has no way to test if it points to the heap or the stack (or global area). Such a routine (if you must make one) had better force callers to pass in an explicit flag “isOnHeap”.

As stated in http://stackoverflow.com/questions/7383572/c-shared-ptr-of-stack-object, the function should be designed to receive a shared_ptr as argument, not a raw ptr.

## participants in a swing event ecosystem

Some interviewer once asked me to comment on the multi-threading issues in swing. I said that’s easier to me than the chaos and entanglement passing objects between code modules. Half of that confusion relates to events.
Just to give you a taste. A lot of event listeners are inner classes so a callback method therein has handles on
* all (even private) fields of the enclosing class
* final fields of the objects pass to the listener ctor, which are implicitly saved as hidden fields of the listener INSTANCE,
* fields of the event source object (typically a jcomponent)
* fields of the event object — PropertyChangeEvent for example

—————–
In a typical [1] swing event data flow, there are 3+1 roles
– jcomponents
– listeners — some can be jcomponents
– event objects
– event tasks
[1] in real life, the set-up is often complicated by thread, inner classes etc, so it’s important to look at simple yet realistic examples.
Now, many of these roles are simple only conceptually — actual object-graph is a bit complicated.
Jcomponent is the most Concrete thing among these mysterious creatures. Jcomponents can be an event source, and can optionally implement an listener interface such as ActionListener.java.
Event tasks, as described in another blog post, exists inside the JVM only, and not in java api. Each task points-to exactly one listener instance and one event object.
The listener role is tricky. In my experience, half the listeners are jcomponents, and half are non-jcomponents.
How about Action instance? I feel it is a glue-class for listener and jcomponent. See http://bigblog.tanbin.com/2009/10/abstractaction-brief-intro.html

for YuJia – parameter optimization for the strategy

The SP strategy has 2 parameters to optimize — g and j. Very briefly, we monitor Price(futures) – 10x Price(ETF). When the spread exceeds g (a positive param), then buy the ETF and sell the futures. Remain market-neutral. When the spread drops below j (positive or negative param), we close all position and exit. The goal is to find optimal values of j and g to produce highest return on investment.

The matlab fminsearch and fmincon optimizers could find only local minimum around the initial input values. If our initial values are (j=-1, g=3) then the optimizers would try j values of -0.95, -1.05 and similar values. Even though the optimizer would try dozens of combinations it would not try far-away values like -5 or +5.

Therefore these standard optimizers are limited in their search capabilities. We took inspiration from a sample solution to Homework 3 and came up with our own optimizer — i call it randshock. It’s very fast and fairly stable.

  unbeatenTimesMax=1000;
  yellowJersey = 0;
  unbeatenTimes = 0;
  while (unbeatenTimes<unbeatenTimesMax)
      Target = -getMetric(mu);
      if Target > yellowJersey
        unbeatenTimes=0;
        unbeatenHistory=nan(unbeatenTimesMax,1);
        yellowJersey = Target;
        mu_yj = mu;
        fprintf(‘! n’);
      else
        unbeatenTimes = unbeatenTimes + 1;
        unbeatenHistory(unbeatenTimes) = Target;
        fprintf(‘n’);
      end
      power = 2*log(5)*rand(1,1)-log(5);
      multiplier = exp(power);
      %hist(multiplier,100)
      g = mu_yj(2) * multiplier;
      gap = g * (0.5 + rand);
      j = g – gap;
      mu = [j g];
  end

The entire optimizer is only 20 lines short. Can be easily tested by selecting ENV.MODE_OPTIMIZE. It gives a random “shock” to the input params, and checks out the result, then repeat. If any result is better than ALL previous results, then this result gets the yellow-jersey. We would declare it the final champion only after it survives 1000 random shocks.

The random shock is designed to be big enough to reach far-away values, and sufficiently “local” to cover many many values around the current param values i.e. the current “yellow-jersey”.

Without loss of generality, we will focus on the most important parameter “g”. The shock is implemented as a random multiplier between 0.2 and 5. This means the randshock optimizer has a good chance (1000 tries) to reach a value 80% below the yellow-jersey or 5 times above the yellow-jersey. If all values between these 2 extreme values don’t outperform the yellow jersey, then there’s no point exploring further afield.

Between these 2 extremes, we randomly try 1000 different multipliers. We apply the multipler to the current “g” value. If none of them beats the yellow jersey, then we conclude the optimization.

If any one of the 1000 tries is found to outperform the yellow jersey, then it gets the yellow jersey, and we reset the “unbeatenTimes” counter, and give 1000 random shocks to the new yellow jersey.

In theory this process could continue indefinitely but in practice we always found a champion within 2000 shocks, which takes a few minutes only.

Using the randshock optimizer we found optimal param values, which are the basis of all the analysis on the strategy. Next let us look at the optimal parameter values.

The optimal values are g=6.05 and j= -2.96. ROI = $194k from initial seed capital of $1k. In other words, $1k grew almost 200 times over the InSample period (about 10 years).

Using these two optimized parameter values, the OutSample period (from late 2009 till now) generated a profit of  $7576, from initial seed capital of $1k. The return is not as “fast” as InSample, but still the optimized values are usable and profitable.

binomial trees — flexible and adaptable

Binomial Tree is widely used in investment banks such as GS and Barcap. One of the most popular numeric methods for option pricing. I think the only other common pricing methodology is PDE, but less popular.
Simpler pricers use formula directly. So I’d say binomial tree is the #1 most practical and wide-spread pricing methodology for non-trivial instruments.

All binomial trees I have seen are interlocked. After 54 intervals, there are exactly 55 possible prices. See http://bigblog.tanbin.com/2011/06/option-pricing-recombinant-binomial.html

“Level 55” refers collectively to the 55 tree nodes after 54 intervals. The word “Level” should NOT be used for price-level. “Level” means height of the tree as it grows. Rotate the tree to put the root down to visualize the 55 “Levels”. However, most btrees are drawn left to right because time (X-axis) grows rightward, and price values are naturally arranged along Y-axis.

The interlock/recombinant rule is simplifying, and all intervals are equal on a tree, but many other aspects of the tree are flexible/complex and need to be modeled. It’s these flexibilities that allow the tree to be so versatile and adaptable.

– the probability of an upswing at each node is unique and probably independent. On the 5 nodes at Level 5, there are 5 distinct up-probabilities, and 6 on the next level.
– the magnitude of the upswing at each node is independent. Ditto the downswing. However CRR btree don’t provide this flexibility.

Trinomial tree isn’t popular (perhaps never used) in the finance industry.

each task instance in EventQueue has ptr to 1 listener

EventObject.java doesn’t contain a “handle” [2] on the listener INSTANCE, but look at an event-queue-task (“qtask”) in the Swing EventQueue.

EDT runs all callback methods (such as actionPerformed) sequentially [1]. So if a jtable and a jtree both listen to an event, EDT must execute the 2 callbacks sequentially.

Imagine 2 qtask INSTANCES are in EventQueue. It’s instructional to imagine a qtask having 3 handles
– on that one listener object, sometimes a jcomponent. See http://docs.oracle.com/javase/tutorial/uiswing/events/actionlistener.html
– on the event object, which provides a handle
– on the source object, perhaps a button or table cell

Note in our simple example one source object (button) generates one event (click) at a time. Given the source maintains 2 pointers to 2 listeners, that one event object must be “delivered to” 2 listeners — 1:2 mapping.

Now the English word “deliver-to” is nice but rather vague or (worse) misleading and can permanent damage some learner’s brain. Listener is unlike some jms client process capable of processing event. Instead, Listener is typically
– a stateful object (perhaps a jcomponent) holding a handle on some RESOURCE — DB, bunch of jcomponents..
– exposing a callback like actionPerformed()

So delivery-to means “EDT executing that non-static callback method on the listener instance, passing in the event object”. This is a crucial and non-trivial data-flow worth a hard look and analysis
* Business logic is in … the method source code;
* Data is passed via …. the event object and the event source
* Resources needed to handle the message are in ….. the listener object’s non-static fields

[1] even though a callback method may dispatch the task to a worker thread or back to EDT again – by invokeLater() i.e. packing up the task into a qtask and “enqueue” to the end of the event queue.

[2] basically a reference or pointer, but it’s good to use a generic term here.

type info stored inside instances – c#, c++, java

Java is simple and clean — all non-primitive objects has a pointer to the runtime (not compile time[1]) class object, which lives in the permanent generation. This enables getClass().

C++ is similar. The RTTI info is often stored in the vtbl for each *virtual* class.  This means each java/c++ object needs only 4 bytes of real estate for vptr. (The type info is N bytes of real estate on the vtbl for the entire class.)

[1] Note the class object for the DECLARED type doesn’t need to be stored in the instance. It’s only used by the compiler and not needed at runtime. RTTI stands for “RunTime…”

—-C# is less simple. See P54,104 [[c#precisely]]
– C# reference objects are probably same as java.
– C# struct instance has no “runtime type” and has no real estate carved out for it. A struct instance doesn’t (need to) know it’s own type.
** (C struct instance has no “runtime type” either, since C has no vtbl.)

–Boxing of struct type is trickier. Even though a struct instance has no pointer to the STRUCT type object, a boxed version, as all c# reference objects do, does hold a pointer to the CLASS object.

tcp/udp use a C library, still dominating

History – The socket library was created in the 1980’s and has stood the test of time. Similar resilience is seen in SQL, Unix, and mutex/condition constructs

In the socket programming space, I feel C still dominates, though java is a contender (but i don’t understand why).

Perl and python both provide thin wrappers over the C socket API. (Python’s wrapper is a thin OO wrapper.)

Sockets are rather low level /constructs/ and performance-critical — latency and footprint. OO adds both overheads without adding well-appreciated or much-needed OO nicety. If you need flexibility, consider c++ templates. All modern languages try to encapsulate/wrap the low-level details and present an easier API to upper-layer developers.

Choose one option only between —
AA) If you write tcp/ip code by hand, then you probably don’t need OO wrappers in c#, java or python
BB) If you like high-level OO wrappers, then don’t bother with raw sockets.

My bias is AA, esp. on Wall St. Strong low-level experience always beats (and often compensates for lack of) upper-layer experience. If you have limited time, invest wisely.

I feel one problems with java is, sockets are low-level “friends” of ints and chars, but java collections need auto-boxing. If you write fast java sockets, you must avoid auto-boxing everywhere.

Q: is java socket based on C api?
A: yes http://stackoverflow.com/questions/12064170/java-net-socket-implementation

binary options, briefly

cash-or-nothing binary option and the asset-or-nothing binary option. CON is like a simple $5 bet on Worldcup Final.

I feel Binary options are important for
* interviews
* Often embedded in structured deals

Greeks (not that important) — Since a binary call is the first derivative (mathematical sense) of a vanilla call with respect to strike, the price of a binary call has the same shape as the delta of a vanilla call, and the delta of a binary call has the same shape as the gamma of a vanilla call

Usually European style

Exchange-traded, and also available OTC

Underlier can be FX, index, ETF or single stocks.

In FX, vanilla and Barrier options (knock…) are more popular than Binary options.

##python – convert btw dict/tuple/list and string, briefly

* any dict/tuple/list ==> serialize to string? use repr() or backquote, or str()
** how about deserialize from string? use eval()

list/tuple ==> dict? dict() ctor
list/tuple ==> list? list() ctor
list/tuple ==> tuple? tuple() ctor
frozenset ==> set? set() ctor

— in real projects, half the non-trivial data conversions involves dict —
dict ==> list of keys? myDict.keys()
dict ==> list of values? myDict.values()
dict ==> list of pairs? myDict.items()

random list-of-pairs ==> various “roll-up” dictionaries? See defaultdict(list) and defaultdict(set) on http://docs.python.org/release/2.5.2/lib/defaultdict-examples.html

Tx Monitoring System: distributed cache

I believe I learnt this from an Indian consultant while working in Barcap. Perhaps gigaspace?

Basic function is to host live transaction data in a huge cache and expose them to users. Includes outgoing orders and incoming execution reports. I think market quotes can also be hosted this way.
Consumers are either sync or async :
1) Most common client mode is synchronous call-and-wait. Scenario — consumer can’t proceed without the result.
2) Another common mode is subscription based.
3) A more advanced mode is query-subscription (similar to continuous query), where
– consumer first make a sync call to send a query and get initial result
– then MOM service (known as the “broker”) creates a subscription based on query criteria
– consumer must create a onMsg() type of listener.

Query criteria are formatted in SQL format. In a select A,B.. A actually maps to an object in the cache.

Major challenge — volume. Millions of orders/day, mostly eq, futures and options. Gigabytes of data per day. Each order is 5kB – 10KB. One compression technique is FIX style data-dictionary — Requester and reply systems communicate using canned messages, so network is free of recurring long strings.

All cache updates are MOM-based.

Q: when to use async/sync?

A: Asynchronous query – needed by Live apps – need latest data
A: Synchronous query – reporting apps

python features (broad) actually used inside 2 Wall St ibanks

If a python project takes 1 man-year, then the number of python features used would take 2 man-weeks of learning, because python is an easy-learning language. In such a project you probably won’t need fancy features like threading, decorators… Most of the work would be mundane.

After the initial weeks of exciting learning curve, you may feel bored. (c# took me months/years). You may feel your peers are deepening their expertise in c++ or wpf …

The tech quiz would be mostly on syntax.

========= the features =========
? lambda, closure, apply()
Everyday operations on list/dict/tuple, str, file object
** for-loop, conversion, filter(), map(), aggregation, argument passing,
Filesystem, input/output
Basics of module import (no advanced features needed)
command line processing + env vars
create functions –composite argument passing
config files and environment config like PYTHON_PATH

———a bit advanced and less used———
creating classes? fewer than functions, and usually simple classes only
list comprehension, generator expressions
basic platform-specific issues. Mostly unix
inheritance, overriding
threading
spawn new processes
decorators
* Python calling C, not the other way round. In financial analytics, heavy lifting is probably done in C/C++ [1], so what’s doable in python? I guess the domain models (market models, instrument models, contract models …) are probably expressed in python as graph-aware python objects.

[1] I have a book on option pricing using things like Finite Element Method. It seems to use many c++ language features (template etc) but I’m not sure if java or python can meet the requirements.

## lasting achievements as a techie@@

Backdrop: Technological innovation is fast (less so in commercial banking, i was told) so “lasting” means … maybe 5 to 10 years (rarely longer)?? Few achievements meet this criteria.

See also http://bigblog.tanbin.com/2010/09/accu-and-longevity-compare-with-civil.html, http://bigblog.tanbin.com/2011/02/classical-engineering-field.html

– open source — software have real lasting values. I guess you can learn to Read their source code — usually in C.
– [L] contributions to foundation modules such as VM/CLR, compilers, threading library, GC, STL — real lasting values

What skills contribute to lasting value-add to an organization?

– tuning – DB — needed in many big and small systems
– tuning – low latency systems — more rare
– [L] instrumentation i.e. refactor/design a given (complex) system to make it easy to trace and follow
– [L] ? introspection – tend to be rather powerful in many new languages
– ? interpreting bytecode; decompilers
– ? system security; white hat hacker

[L = I feel most of these skills are low-level]

In theory almost everything can be learned by a young guy in a few years provided they get full (rare!) access to all source code and manage to make sense of it all. However, look at how many bright young people become kernel developers even though so much open source operations systems exist.

python – some performance tips { AndyZhao

1) *.pyc = java class files, pre-compiled. Similar to jdbc prepared statement.

2) PYTHONPATH = classpath. Probably the most important performance tip, according to Andy.

3) Minimize run-time name look-up —

from time import time
from math import sqrt
import math
d=0.0
start = time()
for i in xrange(10000000):
    d += sqrt(i)        # one name look-up for “sqrt”
print time() – start

d=0.0
start = time()
for i in xrange(10000000):
    d += math.sqrt(i)           # 2 name look-up for “math” and “sqrt”
print time() – start

c# namespace vs java packages

All of these have a BASIC job duty — partition the name space hierarchically, just like internet domain names.

But java package also have *additional* duties or values —

Java – access control by package
Java – Default access is package-access (C#/C++ default to private.)
Java – packages define what’s included in a jar
Java – packages define physical directory tree. I feel this is a simple and clean design.

In java, perl, python, c++, c# …, A namespace is imported by a using/import (not “include”) directive. C# inherits the syntax from c++.

templates in c++ standard library

Many (if not most) c++ standard library classes/functions are templates (except those from C). STL is part of it. See also the list in last item in [[more effC++]]

– string
– iostream
– auto_ptr

How about boost? All of the ones I have seen are templates.

boost thread
boost tuple
boost serialize
boost smart pointer
boost regex

python object^variable, Type, Value, immutable, initialize..

[[Python essential reference]] points out that an Object [1] has an address (immutable), a value and a Type, which is the so-called run-time type. An object is always a run-time thing and doesn’t have any “declared type” per-se.

Now contrast a variable, which is 1st) a compile-time thing, and 2nd) a run-time thing when bound to an Object. Think of a variable as a nickname for an object otherwise identified by address. Same nickname can later to given to a different object — variable re-assigning/re-binding. See also post on rooted vs re-bindable variables.

In most languages, a variable has a name (immutable), Type (immutable), and after initialization a Value (See footnote [3] for a concrete example.) —

* The type of a variable is the declared type. No such thing as a run-time type because a variable exists in source code only and often disappears after compilation.
* (The type of an object is the run-time type. No such thing as a declared type, because that is a compile-time concept and objects exist only at run-time.)

The “type” concept is non-trivial. Hardware knows bits and not types. Types are created by compilers. Run-time type comes into existence when the run-time instantiates a known type — int, Account etc.

All programming languages rely on types, even dynamic ones. Python/perl variables (not “Objects”) are typeless because the same nickname can re-bind to vastly different objects at run-time. This is one of the fundamental defining features of dynamic languages vs strong-typing languages.

#### Value means ….
# Value (or state) of an object is usually mutable.
# Value or “Binding” of a variable is usually changeable, but there are important exceptions [2]
## when we say a variable myAge is immutable, we mean the object behind the “nickname” is immutable. Mutability traditionally refers to object state.

++++ initialization means …
+ object initialization means changing the bits in the storage-location from some random uninitialized values
+ variable initialization means Assigning a “nickname” to some object. Until initialized, the nickname is unbound but languages often null-initialize by default.
++ in c++ however, declaring a variable of MyType implicitly instantiates MyType on the stack or (if the variable is a field) inside a host object’s real estate.

—————–
Now we realize these most basic “nouns” are a minefield of confusion. Why bother? Well, they matter when you compare languages and read technical documentations.

====Some concrete illustrations using pythyon:
= python tuple variables are re-assignable, but immutable
= python string variables are re-assignable, but immutable (=java). All string methods and operators are readonly.
= python simple numeric variables are re-assignable but probably immutable. “a=2; id(a); a=3; id(a)” shows the variable re-bound — bold departure from C nonref variables. Effectively, such a numeric Object is an immutable copy-on-write object with a reference count.

[1] defined as a storage-location in [[ARM]], be it 1 byte or 1000 bytes. I don’t like the jave redefinition of Object.
[2] such as c++ references and primitive/value-type/nonref variables like myInt. I’d say the population of everyday variables are 50% re-bindable and 50% rooted, just like male/female populations. This is actually the worst minefield of confusion. In C, all non-pointer variables are rooted and all pointer variables can re-bind. C++ reference variables muddy the water. java/c# followed (java — cleanest rules). Python deviates further.
[3] In the simplest traditional scenario, a C myInt variable is permanently bound (rooted) to a 32-bit mutable object — immutable binding, address, run-time type, declared-type, name, but mutable state.

offload non-essential processing to helper threads

update: https://wiki.sei.cmu.edu/confluence/display/java/LCK09-J.+Do+not+perform+operations+that+can+block+while+holding+a+lock mentions a queue for logging requests.

In many real time trading systems, a novice threading developer may decide to dispatch some non-essential logging[1] logic to a task queue backed by a thread pool, without measuring the resulting latency. I don’t always have time to profile such trivial code, so I am not sure if latency might be better if we simply execute the logging code in the current “request processing” thread.

However — here’s the key point — even if separate threading doesn’t help in this case, it’s still important to know how to use threading in similar contexts.

I’d rather have a developer with threading know-how even though we don’t need it now, than a developer uninitiated to the intricacies of threading.

If you are wondering why the fuss over non-essentials, here’s an example — In some of the busiest, most time-critical trading engines in the exchanges/ECNs, there’s just a single thread sorting/matching quotes for a given symbol. This thread has to be lean and mean. All non-essentials must be delegated. We just mentioned a naive Design #1.

Design #2) Another common technique is messaging. Send a message with all the details, and leave the rest to the receivers. I feel this is slower than threading. Serialization, data copy, network bottleneck, socket handshake. Fastest thread should be network-free and disk-free.

Design #3) Another producer/consumer design uses a pre-allocated (circulalr) array of Requests. Main thread just set a flag to mark a request as executed and move on. Some consumer threads would clean it up and remove it from the array. Typically, we have a single producer and a configurable bunch of consumers, just like a thread pool. If there are enough consumer threads, then the buffer [2] will be almost empty. I feel it’s helpful if the buffer fits into CPU cache.

Note all the designs are producer/consumer.

[1] or other non-essential work such as serialization, persistence, replication, comparison, verification, DB query, or web download. See http://bigblog.tanbin.com/2010/10/gemfire-write-behind-and-gateway-queue.html

[2] You can call it a task queue, but physically it’s really a simple memory buffer, the simpler the faster.

stick-out, eccentricity, US vs Asia, high-flyer vs low-flyer

On 29 February 2012 02:54, LS wrote

Being direct, aggressive, loud is not what I meant by “sticking out”.  It is a work style.  It will help you push back, get attention, and not become an easy target.  Of course there are drawbacks, so it is just one of the work styles.  Not sticking out doesn't mean you have to keep low profile. 

What I mean is something completely unrelated to work.  A habit few or no one has, and make people feel it.  I will make up a couple of examples — you don't have them, I am just describing some examples.
1. If a person has to wash hand every 5 minutes during meal and therefore needs to pass other people when having a group lunch;
2. He eats very slow so people have to wait at group lunch.

I don't think there is any problem with sticking out unless you feel such kind of behavior hurts your career.  You may want to talk to the high-flyers around you to get some feedback.  Good luck.


Hi LS,

Hope your family is doing well in Beijing's winter.

I was thinking about the concept of keeping a “consistent” and harmonious appearance in the office, so that I don't look different from others. I now feel I don't want to change myself too much.

I feel many colleagues do show a bit of “personality” in paralinguistic (body language, silences, voice, hesitation…), email wording etc. I feel US big company culture is more relaxed and a bit more “individualistic” than Singapore.

There's an important backdrop though — Each colleague's rank in the office serves as the backdrop of the interaction. Many leaders do show a bit of unconventional personality. So do many high-flyers. Low flyers probably should not compare themselves with those.  I now remember in GS I was low-flyer and semiconsciously kept my head down. My team lead was abrasive, outspoken, loud (and aggressive) but he's a high flyer. Such a personality is very much liked by some, hated by others.

I now feel a fundamental factor why I developed my personal style is that I had been a relative high flyer in small companies for most of my formative years.

A basic (and rather relevant) principle is “if you aren't good at standing out, then don't stick out”. I have realized that after my initial rise as a 2nd-grade “computer wizard”, I am no longer good at standing out. In fact, my personal style is not welcome in a typical Chinese context, though more accepted in American context. In short, I'm not good at standing out. So it's very important that I don't stick out.

All my role models (like GS team lead, my sister) are in leadership roles, not necessarily management role. For them, being invisible isn't a good thing. But I also recognize many good leaders keep a low profile.

pure-output param in C++ standard library

Remember sybase/mssql stored proc can have pure-output param. Some of my blog posts (http://bigblog.tanbin.com/2011/06/output-param-in-sybasae.html) point out that @param1=@localVar2 actually assigns left to right !

Anyway, ansi C pointer parameter also supports pure-output param. Example:

Case A) pthread_create() has a ptr-to-pthread_t param. This param is purely for output only. Caller allocates a buffer then gives its address to pthread_create(). This function populates that buffer with (not the address but the value of) a new-born thread’s handle

The buffer deserves scrutiny. If caller were to declare an uninitialized ptr-to-pthread_t then it would allocate 32bits for the pointer. But in this case *content* of the buffer might possibly exceed 32-bit if it is a struct. If you assume it’s a struct, then you need to allocate the struct, then pass its address to pthread_create.

In most implementations, pthread_t is a typedef (supported in ansi C) for an integer, but it’s instructive to think of it as a struct.

Allocation could be on heap or stack, but in threading apps, stack allocation is often meaningless.

recvfrom() also has a pure-output, in addition to a in/out parameter. See http://beej.us/guide/bgnet/output/html/multipage/recvman.html.

pthreads mutex variable, briefly

The lock function [1] accepts an _address_ to a mutex object. So you can either have a ptr-to-mutex variable, or a nonref variable and pass its address. It’s hopeless to pass-by-value (pbclone) because multiple threads must share the same mutex instance, not copies of it.

Since address is needed everywhere, it’s slightly simpler to use a ptr, but don’t forget to free the malloc.

Typically you can’t avoid malloc, because a mutex object on stack is not very useful. A mutex is typically used by multiple threads.

[1] Actually all pthread mutex functions

virtual function adding complexity #letter to friend

Hi LA

I guess why many C programming teams avoid virtual keyword is because this innocent-looking keyword can cause complexity explosion when mixed with other elements —

– dynamic cast for pointer or reference variables
– pure virtual functions
– slicing — without virtual, slicing is more consistent and easier to debug
– passing a RTTI object by reference — note an object is an RTTI object IFF it has a virtual pointer.
– throwing an RTTI exception object by reference, by pointer or by value — again, things are simpler without virtual
– member function hiding (by accident) — without virtual, the hiding rule is simpler
– non-trivial destructors
– pointer delete — for example, deleting a derived object via a base pointer is undefined behavior if the destructor is non-virtual. If you avoid virtual completely, we would less likely write this kind of buggy code.
– double pointers — pointer to pointer to an RTTI object
– double dispatch — usually involves virtual functions. Double-dispatch may not be very meaningful to non-RTTI objects.
– container of RTTI objects, such as vector of pointer-to-base, where each pointee can be a Derived1 object or Derived2 object… — again, non-RTTI cases are simpler
– templates — remember STL internally uses no virtual function, and perhaps very little inheritance
– smart pointers holding RTTI objects
– private inheritance
– multiple and virtual inheritance

Some experts advocate all base classes should have virtual destructor. In that case, avoiding virtual means avoiding inheritance. That would definitely reduce complexity.

stay hands-on in financial IT

Better stay hands-on in financial IT, whether you come back to US or stay in S’pore long term. In Singapore, most hands-on developer jobs are less ideal (for older guys) than management jobs. Majority (above 70%) of my Singapore peers of my age group try to build a career path away from hands-on. Among the other 30%, some try to stay as hands-on architects or hands-on tech leads. If we were to stay in S’pore, such roles are ok till age 45. In the US, that age limit seems to be 55. Big question is whether you could become and survive as a hands-off manager. I think Yang tried but he still relies heavily on his tech skill. My competitive strengths (if any) in management roles are

* attention to detail
* reasonably good domain knowledge
* good rapport with some developers
* experience working overseas — many finance projects are global in nature

boost::serialization – learning notes

My learning notes based on the very clear and detailed tutorial http://www.boost.org/doc/libs/1_49_0/libs/serialization/doc/index.html

boost serialization features

* deep copy of pointer fields
* properly deserialize pointers to shared data
* proper support if a field is an array i.e. bunch of polymorphic pointers to Base and Der
* derived class serialization will include base instance state
* data and code portable across platforms – depend only on ANSI C++ facilities.
* when a class definition changed, older files can still be imported to the new version of the class. Similar to a hardcoded java serailVersion
—- More advanced —-
* fine if a field is a STL-container
* special considerations (tutorials) for shared_ptr — if a pointer field gives memory issues, convert it to a shared_ptr.
* special considerations (tutorials) for pimpl
* special considerations (tutorials) for xml serialization

In the simple GPS class (a serializable DTO), the class author must intrusively introduce a serialize() method, to be called by Hollywood during serialize/deserialize.

Intrusive solution is tailor-made by member function. Alternatively, one might opt for one-size-fit-all — non-intrusive serialization for classes without changing class definition. Class must expose enough information to save/reconstruct instance state. In the simple GPS example, we presumed that the class had public members – not a common occurrence.

The actual rendering of serialized data is implemented in the archive classes (text or xml or binary…). Thus the stream of serialized data is a joint effort between 1) serialization magic and 2) the archive class selected. It is a key design decision that these two components be independent. Interfaces to all archive classes are all identical. Once serialization has been defined for a class, that class can be serialized to any type of archive.

3 implicit conversions by c++ compiler – const

# a literal “any string” is treated as char const *. If you must assign such a literal string to a ptr-to-char variable, you must cast away the constness (Macquaries)

# If you have a const method in a non-const class instance, and if inside that method you access some instance fields, those fields are implicitly converted to const variables. See P215 [[eff STL]]

# as stated in other posts (like http://bigblog.tanbin.com/2012/02/c-const-methods-mean-this-is-bitwise.html), in a const method, “this” is converted to a ptr-to-const.

# http://bigblog.tanbin.com/2012/04/non-primitive-field-in-const-method.html

stdev measures how Fast price fluctuates

Stdev (MSExcel function name) measures dispersion in a sample. In the context of historical vol, stdev indicates

1) dispersion
2) how Fast price swings

If you plot log(periodic Price Relative) in a histogram, it will be a bell curve.  Periodic typically means “daily”, meaning we compute closingPrice(Day N)/closingPrice(Day N-1) and plot the log of these price relatives in a histogram.

Any such bell curve will Flatten (scatter) out if the sampling period lengthens from 24 hours to 7 days or 30 days, but a high historical-vol means a flat bell curve at a high sampling frequency (such as daily or hourly).

make dtor pure-virtual 2make class abstract #Google style guide

Many c++ authors say that if you want to make a non-abstract base class abstract, but don’t have any method to “purify” (i.e. convert to pure-virtual), then a standard practice is to purify dtor by adding “=0” at end of the already-virtual dtor.

Note: all base classes should have their dtors virtualized, according to many authorities, but Google style guide says “only if base class has virtual methods”

I wonder why not add a dummy pure-virtual method. (I think in practice this is fine.)

I believe the answer is efficiency, both space-efficiency and possibly run-time efficiency. Adding a dummy pure-virtual adds to every class size and vtbl size, but not instance size. Also every non-abstract sub-class must implement or inherit this dummy method.

Truly algorithmic trading on autopilot

Bulk of the software is written by “infrastructure engineers” using STL, sockets, system calls(yes)  and C tricks — all low-level stuff to optimize speed, and a small chunk is written by the algorithm researchers.

At SOD, operations start the trading engine and it runs on autopilot. The human control is provided not by a human trader but by the operations team. In a sense, the ops team are the traders.

glibc, an archetypical standard library (vs system calls)

Background — I was looking for a concrete example of a standard library.

Best example of a standard library is glibc, produced by GNU team. If you strip the G, it’s “libc” — THE C standard library.

“Standard” means this library is a “carpet” hiding all the platform-specific differences and presents a uniform interface to high-level application programmers — so-called App-Programmer-Interface or “API”.

There are many industry standards for this same purpose, such as POSIX, ANSI-C (which standardizes the C programming language + standard lib). Glibc supports all of these standards.

To clarify a common confusion, it’s worthwhile to understand this simple example — glibc functions (like printf) are implemented in platform-specific underlying syscalls.

Q: Exactly What are the platform differences? Short answer — system calls. Remember System calls call into the “hotel service desk”. These syscalls are tied to the processor and the operating system so they are by definition platform-specific. See other posts on syscall vs standard library.

how much math is needed in … #a few finance IT fields

I always maintain that math is key domain knowledge. In the few areas I know, when a bit of math is required, a lot of regular developers would need to brush up before they can “play in the band” (as in a heavy metal band).

– Bond trading platforms? Close to zero math. Definitely less than bond pricing. Developers do need to remember basics like when interest rate rises, bond prices drop, and zero-coupon bonds trade at huge discounts (due to price/yield math relation). Also high-grade bonds tend to trade at lower yield — minimal math.

– Bond pricing? Slightly more than zero math. Price/yield conversion, yield-to-worst, modified duration, dv01 rollup, time-value-of-money are widely used, but developers usually don't need any mathematical insight into them to do the job. Developers just need basic understanding. Hiring managers may think differently.

– Forex trading? Arithmetic only, I believe. Add/subtract/multiply/divide only. Bond math requires high school math. In contrast, Lots of Forex trading IT jobs require primary school math.

– Option pricing? A bit of statistics and calculus. Like in bond pricing, developers don't need any mathematical insight. However, they need a basic grasp of historical vol calculation, implied-vol inversion, delta and other greeks. Even more fundamental concepts include PCP and the option payoff diagrams of about 10 basic strategies, but I don't think developers need these.

– VaR on a portfolio including bonds and options? Requires more than bond math and option math. More statistics. But I suspect a dedicated developer in a bond pricing system may need more in-depth bond math than a VaR developer, because a stringent math requirement on software developers is impractical, overkill and giving the wrong focus to the wrong role. Instead, the real math is usually for the quant analysts. Risk systems use more math than pricing systems, and employ more quants.

5 implicit conversions by c++ compiler – big 4

See also post on the “same” topic. There are too many such implicit things to list in one blog. This one is about the dtor, copy-ctor, assignment operator + ctor — the big 3 + 1.
 
# MyClass myObj; // calls the default ctor
# MyClass myObj5[5] // ditto
# op-new first argument is always a size_t, but when you invoke it you never pass pass in size
# assigning to an existing reference var of MyClass calls the assignment operators implicit
          refVar2.operator=(….) //because reference variables can’t be reseated

# per-class overloaded operator new and operator delete are implicitly static, always. ARM P283
** overloaded op new is (implicitly) inherited

# Say your copy ctor is invoked (often implicitly;-). Therein you need to clone a FIELD of type T. If you don’t use the field initializer syntax, then the copier first calls T’s no-arg ctor to “make room” for the field, then calls T’s operator= to modify its state.

# when you new up a Derived object on heap, and Base class has an operator new, you would use that (implicitly inherited) operator-new unless Derived class redefines (“override” is wrong term) operator-new.
** when you new up a Umbrella object on heap, and the object has a non-ref (not pointer either) field of type Base, you would ignore whatever operator-new Base has. You use either “::operator new” or the Umbrella’s own operator-new to grab a sizeof(Umbrella) chunk. That’s all the memory you need. Part of it will be constructed as a Base object. See [[more effC++]]

10 implicit translations by c++ compiler

See also posts on the “same” topic.

Note many inefficient implicit conversions are selectively, optionally “optimized away” by various compiler options. This adds even more complexity.

# proxy classes (like in [[more effC++]]) rely heavily on implicit translations
# smart pointer are drop-in replacement for raw pointers, and always relies on implicit compiler translations.
# “–any literal string–” anywhere in source code is implicitly converted to a char*const
# unnamed temporary object creation by compiler. [[more eff c++]]
# RVO
# [[c++ primer]] here are 3 identical func declarations because all array params are implicitly converted to ptr param. This is still very relevant because arrays are far more widely used than vectors (C is widespread) —
  void f(int * )
  void f(int[] )
  void f(int[5] )

#51 [[effC++]] Item 19 points out that if your Rational class overloads operator* with a Rational argument, then “aRational * 2” still works because compiler converts it first into
  aRational.operator*(2); // then it needs to convert 2 to a Rational object, so it converts the call into

  const Rational temp(2);
  aRational.operator*(temp);

#50) if f() declares a param of type Animal, you can pass in a Shape variable, if there’s a converter. (See effC++ Item 26). Applies to any LHS=RHS expression like

** Type5 var2 = var3ofUnrelatedType // you get either a Type5 conversion ctor or a conversion method in UnrelatedType, 

#10) overloading the arrow operator ie “->”. See P19/23 [[boost]]. IBM explains — The statement x->f() is interpreted as (x.operator->())->f()

5 implicit conversions by c++ compiler — STL

See also posts on the “same” topic.

#50) [STL] when you use STL copy() to print any container, you implicitly call the ostream_iterator ctor

#40) [STL] when you specify a pred func in a STL algo, there’s a ton of implicit conversions behind the scene. Here’s one example. The pred func is invoked as

  bool flag = pred(*the_iterator_in_this_func_call) // for a unary predicate

#30) [STL] when you put a functor TYPE into  , the specialized template class’s constructor instantiates a functor OBJECT. In short, you specify functor TYPE only — functor Object instantiation is implicit. See P91 [[effective STL]]. P154[[STL tutorial]] shows

 set<char, less > mySetOfChar;

#20) [STL] converting a func name to a func ptr — when you pass the func name as arg to some STL algo. P 202 [[effective STL]]

HASA ^ ISA ^ vs PointTo in c++

In java, hasa always means point2 (or pointTo), assuming The Small object is never a primitive…. In c++, hasa means Host object’s has a piece of its real estate carved out for the Small object “embedded” therein [1]. This is a profound difference from java. Here are some consequences —

* Even if you put a custom op-new into class Small to prevent heap-instantiation of Small, class Host can still instantiate a Small on heap during its own construction. See [[More effC++]]

* const (on the host object or a non-static method) means real-estate const. So the non-ref field should not change. This governs hasa, and has no effect on points-to.

* If a c++ Host object points-to a Small object, then memory leak is quite possible, so Host dtor had better delete that Small pointer. Therefore c++ points-to suffers from memory leak but no such issue with c++ HASA. Host dtor automatically calls Small dtor, in the DCB sequence (see blog post on DCB)

ISA also involves embedding. Therefore, the custom op-new in Small won’t prevent a Small instantiation on heap if Small subclass has its own op-new.

tuning/optimization needs "targeted" input data

My personal focus is performance profilers for latency engineering, but this principle also applies to other tuning — tuning needs “targeted” even “biased” input data, biased towards the latency-critical scenarios.

See also the post on 80/20.

The “80/20” item in [[More Effective C++]] points out that we must feed valid input data set to performance profiler. It think it’s easy to get carried away by a good quality profiler’s findings, without realizing that the really important users (who?) or the critical operations may not be represented by the test case.

This sounds like more than an engineer’s job duty — find the relevant input data sets that deserve performance-profiling most. Perhaps the business analyst or project manager knows who and what functionality is performance-critical.

80/20 rule, dimishing return…

In latency (or any other) tuning, I’m always be suspicious and critical about common assumptions, best-practices….I don’t completely subscribe to the “80/20” item in [[more eff c++]] but it raised valid points. One of them was the need for “targeted” input data — see http://bigblog.tanbin.com/2011/01/performance-profilers-needs-targetted.html

Bottleneck is a common example of 80/20….

Diminishing return is another issue. A discussion on DR assumes we have a feedback loop.

feedback loop — It’s so important to have a typical (set of) test setup to measure latency after tuning. Without it we risk shooting in the dark. In many situations that test-setup isn’t easy…. Good feedback loops focus on the most important users. In some trading apps, it’s the most latency-sensitive [1] trades.

Illustration of reliable feedback loop — to test a SQL query you just tuned, you must measure the 2nd time you run the query, because you want all the data to be in cache when you measure.

cost/benefit — sometimes there’s too much unknown or complexity in a module, so it would take too much time to get clarity — high cost in terms of man-days or delivery time frame. After the research it may turn out to be a hopeless direction — low benefit

[1] largest? no. Largest trades are often not latency sensitive.

am fast at GTD, as Wall St developer

* I grew fast enough for GS. Compared to other banks, GS has 50% of the headcount for any given workload.
* I was fast enough for the Lab project lead. Their standard is even higher than GS. The Lab project lead didn’t complain about my delivery speed though he pointed out Piroz was more experienced and faster.
* Citi senior mgr didn’t complain about my delivery speed at all. Not even once. I was working at a comfortable pace.
* I worked alongside Lab49 consultants who are battle tested fast-coders over many years.
* in every job i managed to steal time for self-study – swing, python, secDB, rv, options, c++, non-blocking I/O, c++ debugger, bond math…
* many trading desk dev teams are elite teams with tiny headcounts. I survived some.
* I generally pay more attention to details than other developers. I could, if I want, switch off this attention.

I guess there are super-developers who are really faster in some specific contexts, but overall it’s hard to be a fair judge considering quality, prod support capability, error checking, knowledge-transfer, automated-testing, help to other team members, client-orientation, maintenance cost, system flexibility, readability, architecture soundness…

what trades & orders require regulatory reporting

US has many more regulatory reporting requirements (than UK) — OATS, Trace, ACT, MSRB, LOPR, Blue sheets ..

–OATS
* Orders, not trades
* Nasdaq mostly + some OTC orders
* Not real time

–ACT — designed for real time transparency
* Eq only
* OTC only
* Reports executions
* 30 sec after execution
* Covers ECN and ATS liquidity venues, in addition to exchanges

–TRACE
* Customized for FI
* OTC only — most FI cash trades are OTC anyway
* Exempted — Most short term FI Within 15 min

–UK
* Eq, FI trades,
* OTC — CFD, OTC options, swaps
* Exempted — commodities/FX and their derivatives.
* Exempted — IRD