forward volatility — from basic principle

See also — variance is additive. More specifically, for n multiple independent experiments with each outcome having var1, var2, …var_n, the sum of the outcomes has variance var1+var2+…var_n. Incidentally, the sum of the outcomes has mean of mean1+mean…mean_n. A random walker makes one “experiment” at each step, and the log(PR) is cumulative.
One basic assumption in is “Given that the underlying random variables for non overlapping time intervals are independent, the variance is additive.”

Q: What’s that random variable? I believe it’s the “r” i.e. log(PR) described in in This variable “r” is additive in itself. Intuitively, if over 4 days gold price moves by a ratio of 120% -> 101% -> 97% -> 99%, then we can ADD r2,r3,r4,r5… to get the overall Price-Relative of Day 5 over Day 1 closing.

where r1-2 represents log($Day2ClosingPrice / $Day1ClosingPrice) = log(PR over Day2), which is another label for the earlier r2.

It turns out this sum is the variance of log(PR1-5), or variance of r1-5 , i.e. the volatility over the 5 consecutive observations. Since the random variables r1-2 r2-3 r3-4 r4-5 follow the same pdf, each variance should be numerically identical.
==> variance over 96 hours and 5 observations (Price1 to Price5) is exactly 4 times the daily variance
==> If we assume 256 trading days in a year, then annual variance is 256 times daily variance
==> annualized vol is 16 times daily vol. If annualized vol is 80%, then log(PRdaily) has stdev = 5%…..

The sum in [1] is variance over Day 1-5. Forward variance over 2-5 can be derived from the 4 individual variance numbers, or from….

However, it’s unfair to compare 2-5 fwd variance against “spot” variance of 1-5 when the holding periods are unequal. Therefore, all the variance numbers must be annualized for a fair comparison.

Basic statistics rule — if Y = X_a + X_b, i.e. 2 independent normal variables, then the variance of Y is sum of the 2 variances.

knowledge@vendor products is far from enough4GTD

Consider typical non-trivial financial app based on 1 or a couple primary vendor products + some database + some JMS server + some app server + some web server. The #1 “primary vendor software” could be a distributed cache, spring, ETL, or something like ION or something else.

Q: How much complexity is in the vendor software, and how much is in the site-specific code including localized customizations of the vendor software?
A: 10% vs 90%

How about the ML Edge b2b trade engine? 70/30% at this stage (March).

harmless productivity tools ^ frameworks like ..Spring

[[The Productive Programmer]] by Neal Ford advocates learning a large number of _simple_ tools for productivity. Tools could be a complex but we use them in a non-intrusive, simple manner. Many other tools promise to help productivity but are traps – spring, hibernate, jaxb, xslt. Once you employ them they tend to become your master.

I have no such issues with perl/shell scripting, testing tools like junit, unix utilities, text editors, eclipse, maven/ant (?) A critical feature is, the option to use the tool in part of my project, and use alternative tools for the remainder. The tool should not be possessive and exclusive.

Hibernate is more gregarious than Spring, as you can mix Hibernate and straight JDBC in different modules of your system. Spring, being a framework, tends to engulf your entire project. You are then constrained by that framework developers’ decisions.

##wall-blaster handy techniques to impelement features

Challenge — to implement non-trivial logic quickly, cleanly, with assurance and peace of mind.

Default (stupid) solution is a java/c++/c# implementation – painful. The challenge feels like a stone wall to drill through. The following technologies are good dynamites. (Additional Background — often, there’s a legacy Prod codebase…)

#1) regex
#2) SQL execute immediate
) perl
) reflection
) rmi
* embedded db
* multiple-statement SQL batch to implement complex logic
* byte code engineering
* DB constraints to replace app validation
* MOM as concurrency tool
* DB as concurrency tool
* file system as concurrency tool
* jmx
* annotation

* javascript, if this fits into the infrastructure
* xslt
* aop

I don’t fancy dependency-injection, ORM, ajax.

tibrv subject based address`^jms topics

tibrv subject based address` (SBA)^jms topics

Among the defining features of Tibrv, [ A) decentralization and B) SBA ] are 2 sides of the same coin. For me, it’s a struggle to find out the real difference between SBA vs jms topics. Here’s my attempt.

#1) In SBA(subject-based-addressing), there’s no central server holding a subject. In tibrv, any sender/receiver can specify any subject. Nice — No admin to create subjects. I tested it with tibrvlisten/tibrvsend. By contrast, In JMS, “The physical creation of topics is an administrative task” on the central broker [2], not on the producer or consumer.

) If a jms broker goes down, so do topics therein.

) a jms publisher (or subscriber) must physically connect to a physical broker process[2]. The broker has a physical network address. Our topic is tied to that physical address. A tibrv subject has no physical address.

) tibrv SBA (subject-based-addressing) uses a subject tree. No such tree among JMS topics. The tree lets a subscriber receive q(prices.stocks.*)  but also q(*.your.*). See rv_concepts.

[2] such as the weblogic server in autoreo.

daily margin call – simplest futures illustration

Simplest and best-known margin calc is in commodity futures, say a Dec oil contract. Let me use it to describe daily margin calc in my own language.

Clearing house computes and issues daily margin calls. Therefore the formula/algo is crafted from the clearing house’s standpoint. Goal is to protect the clearing house from
1) any “reasonable volatility” i.e. a daily price swing smaller than the extreme 0.05% cases. (More extreme cases would show Larger swings.)
Here, let’s assume house estimates 1% maximum daily move.
2) any reasonable/unreasonable member default.

relevant — EOD best bid/ask
irrelevant — Current spot price — irrelevant.
irrelevant — Last transaction price for the same Fut contract — irrelevant.

Suppose current mid-day fut price = {89.99/90.01} ie {best bid/offer}

Suppose I BUY a contract mid-day at the market i.e. $90.01. Clearing house locks up an amount $x as collateral in my margin account to protect house against my default. If I default, on delivery date house still [1] need to BUY from the contract seller @ $90.01 the agreement/transaction price, but how does the house get the cash to BUY? To answer that, let me first Introducing the basic long margin call formula

  EOD-liquidation-value [my long position]  – 1% + x = $90.01 // Soon We will solve the margin requirement $x of a long, using agreement price and best BID at EndOfDay.

Here the clearing house is assuming market moves at most 1% against me by end of Day2. So if I declare bankruptcy on Day2, house liquidates my long position at most 1% below Day1’s EOD mv.

We will denote “current liquidation value of some position” as mv[that pos]

Now, mv [my long pos] is simply the best SELLING price of that asset (the oil contract) when house must liquidate my position. Think hard — That’s actually the end-of-day best BID. That’s $89.99 in our case, assuming no market move.

  x = $90.01 – $89.99 * 99%, roughly $0.02, which translates to $0.02k = $20. Here we assume each full contract is $90k when price = $90

— Now suppose market collapses drastically and moves against me to ($88/$88.02)

mv [my long position] – 1% + x = $90.01 // $90.01 is the me-counterparty agreement price

x = $90.01 – $88 * 99%, roughly $2.01k

A 2.2% Fut price change (89.99 to 88) causes a large margin call. I would say the margin call roughly matches the physical contract valuation drop from $89.990K to $88K

[1] so my default doesn’t cause a chain reaction taking down the seller.

5 parts in socket data structure

— Adapted from

Note accept() instantiates a socket object and returns a file descriptor for it. accept() doesn’t open a new port.

A socket object in memory consists of 5 things – (source ip, source port, destination ip, destination port, protocol). Here the protocol could TCP or UDP[1]. This protocol is identified in the packet from the ‘protocol’ field in the IP datagram.

Thus it is possible to have 2 different applications on the server communicating to to the same client on exactly the same 4-tuples but different in protocol field. For example

Apache at server talking on ( on TCP) and
World of warcraft talking on ( on UDP)

Both the client and server will handle it since protocol field in the IP packet in both cases is different even if all the other 4 fields are same.

[1] or others

10 cool features of notepad++

#1a) The UltraEdit feature — You can type “insert table_2 select” and see it showing on 5 lines as you type.
#1b) select/remove a column of text. Contrast Alt-drag vs regular drag.

#2a) highlight open/close brackets across lines. See red dotted vertical line if you set tab size to 1 or 2. The missing Eclipse feature.
(View -> show symbol -> show indentation guide)
#2b) highlight open/close brackets on the same line
(Settings->Preferences->Misc->highlight matching tags)

#3) smart highlight (Preferences->Misc) highlights all occurrences once you select a word
#3b) permanently mark all occurrence of any string (partial word, multiple words..) See

#4) “update silently” and “scroll to last line after update” (preferences->Misc->File Status Auto-Detection)

#5) double click on a tab to close the document

7a) If you are at the closing tag of a long block, you can jump to the opening tag. (TextFX -> quick -> find matching bracket, with Shortcut Key)
7b) You can also highlight the entire block (TextFX -> quick -> mark)

) (View -> Hide Line) to hide a block of lines
) minimize to tray, to un-clutter the task bar
) to extend vertical space, hide toolbar or menu bar, put tab bar
) “reloadFromDisk” is nice but you can map a custom key (like F5) high latency
) vertical (Preferences->General)

major valuation techniques for volatility instrument

1) formula-based
2) binary tree — fairly simple
3) solve partial differential equations — use numerical methods to “search” for a numeric solution(s) to the equations.

These are the major techniques used against familiar/everyday volatility instruments. For complex/exotic contracts, you need simulation.

Simulation may be justified even for RFQ valuation — Request/trade volume is low, so people spend hours to price a deal.

socket, tcp/udp, FIX

I was told both UDP and TCP are used in many ibanks. I feel most java shops generally prefer the simpler TCP.
Specifically, I learnt from many veterans that some ibanks use multicast (UDP) for internal messaging.

FIX is not much used. It is sometimes used as purely message formatting protocol (without any session management) on top of multicast UDP. (In contrast, full-featured FIX communication probably prefer TCP for connection/session control.)

Similar – Object serialization like protobuf.

professional consultants — tone down personal taste

Respect your client’s preferences and personal tastes. They will inherit and live with the code you leave behind. If they don’t feel comfortable with some part (even if it works perfectly) they have a legitimate reason to change it to something more manageable.

You are not owner of that code. You are writing it for him/her to own.

Imagine you are a carpenter making a dressing table for a client. Is your personal taste or client’s taste more important?

In fast-paced finance IT, sometimes you can be forgiven to adopt your personal favorite choice (rather than client’s favorite) to help you get things done fast — since you are the coder. I feel in some contexts time-to-market is still the paramount consideration.

STL map search and so-called "equivalence"

Question: key-based or pair-based?
A: insert/erase/find/access-by-key[1] are key-based.
A: I don’t know what operations are pair-based

Question: key-matching is position-based or equality-based?
A: position-based, i.e. equivalence-based

[1] access-by-key means myMap[keyA1]. This is the standard dictionary/associative-array access, but not supported by multi-map, since this expression must return a single value.

piecemeal vs one-gulp – c++ object memory allocation

– some operations must be piecemeal
– some operations must be one-gulp

* ctor and dtor are always piece-meal. Reason — each subclass and embedded component class can define its own ctor and dtor.
** For example, dtor execute in the DCBC sequence —

* “operator new” (I don’t mean a new-expression) is always one-gulp. Reason — the memory (“real-estate”) of an entire object must be a contiguous piece of “real estate”, but one allocation for one part of the real estate and another allocation for another part will be dis-contiguous.
** There’s an important implication on P157 [[more eff C++]]

Background —
+ if you are a derived class instance, Your base class instance will occupy part of “your” real estate
+ if you have a nonref[1] field of a another class, That instance occupies part of “your” real estate

[1] “nonref” is something other than a pointer or reference, something not dereferencible. No such field in java, because a non-primitive field is always a reference field in any java class.

mvn – which settings.xml is/are in effect@@

Q: when there are multiple mvn settings.xml files, which file(s) is/are in effect?

A: make the xml malformed and try a basic mvn command (see tutorials) to see which settings.xml file is picked up.

Eclipse mvn plugin actually merges global settings.xml + user settings.xml.

Stackoverflow says –

“Maven always uses either one or two settings files. The global settings defined in (${M2_HOME}/conf/settings.xml) is always required. The user settings file (defined in ${user.home}/.m2/settings.xml) is optional. Any settings defined in the user settings take precedence over the corresponding global settings.”

svn tagging^branching

Definition — A Tag in CVS can be read-write tag, but here we are ONLY interested in readonly tags ie snapshots.

The svn-book chapter [[cheating a simple tag]] ( addresses the common task of ROtag. A few key points —

* as long as nobody ever commits to a ROtag, it forever remains a snapshot/ROtag.
* If people start committing to it, it becomes a branch/RWtag.
* In Subversion, there’s no difference between a tag and a branch. Both are copy-on-write directory copies.
* when you create a ROtag, you can optionally specify a revision number. See The Chapter.
** a revision number is always a ROtag ie a snapshot of the filesystem after each commit.

* unix hard-link functionality
** copy on write
** cheap copy
** Make branches/tags as often as you want

When you copy a directory, you don’t need to worry about the repository growing huge—Subversion doesn’t actually duplicate any data. same concept behind a unix hard link. It duplicates data only when it is necessary to disambiguate different versions of objects — copy on write.

##1 week in ML — stay marketable and employable in financial IT

(another blog post)

Problem — in any bank, a contractor or employee at any age could lose job due to many reasons – economy, new boss…
Problem — too many young developers join the job market every year, often from overseas
Problem — we aren’t getting younger 🙂
Problem — waves of new technology

If anyone doesn’t feel the need to worry about it, i feel that’s unwise. One of my strategies to survive is focus on old and core technical topics –

* threading. Wall street always likes threading, even if their system doesn’t use threading
* data structures – java collections, STL, trees, pointers, basic arrays and basic strings (both outdated but extremely relevant)
* OO – static vs non-static, HASA vs ISA, abstract, constructors/destructors, proxy
* DB table + data model design
* DB tuning

Other areas I like
– memory mgmt
– messaging
– unix
– sockets

These technologies didn’t change for the past 10 years, except some new GC algorithms.

limitations of risk models – Haitao discussion

See also post about FRM critique on risk metrics.

[haitao] Your view on risk management matches with one of our professor’s, he also believes that using risk models such as VaR does not suffice the purpose of risk management, it is really a very broad field that needs to be explored further. This is actually at frontier of academic research, in particular, Professor mentioned 3 flaws associated with VaR: 1. Not considering the magnitude of loss beyond VaR. 2. VaR penalizes firm diversification. 3. Fattail issues related assuming distribution of critical events failed to reflect market situations. I have also attached his research paper drafted in 2010, he is very well known in Fix Income field.

[Bin] I feel VaR is the best concept I know today (except Expected Shortfall) but very, very imperfect. As I said earlier,

I feel any attempt to predict how market reacts to extreme events is futile as such extreme events are rare and we have limited experience with them. — Black Swan theory

How do you predict the impact of a comet hitting the earth, if you have not analyzed 100 similar collisions?

When liquidity dried up in 2008, I was told every bank held on to whatever cash it had and refused to lend. All our models fail. US government had to create about 1000 billion new dollars. No one can predict this either.

Similarly, some governments (Germany after WWI??) prints so much money that a cows price shot up not 100% but 100,000,000% in a year. No risk system can handle such inflation. I remember photos of German children playing with bundles of bank notes as toy bricks.

More recently, Iceland, Greek and Irish governments went bankcrupt. I doubt any VaR system accounts for such extreme possibilities.

What kind of risk system predicted the impact of Freddie and Fannie failures (or Lehman, Bears and AIG)? I doubt. I feel even the strongest (like Goldman Sachs) would not survive if AIG goes bankrupt without government bailout, since AIG is insurance for GS.

You can have lots of smart hedges to protect big risky positions but what if the instrument used in the hedge itself becomes illiquid and worthless, or the counterparty is unable to fulfill its obligation? Can risk models predict that half of all market participants go bankrupt together?

Many derivative markets are subject to tighter regulation. Perhaps liquidity and spread will worsen?? VaR systems must adjust for such policy changes and predict the implications? Hopeless in my opinion.

Sometimes I feel the risk analysts are ostriches burying their heads. They know a lot of extreme market events are not accounted for in their system but they just wish them away (just to keep their jobs and keep their research projects alive). They see the emperors new dress is naked but dont dare to say.

[haitao] I recall some of the interview questions from BB in Hong Kong and Singapore:

a. All sort of behavioral questions: Tell me a time…Why us…Why this position…Leadership…

Problem solving…Resolve Conflicts…etc
b. What do you think tax cuts extension impact on yield curve?
c. What is your view on Asian markets particularly Chinese stock market in 2011?
d. Why do you think Commodity price should surge even higher? What are the factors cause such dynamics?
e. For historical US and Japanese interest rates, one of them is normally distributed and the other is log-normally distributed, which is which? Justify your answers?
f. Explain Fix Income to a layman.
g. Discussion on recent market events.


overnight risk reporting in portfolio management

I talked to a big portfolio mgmt (PM) firm. Team owns and delivers nightly risk reports to traders (+ perhaps fund managers). According to the team mgr, the most important sister team is the quant team, who are often PhD's but not professional coders. Quants are really qualified to create models but these quants actually implement their models in c++.

There's a large amount of data in DB. Nightly job reads in these data and analyzes them using the c++ models, then writes data back into DB.

This is a heavy-duty number crunching batch job, heavy on DB, light on network – no socket programming.

Logic is mostly in perl, c++, shell and DB. DB holds significant amount of logic, just like Goldman Sachs PrivateWealthManagement. It turned out c++ implements more business logic than perl. These perl scripts are considered low-logic, but if there's a lot of perl, then I believe there's a lot of logic.

Perf is the biggest issue. Job must complete in 12 hours, before a 3am deadline, without break. If it breaks, there will be … delays and …? Bottleneck is DB. There's spare hardware capacity underutilized but the DB server is on its knees. I have heard of the same many times, in GS, citi… so I guess this is hard to avoid. Risk system is probably worst affected.

%Q: stress testing? Monte Carlo?
A: the reporting system doesn't do those. Those are probably the job of quants.

%Q: is VaR the key output?
A: no. duration, curve duration, spread duration

%Q: is matrix and “vectors” used in the c++ code, like those in matlab? So it goes well beyond STL?
A: yes quants use matlab and mathematica to develop the concept, and then use c++ to implement it. We do have our own data structures beyond STL.

%Q: how much domain knowledge required in the analytical work?
A: more of an aptitude and attitude to learn

tibrv latency tips – from tibrv documentation #logging

1) offload to worker thread — When inbound messages require lengthy processing, we recommend shifting the processing load asynchronously. Quickly extract data from the message, and process it in another thread.

* compare EDT/swingWorker

In a ML muni trading engine, we designed several “grabber” listeners
– grab messages and route them to different processing queues, based on hashcode.
– grab messages, append a short code to the subject, then republish to the same queue. Not sure if this works.

2) reduce message size — Avoid XML. RV now supports integer field identifiers similar to FIX. 16-bits and much smaller than String field names.

3) reduce logging — I always felt logging hurts performance. Now confirmed in Tibrv manual! When logging is required for monitoring or auditing, shift the I/O burden to another computer to log messages without introducing a time penalty

same case-expression in q[SELECT/GROUP-BY]

See P417 [[transact-sql programming]]. In GS and Citi I have written many ugly queries where a large case expression appears in SELECT and again in GROUP-BY.

Not a performance problem, but a maintenance problem.

My advice — Have no fear. You can use Perl or Java to manage the ugly query. Not possible if you need it in a stored proc and have no experience with execute-immediate. It’s still ok. There’s often no better alternative.

##essential features@FIX session #vague

Top 5 session-message types– Heartbeat(0), Test Request(1) ResendRequest(2), Logout(5) / Logon(A).[1] This short list highlight the essential functionality of the concept known as “FIX session”. I feel the most important features are sequence number and heart beat.

FIX session concept is different from a database server session. I feel FIX session is more like a TCP connection with keep-alive — says A FIX Session can be defined as “a bi-directional stream of ordered messages between two parties within a continuous sequence number series”… very much like TCP. Each FIX Engines is expected to keep track of both sequence numbers — The Incoming Sequence Number expected on inbound messages received from the counter-party.

A broker FIX server maintains many FIX sessions. Each FIX session has its own pair of sequence numbers. When a FIX client connects to a FIX server, server would check the SenderCompID and TargetCompID values to pick the pair. If there are some persisted sequence values, then FIX server resumes the lost session. If another FIX client also uses the same SenderCompID/TargetCompID values, then the FIX server might assign the lost session to this client. Sometimes server requires password login before resuming a lost session. is a blog post by a fellow developer in finance IT. is another blogger

[1] In parentheses are MsgType values, which are ALWAYS single-characters.

##finance jargons asked by interviewers

what’s theta for an option?
What’s forward rate?
[Barc] Define bond duration. Are there other definitions of duration?
[MS] Bond with an option
Dv01 – how to calculate
[MS] when and why would issuer call a bond?
[MS] How to price an interest rate swap
[MS] Call-put parity
[MS] How does volatility affect a call option’s price? How about a put? Why?
[MS] How is callable bond priced. Why do investors like/dislike callable bonds?
[Citi] How are interest rate and bond price related?
[MS] risk-free bond
[Ms] Price-yield conversion – both ways
[MS] Why is the price-yield curve convex?
[CS] On-the-run treasuries vs off-the-run
[MS] Can discount factor be greater than 1.00?
How do you compute PnL in your system?
Bond pricing algorithms/models in your system
Benchmarks used in your system?
Butterfly strategy
Busted trade
Market order vs limit order

2011 Pimco c++ #+perl/SQL #done

Many obscure QQ questions. Be selective what to study!

Q: diff between malloc() and new()?
%%A: must cast void ptr; must call ctor
A(hind sight): malloc need to be told the size. In contrast. new() relies on knowledge about the class
A(hind sight): array new can be done with malloc too
A (hind sight): new can be a (static) method, therefore inherited or hidden.
A: q(new) would put some housekeeping data in a header. Therefore, you can’t use free() on a new() ..

Q: ok, so malloc doesn’t call ctor, but can I pass the malloc void ptr to a ctor for initialization?
%%A: placement new will initialize a block of memory without allocation.
%%A: it’s not common. C++ language doesn’t encourage this operation.
A(hind sight): ctor doesn’t take a ptr argument?
AA: placement-new — Also see

Q3: can you call a dtor explicitly?
%%A: yes though not a good idea. See C++ FAQ
A (hindsight): placement delete

Q3b: when would yo do that?
%%A: when i want to free a large block of heap memory as early as possible, and the dtor is not scheduled to run any time soon. But without delete() the heap memory isn’t freed. Probably delete can achieve the same purpose without looking ugly.
%%A: again, not  common and not encouraged by C++ language. I don’t think i would ever do that.
A(hind sight): placement new? Correct 🙂

Q5: can a dtor throw exception as the last statement?
%%A: no cos the dtor can be invoked as part of stack unwinding

Q5b: so what?
%%A: stack is unwinding due to another exception, and now your dtor creates a 2nd exception object. System is going to lose critical information in the 2 exceptions. The original exception handling flow is halfway through.

Q: given an instance of an absolutely empty class, can I downcast a ptr (or reference) to it?
%%A: assuming this class inherits from another empty class, you can’t since there’s no vtbl.

Q4: can you specialize a template with a ptr type as the type argument?
%%A: yes vector of raw ptr is common

Q4b: what must this template do to accommodate ptr template arguments? Too advanced and not one of my Tier 1/2 specializations
%%A: handle dtor, assignment and copier.
%%A: any memory-management operations must be customized. Default behavior is not good for ptr type arguments

Q: which part of c++ is most interesting to you?
%%A: low level access. Few other languages provide the low-level access

The remaining (perl and sql) questions are less daunting.

Q: perl pass an array by reference into a sub?
%%A: you can pass the address of the array. Array can be modified in-placer by the receiving sub

Q: chomp?

Q: perl open a file in append mode?
A: use “>>”

Q: when would you create a non-clustered index?

Q: select 2nd highest salary from a salary table?

Q: if my query need all 3 columns of an index (id, name, salary), how would you order the 3 columns when creating the index? Suppose salary has the highest selectivity
%%A: salary first

check sybase user connections

Total no of allocated connections are 1700.

1> sp_configure 'number of user connections'
2> go
Parameter Name                 Default     Memory Used Config Value
Run Value   Unit                 Type
—————————— ———– ———– ————
———– ——————– ———-
number of user connections              25     1990886        1700
    1700 number               dynamic

physical ^ virt memory — when do we see which

V – when you print out the address of any C++ object.
P? – out of memory
P? – committed memory as reported by various tools?
P – seg fault or “general protection error”
V – stack and heap sections growing towards the center. Note heap is contiguous only virtually — Physically discontiguous as your app requests big chucks of physical mem from OS.

Q: is physical memory address knowable?
A: Probably no. “If, while executing an instruction, a CPU fetches an instruction located at a particular virtual address, or fetches data from a specific virtual address or stores data to a particular virtual address, the virtual address must be translated to the corresponding physical address. This is done by a hardware component”

Q1: is virtual address knowable in c?
A: yes

Q1b: is virtual address knowable in java?
A: probably no

y banks hire contractors more easily than employees

Employee hiring is a longer process. HR and entire firm probably perceives employees as a more “permanent” addition to the team. Employees are allowed to transfer internally, so once an employee is
brought in, she tends to stay for a long time. Due to the permanent nature, employee head count needs more approvals. It’s like buying a home vs renting a house.

Consultants are sometimes hired for a test-drive. If he’s good, then offer him a perm role. If he’s not suitable, then let him go painlessly.

Companies invest in employees. Employees receive training and development, reviews. Employees are potential leaders of the company.

Leadership is a critical element at every level of the organization.

Consultants can’t help – not allowed.

Last but not least, employee layoff is much harder than for contractors. Companies often prefer employees to resign than a forceful termination.

fwd: heap memory allocation – java/c#/C

Thanks Nigel or the article. A quick glance suggests to me the allocation (not the de-allocation) procedure is no different from malloc, logically.

At run time, the memory management library (some functions pre-loaded into the code section of the address space) gives out small chunks of memory to the requesting application. When there’s insufficient “small chunks” to satisfy a request[1], the library synchronously grabs a large block from OS, and adds this large block to the private free-store, which is private to the process, managed by the library. This is the malloc() behavior I know.

[1] perhaps due to fragmentation

This “library” is compiled binary code. For c/c++, I think this means platform-specific object code. For java, this means platform-independent java bytecode, or more likely platform-specific native code.
For dotnet, I guess it’s windows native binary code. Now, is this binary code compiled from a C linker/compiler that used the standard malloc() function? I think it’s possible. If the dotnet runtime is itself written in C, then the runtime (including the  memory library) is compiled/linked with a C compiler/linker. It’s therefore possible to use the malloc library.
Alternatively, I guess it’s also possible to create a custom memory module with an equivalent function to malloc(). I make this claim because Microsoft created its own version of C compiler and C memory management library since the early days of C.
Subject: RE: question on heap memory allocation
Does this use malloc ?
Subject: question on heap memory allocation
Hi Nigel,
Someone speculated that “in any programming language, heap memory is allocated always using the C function malloc(). There’s no alternative.” I know JVM and dotnet runtime are both written in C/c++ so at run time, heap memory is grabbed from OS probably by malloc(). Not sure about other languages.
Is my understanding correct?

## 6 kinds of q[ del ] in python

(Note in most quick and dirty scripts, we seldom need to delete stuff. In other scripts, deletion is much less common than insertion.) If you are overwhelmed, just remember Most common uses are dict-delete and list-delete-by-index.

1) del myDict[“some key”] # syntactically closer to List-delete than Attribute-delete

2) del myList[index] # [Note A]
2b) del myList[ sliceStart : sliceEnd ]

[A] myList.remove(someValue) is deletion by value, not by index

——— below are more advanced (read “obscure”)
3) del myVar # removes the name “myVar” but not necessarily the object. Note this is Not related to
3b) __del__(self) # like java finalizer. I think this runs only when the ref count drops to zero

4) del myObj.myAttr # same as
4b) __delattr__(self, myAttr) # basically remove from the idic

5) these are defined in built-in dict and list classes
__delitem__(self,index_or_key) # implements list-delete or dict-delete
__delslice__ # implements slice-delete

6) advanced Descriptor-delete
__delete__(self, instance)

DB performance – focus on read or write@@

1) Concurrent Write is the focus of DB performance tuning in some low-latency apps, esp. execution engines at the heart of ECNs and broker/dealer systems. Every execution must be permanently recorded (transactional) before sending out responses to counter-parties and downstream. DB read is avoided with caching.

1b) For market data engines, non-transactional DB write is still slower than flat file write. If a flat-file-write operation is A, then a database write operation involves A+B. If you don’t need the DB features, then you don’t need B. A is always faster than A+B.

2) Read is the focus of DB performance in other domains where data volume is huge — search, reporting, ERP, back office, asset management, BI, Data warehouse,

Those are the 2 domains I see more often. There are other contexts —

Write can be the bottleneck in high volume batch ETL. We used to rely on sybase bcp.

python tuples aren’t waterproof immutable

–Based on
t = ([1],[2])
# apply the id() function to each item in t
[47259925375816, 47259925376392]

([1, 0], [2])
map(id,t) # unchanged
[47259925375816, 47259925376392]

So tuple deviates from java immutability, which mandates t[0] returning a clone — essentially copy-on-write.

A tuple is like an ordered club roster written with indelible ink. The members of the club may change jobs, age, salary etc but the roster remains the same: same members, same SSN like python id(), same ranking.

In C++ lingo, the “t” tuple has 2 pointers on its real estate. It qualifies as immutable since the two 32-bit fields remain _bit_wise_constant_. The pointees live outside the tuple’s real estate and are Editable.

non-trivial financial data structure in C/C++

Use a struct to represent an umbrella object like an IRS contract. One of the members could be another struct for a schedule. In this case the inner struct is embedded physically in the umbrella struct’s memory footprint. Size of the umbrella includes the inner struct. I believe this is like C# value type. No such thing in java.

You can also use pointer to link the 2 structs, in the java HAS-A fashion. Size of the umbrella object could be smaller than the nested struct.

I think in some cases boost::tuple might be more manageable and more productive.

Note STL is not mentioned!

risk free bond isn’t risk-free

(Thanks to Eric for the enlightenment.)

Risk free treasury is credit-risk free, but not rate-risk free.

If you buy a large quantity of 30-year long bond at $99 and hold it till maturity, and valuation drops steadily from $99 to $98, $97… then your balance sheet will reflect an unrealized loss every year, a large loss if the position is large.

Many people actually looks at your balance sheet. So this large unrealized loss will affect your company's financial health.

If you use this bond as collateral, the depreciation will affect its collateral value. If you borrow money using this collateral and lender demands additional collateral due to depreciation, this risk-free bond could make you bankrupt.

Compared to equities, risk-free bonds won't lose the entire value. You will get back the face value if you hold it to maturity, but your investment has a poor return relative to the prevailing interest rates available on the market. Suppose your coupon is 100bps, but interest rate moves up after your purchase. If you don't liquidate the losing position, you would keep earning the 100bps while other long-term investments would have fetched 200bps returns.

other sound byte reminders — empathy, attitude..

2012 – make people feel comfortable with you
2011 – imagine most people’s average antenna is much longer
2011 – Identify your eccentricities and unlearn — conform. Shift midpoint of your spectrum of normalcy.
2010 – Make your manager look good
2004 – “Shrink” and “Be invisible
2006 – Identify role models and imitate
2005 – Assume other people are potentially intolerant, cynical, over-sensitive, fragile, inferiority-stricken, and won’t give me the benefit of doubt
2000 – Focus the next 14 days on making someone truly happy

Seek weekly/monthly feedback — GS career advice
empathy, protocol, etiquette
Be a “teammate” rather than a mere co-worker
Be open to bitter medicines, hard-to-swallow medicines

slist: remove consecutive dupes #Macq

struct litem {
     char data;
     litem* next;
void myDelete(litem * i){
  delete i;
int remove_consecutive_duplicates( litem*& list ){
  vector trash; // collection of bad nodes' address, to be DELETE'd
  litem* keep = list; // this pointer only points to “good” nodes to keep
  for (litem* p = keep->next; p;  p=p->next ){
    if (keep->data != p->data) {
      keep->next=p; // often unnecessary, but very cheap
  keep->next = 0; // keep is now the last good node. It may or may not point to null. This  is cheap insurance.
  int ret = (int) trash.size();
  std::for_each(trash.begin(), trash.end(), myDelete);
  // now, trash.size() == ret probably, as trash holds a bunch of stray pointers, but just in case size changes in the foreach loop, we save the original size in advance.
  return ret;
void dump( litem*& list){
    for (litem * p=list; p; p=p->next){

data< “;
int main23(){
   litem* a = new litem();
   litem* a2 = new litem();
   litem* a3 = new litem();
   litem* b = new litem();
   litem* b2 = new litem();
   litem* b3 = new litem();
   litem* b4 = new litem();
   litem* c = new litem();
   litem* c2 = new litem();
   litem* c3 = new litem();
   a->next = b;
   b->next = c;
   c->next = c2;
   c2->next = a2;
   a2->next = b2;
   b2->next = b3;

Fwd: prod support emails; learning big codebase (Citi munu)

category: GTD, cope


In your first 3 months, do you have bandwidth to study half the prod support emails? What type of issues do you choose to look into?

I first chose to follow those issues related to my project and my sub-system. It turned out my sub-system shares a lot of common code with other sub-systems, common code like messaging, caching, javascript, env set-up in ksh…. I was reluctant to look into them. In hindsight, I feel it’s best not to get hung up on those since they couldn’t help me show progress in the first 2 weeks. Expectation on me (from the Lab49 project manager) was to show progress every week.

I chose to follow issues about those relatively “simpler” modules. I chose to give up on any discussion thread I can’t easily understand, as they require too much investigation, and I can’t afford to spread myself too thin. This might be a poor learning habit.

I chose to focus on the autosys setup, because that’s how our servers are started, so I can trace to see the start-up parameters.

I realized some colleagues are busier than others, so I chose to bug those (which?) colleagues and spend more time on the subsystems they know better. How fast I learn is proportional to how much help I get.

I spent a lot of time setting up remote debugging so I could investigate a live UA server by myself. This is a big investment of my time, and manager doesn’t always see the value. I took the risk of this investment. I think it paid off.

I have a poor habit of focusing too much on environment set-up including IDE, build process, access to log files, access to database. I guess many fast learners do not get hung up on those. Now I feel as consultants I must deliver from the first week or so. If I invest time upfront setting up my tools and environment, it might help my productivity later, but initially it will hurt my productivity. One of my managers (the Lab49 project manager) had just a few weeks to form an impression of me as somewhat slow.

You mentioned once that due to the asynchronous nature, you spent lots of free time text-searching in the codebase. This is another personal investment that may not bear fruit? But it did bear fruit?

Generally, I found it challenging to understand most prod support issue. They always involve some system knowledge unknown to me. The learning process is a snowball process – you build up a good understanding of some system, and use that knowledge to understand related systems.

How do you start your snowball?

OutOfMemory ≠ allocation failure

IBM says —
Q: I am getting an OutOfMemoryError. Does this mean that the Java heap is exhausted?
A: Not necessarily. Sometimes the Java heap has free space but an OutOfMemoryError can occur. The error could occur because of
* out of memory but not due to a call to new()
* Excessive memory allocation in other parts of the application, unrelated to the JVM, if the JVM is just a part of the process, rather than the entire process (JVM through JNI, for instance).

Q: How can I confirm if the OutOfMemoryError was caused by the Java heap becoming exhausted?
A: Run with the -verbosegc option. VerboseGC will show messages such as “Insufficient heap space to satisfy allocation request”

Q: When I see an OutOfMemoryError, does that mean that the Java program will exit?
A: No. My experiments below show you can continue even after new() failed.
    static ArrayList list = new ArrayList();
    public static void main(String[] kkkj) throws InterruptedException {
        byte[] array = null;
        for (;;) {
            out.println(Sizeof.total_ie_Committed() + “\t>  ” + Sizeof.used()
                    + ” <– bytes in use. Trying to new up….");
            try {
                array = new byte[20 * 1000 * 1000];
            } catch (OutOfMemoryError e) {
                StringBuilder stat = Sizeof.printHeapSegments4JDK6();

CDS trading platform – IV

%%Q: how is JNI used here?
A: some calculation module is in C; some external API is in C. It’s a pain to rewrite things into java, but “we do want to”.

%%Q3: how does a CDS dealer hedge its exposure after it does a deal with a client?
A: It creates an offsetting deal with someone else

%%Q3b: but the dealer can’t disappear from the scene, right?
A: right. Consider an original mtg lender who sells the mtg off to Fannie. The mtg is completely off its books, but every month it still collects the installment. Similarly in the CDS case, the dealer still collects the quarterly premium, with or without the hedge.

Q: personal experience with MOM tuning?
%%A: message rate; subscriber population (RV handles this gracefully); message size and depth
%%A: one slow subscriber can cause message build-up in the broker, but TTL can help.

Q: how is quick sort used on array list and linked list?
%%A: random access needed. In STL, the sort function template is usable on only selected containers, whereas linked list (and other containers) has its own sort() method because the standard sort function is inapplicable.

remove constness from a char const* #Macq

This is a tested solution to an investment banking c++ interview question . I forgot to add the const for L/R. Note there’s no need to cast away constness. L is a reseatable ptr to a const char.

bool is_palindrome(char const * str){
  if ( !strlen(str) ) return true; // 0 size string is considered palindrome
  char const* L= str; // left ptr
  char const* R= str -1 + strlen(str); // the char before the . right ptr

  for (;L<R; L++, R–){
    while( tolower(*L) <'a' || 'z'< tolower(*L)) L++; // skip all non-alphabets
    while( tolower(*R) <'a' || 'z'< tolower(*R)) R–; // skip all non-alphabets
    if (*L != *R) return false;  
  return true;
int main(){
   bool ret = is_palindrome(“ab ca”);

Tick data repository – real-time/historical mkt-data

I think this was a CS interview…

Used for 1) back-testing, 2) trading signal generation, even if no real order sent, and 3) algorithmic real-$ trading.

First target user is back testing. Users would also try the 1 -} 2 -} 3 in that sequence.

Stores FX, FI and equity market data. Therefore the system treats everything just as generic tick data either quote ticks or trade ticks.

Multiple TB of data in memory. Time-series (not SQL) database.

Created using c++ and java. Probably due to sockets.

IR drv trading system

IRS is the primary product in an IRD desk.

Major components of an IRD trading system – 1) booking/capture (b/c), 2) Risk/PnL. Actually, Risk is not a desk-specific system. Rather, deal pricing/risk analytics system is cross-asset and covers IRD, bonds, structured products, FX, and used to cover Credit.

B/C is the primary app for traders. For strategists, primary app is pricing/risk system (c++). For operations users, B/C offers STP confirms and settlement i.e. periodic payments.

B/C system actually expands to cover the full life cycle of a deal – resets, assignments, unwinds

Key object is a trade or a deal or contract. Firmwide risk engine and pricing engine (for IRS bids/offers) need these “positions”. For real time risk and real_time_pricing, all deals need fast processing.

%Q: 2nd biggest product after IRS?
A: FRA, but we also trade cross-currency swaps, swaptions, caps/floors
A: There are flow IRD deals and exotic, “structured” instruments. Exotics are low volume, high margin. But momentum is in the flow business — Growing volume. Arms race.

Even the commonplace contract in the flow business could be more complex than a stock/bond trade. Customer wants a tailor-made deal , so deal is complex – with 100+ attributes, compared to 20+ for trades by other desks.

IRS has no secondary market. The only way to get out of a contract is unwind.

There’s regulatory pressure to clear IRS through a clearing house, possibly CME.

There’s close link btw Treasury desk and IRS desk. When trader takes a T position, she (tries to?) immediately enters an IRS deal.

Flow IRD Margin is better than Treasuries and FX, and trade volume is lower.

IRD Contracts with collateral and margin might become more widespread when IRS gets standardized with a central clearing house. Right now, some of these contracts have daily revaluation and daily settlement, but in a separate margin account.

%Q: 24-hour?
A: no. Users sit in US, Europe, Asia, South America but NY is the last office to close the day.

mkt-data engines hate pre-java8 autoboxing

(See also post on market data and size of Object)

High volume market data systems deal with primitives like ints, char-arrays and occasionally bit-arrays These often need to go into collections, like vector of int. These systems avoids DAM allocation like a plague.

Specifically, Java collections auto-boxing results in excessive memory allocation — a new object for every primitive item. A major justification for c++ STL, according to a CS veteran.

Sun also says autoboxing is inappropriate for high performance.

C# generic collection is free of autoboxing, just like STL, so is the primitive streams in java 8. But I don’t know if market data systems has actually embraced primitive streams.

MS IR drv IV

Q: how do you test a critical change to your market-making logic?
%A: mostly based on experience
%A: guided by user test cases + dev test cases
%A: brain storm on corner cases

Q: any size limit on TCP/UDP data transfer?
Q: UDP packet sequencing?
Q4: what do u know about class loaders?
Q4b: how can a singleton be “broken” due to class loaders, if there’s the up and down rule?
Q: How does J2EE uses class loaders?
Q: define scalability?
Q: real time features in your system?
Q: why do issuers want to recall a bond?
%A: to stop paying high coupons and exploit the current low interest rate environment

Q: why is bond recall a disaster for a holder?
%A: The high coupons were a windfall.

Q: what’s a bond with options?
Q(probably a ticket data question): NFS vs SAN

some Design questions at my interviews

Q: when you start a green field project and multi-threading is required, what are the key points you keep in mind about multi-threading.
A (my Answer): keep things really simple. For example, immutables; fewer locks; less granular locks;
A (now i think): have an idea where MT can really help and where it has marginal impact. In many cases ST works better, without any data races or locks.
A: It’s also possible to go for multi-processing instead of MT — lower risk (fewer MT hazards); simpler; better-understood
A: sometimes it makes sense to leverage on existing MT features in a database such as running code in the DB

Q: in your sybase/java/perl system at GS, what are the performance bottlenecks? How did you address them?
A: DB query.

Q (MS): how do you define scalability, in your specific contexts?
%%A: generically, it means nearly linear throughput improvement without code change.

Q: what strengths/complaints do you know about C++ STL?
A (my Answer): STL is more powerful than anything before or after it, but too complex. For example, I can have a vector of reference to pointer to pointer to int. No such thing allowed in java.
A(my Answer): STL debugging is harder
A (now i think): not much OO, but heavy usage of templates. Some people like/dislike it
A (now i think): thread-safety is minimal
A (now I think): storing pointers in containers is widely required but not well-supported without smart pointers.
A (now I think): storying interface or base types is widely required in Java but similar situation as above
A: performance very good.

Q(BGC): key differences between Perl and c++?

option valuation is never $0 (due to time-value and vol)

basic question: look at the option-payoff/asset-price curve of a call option. It’s flat till X then it rises linearly with asset price. But in-the-money and out-of-the-money options both have some fluctuating value. timeValue is alwyas positive, even with $0 intrinsicVal (out of the money). Why?

A: an out-of-the-money call has $0 valuation only on the expiry date or 0 volatility. Before expiry, it has a non-zero chance (due to vol) to become in-the-money. It has a non-zero valuation.

Imagine someone selling a deep out-of-the-money eurodollar call optoin (op on futures) with K=5000 bps/yr. K is too high, but what if the option premium is $0 ie free? I would “buy” it since there’s some chance eurodollar futures contract price (expressed in bps) may escalate to that level.

baptism of fire – get a team lead role

Given the weakness in empathy, protocol, etiquette… there’s a Contrarian suggestion to actively get into a leadership/coordinator/client-facing role in a big bank.

Will I become a bull in a china shop? Look at %%history.

Build track record, hard evidence — builds self-confidence and also credibility.

Get on the offensive rather than the defensive. I’m sick and tired of listening to wise (and caring) friends’ advice what I am not good at. The more I try to avoid mistakes, the more mistakes I see in myself, the lower my self-image. In fact, I left GS not just because of empathy, but more due to salary and productivity.

For example, I have more lunch meetings with all kinds of friends. So does Mithun, though we aren’t very empathetic people.

baptism of fire.

However, China and Singapore leadership roles might be too dangerous, where the magnitude of damage is probably too high.

My GS team lead isn’t good with etiquette and protocol but he gets things done. I think users appreciate him and one of his bosses adores him.

Treasury algo quoting engine #CS

1) In market-making, a quoter engine is driven by a specific model (among many models), and trades using a model account. It reacts to market data [1], and based on inventory will publish/withdraw bids and offers – collectively known as quotes. In theory, a quoter could maintain quotes at different price levels, but more typically, a quoter maintains multiple small[2] quotes at the same price level, but why? [3]

2) Beside market-making, another model is RFQ or bid-wanted / offer-wanted. See post on [[2 ways to sell a security]]. In treasury market, a RFQ is often sent to a small number of dealers only.

3) In terms of alpha strategy, there are 2 main “views” of price movement – mean-reverting vs trending. These are “views” on the market.

Same system can trade T futures, but it’s more complex due to the basket.

[3] A: those different quotes will join the order queue at different queue-positions. They will execute at different times. You can withdraw them individually.
[1] interdealer – BrokerTec and eSpeed; dealer-customer markets like tradeweb; orders from private clients not on the open market,
[2] low quantity, low risk — fine grained

Since each treasury position ties up a large sum of money, automatic financing is important. No margin as far as I see.
– After you buy a bond, automatically the bond is lent out on the repo market, so trader can use that money to pay the counter party.
– After you short sell a bond to get $20M, automatically you borrow the bond on the repo market, using the $20M proceeds.

practical sort algorithms in popular languages

Important for …. real world software, interviews..

# 1) mergesort — chosen as
– primary sorter of java,
– primary sorter of perl,
– primary sorter of python

#2) quicksort — chosen by
– primary sorter of dotnet’s List.sort()
– primary sorter of STL (in a modified form — introsort)

heapsort – used in embedded, also part of the market-leading introsort
[iv] counting sort? important to natural words and numbers?
radix sort — important to natural words and numbers

# eor
[iv] bubble sort — easiest to code in an interview:)
[iv] insertion sort — simple
[iv] bucket
[iv] counting
[iv] radix


credit risk analysis of collateral valuation

Hi XR,

You told me most of the credit risk analysis is on collateral. I guess a client hands over some bond to BofA as collateral, in order to secure some loan? I know collateral is a very common requirement of many banking deals. One of the most common is a commercial bank loan.

In that case, how often is the collateral revalued? In margin systems, assets in the margin account is revalued daily, and the bank (the lender) could make “margin calls” on a daily basis.

In your case, if a collateral has a reduced valuation (say 50% lower), what does BofA do? Clearly the bank is at risk when the collateral it holds is almost worthless. I believe any serious banker would immediately demand additional collateral. Is there some system that automates the decision making flow? First system must gather the collateral data, then revalues each, then compare with the collateral requirement for each deal. If new valuation is lower than required collateral, then system need to inform the bankers or traders of the exposure, vulnerability and credit risk. How is this done in your case?


Condition^Lock objects in pthreads and java

I feel Windows has a different implementation altogether…

Pthreads (and many threading systems) employs a classic 2-object design for thread synchronization:

1) a mutex lock object. Think of it as a bathroom.
2) a condition object aka condition variable. I call it a waiting room.

In Pthreads, Condition object offers

* wait() and timed_wait() — similar to Object.wait()
* signal() and boradcast() — similar to Object.notify()

In POSIX, if Thread A wants to call these methods on the condition object, then A must hold the mutex. Therefore the 2 objects are tightly coupled.

Java 1.5 provides Lock and Condition objects. 1.5 lets you have 2 “waiting rooms” tied to a single mutex.

  Condition cond1 = myLock.newCondition();
  Condition cond2 = myLock.newCondition();

Now you can target your notification to a subset of waiting threads, rather than indiscriminate notifyAll()

In java 1.4, you grab myMutext and call myMutex.wait(). Single-object design.