socket shared between 2 processes

Common IV question: In what scenarios can a socket be shared between 2 processes

Background — a socket is a special type of file descriptor (at least in unix). Consider an output file handle. By default, this “channel” isn’t shared between 2 processes. Similarly, when a packet (say a price) is delivered to a given network endpoint, the kernel must decide which process to receive the data, usually not to two processes.

To have two processes both listening on the same network /endpoint/, one of them is usually a child of the other. The webpage in [1] shows a short python code illustrating this scenario.

https://bintanvictor.wordpress.com/2017/04/29/so_reuseport-socket-option/ shows an advanced kernel feature to let multiple processes bind() to the same endpoint.

For multicast (UDP only) two processes can listen to the same UDP endpoint. See [3] and [2]

A Unix domain socket can be shared between two unrelated processes.

See

[1] http://stackoverflow.com/questions/670891/is-there-a-way-for-multiple-processes-to-share-a-listening-socket

[2] http://stackoverflow.com/questions/1694144/can-two-applications-listen-to-the-same-port

[3] http://www.tldp.org/HOWTO/Multicast-HOWTO-2.html

SO_REUSEPORT socket option – hungry chicks

With SO_REUSEPORT option, multiple TCP server processes could bind() to the same server endpoint. Designed for the busiest multithreaded servers.

http://i.dailymail.co.uk/i/pix/2011/03/03/article-1362552-0D7319F3000005DC-882_634x357.jpg – a bunch of hungry chicks competing to get the next worm the mother delivers. The mother can only give the worm to one chick at a time. SO_REUSEPORT option sets up a chick family. When an incoming connection hits the accept(), kernel picks one of the accepting threads/processes and delivers the data to it alone.

See https://lwn.net/Articles/542629/

joining/leaving a multicast group

Every multicast address is a group address. In other words, a multicast address identifies a group.

Sending a multicast datagram is much simpler than receiving…

[1] http://www.tldp.org/HOWTO/Multicast-HOWTO-2.html is a concise 4-page introduction. Describes joining/leaving.

[2] http://ntrg.cs.tcd.ie/undergrad/4ba2/multicast/antony/ has sample code to send/receive. Note there’s no server/client actually.

 

2 Active connections on one server socket

This is not the most common design, but have a look at the following output:

remote          local        state
*:*           - 4.3.2.1:5000 LISTENING
1.2.3.4:12345 - 4.3.2.1:5000 CONNECTED
4.5.6.7:83247 - 4.3.2.1:5000 CONNECTED

What needs to be unique, is the 5-tuple (protocol, remote-ip, remote-port, local-ip, local-port).

The listening TCP server will start a new thread/task/process on each incoming connection.

The “accept, move the connection to a dedicated server endpoint, then go back to accept()” is probably another design.

http://stackoverflow.com/questions/11129212/tcp-two-different-sockets-sharing-a-port also says

Multiple connections on the same TCP server can share the same server-side IP/Port pair as long as they are associated with different client-side IP/Port pairs

##some@the significant coding IV experiences

Sugg: practice with additional (real!) cpp coding IV. No pain no gain. No shortcuts.

“webex” is shorthand for any form of remote screen sharing tool

Lang onsite? IDE? firm P/F
J onsite paper BGC P 2017
J home IDE BGC -connected P 2017
J home IDE HSBC P 2017
J home IDE pimco – iterator P 2017
J onsite paper pimco -Zoltan P 2017
c webex paper Citadel 😦 2017 ez question
py webex paper Broadway -hashtable P 2017
J home codility Baml – FX options 😦 2015 too lazy
c home codility Jump 3rd 😦 2015
c home IDE jump – order book P 2012
c home IDE Tetris 😦 2015 coding style poor
c webex paper bbg -London 😦 2015 unable to understand the Q
c onsite paper bbg -2nd time 😦 2011
c# webex paper Reuters – eikon P 2013
swing webex IDE Barclays – Barx P 2012
j onsite IDE RBC – Korean mgr p 2010

append.bashrc.txt

 

<< ______________end_of_comment_1________________
modified [28 Apr 2017]
______________end_of_comment_1________________
[ -z "$(echo $- | grep i)" ] && return # this same line should be on top of append.bashrc_citi.txt
# # # # # # # # # # # # # # # # # #
#
# << var assignment >> should be first
#
# # # # # # # # # # # # # # # # # # # # # #
export PS1='\n\s!\! \u@\h [ \t \d ] \w/ \n\$ '
export EDITOR=vi # primarily for sqlplus
export VIMINIT=":se nu | :map v :w" 
export EXINIT=$VIMINIT # ignored if .vimrc found!
LESS="--ignore-case"
export TMOUT=987654; [ $root ] && export TMOUT=100600
HISTSIZE=900
set -o emacs
set +o nounset
export PATH=/usr/xpg4/bin:$PATH; [ $(id -u) = 0 ] && root=yesur 
set +o noclobber; [ $root ] && set -o noclobber
sav=$LOGNAME
# # # # # # # # # # # # # # # # # # # # # #
#
# PAGER and TERM:
#
# # # # # # # # # # # # # # # # # # # # # #
export PAGER=more
export PAGERE=more
if [ -z "$(which less |grep '^no less')" ]; then
 export PAGER="less" # breaking some x terminals but ok in "console"
 export PAGERE="less -E" # -E needed for dir-viewing, file-viewing but affects man 
fi
export TERM=vt100 # 'linux' breaks vi in gnome-terminal; 'linux' imperfect for solaris
# # # # # # # # # # # # # # # # # # # # # #
#
# << aliases >> ... sortable
#
# # # # # # # # # # # # # # # # # # # # # #
alias %=cd_l # typing trainer
alias ..='cd_l ..'
alias cp='cp -i' # defuse
alias egi='export |grep -i'
alias h=history
alias hgi="history |grep -i"
alias j1="fg %1"
alias j2="fg %2"
alias killthem="perl -pe 's/^\S+(\s+\d+).*/\1/ or s/.*//'|xargs -pt kill" # p str1 | killthem
alias m=$PAGER
alias mv='mv -i' # defuse
alias p=ps_grep
alias path=" echo \$PATH |perl -pe 's/:/\n/g' |sort -u|$PAGERE "
alias rm=myrm
alias s=' source $HOME/.bashrc '
alias t='l -t'
# # # # # # # # # # # # # # # # # # # # # #
#
# << functions >> ... difficult to sort
#
# # # # # # # # # # # # # # # # # # # # # #
cd_l(){
 [ $# -eq 0 ] && l && return
 [ -n "$( file $* | grep directory )" ] && cd $* && l && return
 [ -n "$( file $* | perl -ne 'print if /text|script/' )" ] && m $* &&
/bin/echo -e "\n\n" && l $*
}
d(){
 [ $# -ne 0 ] && cd $*
 [ `pwd` = '/' ] && target=/ && echo I am in /
 echo "In MB :"
 eval du -ks * |sort -n|perl -pe 's|^(\d+)|$1/1000|e'
}
notinuse_g(){ # bug with g -v
 pattern=$1; shift
 cmd=" grep -i \"$pattern\" $* "
 /usr/bin/printf '%s\n' "cmd=__$cmd __"
 eval $cmd |$PAGERE
}
l(){ # can't be completely replaced by 'cd_l' or ']', because "cd_l -tr dir1" is confusing and should be avoided
 /bin/ls -alFs $* |$PAGERE
}
myrm(){
 cmd="mv -i $* /var/tmp/$sav/ "
 /usr/bin/printf '%s\n' "cmd=__$cmd __"
 eval $cmd
}
ps_grep(){
 ## cmd1='/bin/ps -ef' # truncation risk
 # ps auxwww : inconsistent for root vs non-root
 cmd1='/usr/ucb/ps auxwww' # |grep -v grep -- no cos some real commands contain grep
 for f in $*; do
 cmd1="$cmd1 | g $f"
 done
 eval $cmd1
}
sav(){ # proven since 2007
 suffix=$RANDOM$(date +"%H%d%b") # to avoid misleading readers,make the suffix ambiguous
 for f in $*; do
 f=$(echo $f|perl -pe 's|/$||') # sav dir/
 b=`basename $f`
 (
 cd `dirname $f` ; 
 tar cf - $b | (cd /tmp; tar xpf -)
 /bin/mv -i /tmp/$b $b.b4${sav}edit$suffix
 [ -d $f ] && opt=' -d' 
 eval ls -lt $opt $b*
 )
 done
}
# # # # # # # # # # # # # # # # # # #
#
# << exa group of companies >> ... an exercise in grouping
#
# # # # # # # # # # # # # # # # # # # # # #
exa(){ #testing: exa -t ss ~/.ssh
 local ops
 while is_opt=`echo $1 |perl -ne 'print if /^-/' ` && [ "$is_opt" ]; do
 ops="$ops $1"; shift
 done

 if [ $# -eq 1 ]; then
 fullpath=$1
 shortname=`echo $1|perl -pe 's|/$||;s|^.*/||' `
 else
 fullpath=$2
 shortname=$1
 fi

 [ -x $fullpath ] || return
 # export "$shortname"="$fullpath"
 set $shortname="$fullpath" # not working as of 2017
 eval $shortname="$fullpath"
 prj="$fullpath"
 alias "$shortname"="cd $fullpath;l $ops"
 alias prj=$shortname # USE MORE
}
exa /tmp/
# # # # # # # # # # # # # # # # # # #
#
# << path stuff, potentially dangerous >>
#
# # # # # # # # # # # # # # # # # # # # # #
add_path(){ 
 [ -r $1 ] || return
 [ "$(echo $PATH|grep :$1)" ] && return
 # check world write perm, esp for root
 PATH=$PATH:$1
}
set_manpath(){ # very slow on some systems. Run this when u need it.
 for d in `ls -d /[uo]*/*/man /[uo]*/*/*/man /[uo]*/*/*/*/man`; do
 export MANPATH=$MANPATH:$d
 done
}
set_path(){
 add_path /usr/sbin
 add_path /usr/bin
 add_path /sbin
 add_path /usr/local/bin
 add_path /usr/cluster/bin
 add_path /usr/openwin/bin/
 add_path /usr/ucb
 add_path $ORACLE_HOME/bin
 add_path /usr/sfw/bin # pre-installed
 add_path /opt/csw/bin
 [ $root ] && return
 add_path . 
 add_path $HOME/bin
}
set_path

2H life-changing xp# income,home location,industry…

Here’s a real story in 2010 — I was completely hopeless and stuck in despair after my Goldman Sachs internal transfer was blocked in the last stage. I considered moving my whole family back to Singapore with an offer, and start my job search there. I was seriously considering a S$100k job in a back office batch programming job. Absolutely the lowest point in my entire career. After licking the would for 2 months, I started looking for jobs outside Goldman and slowly found my foot hold. Then in early 2010, I passed a phone screening and attended a Citigroup “superday”. I spent half an hour each with 3 interviewers. By end of the day, recruiter said I was the #1 pick. I took the offer, at a 80% increment. In the next 12 months, I built up my track record + knowledge in

  1. real time trading engine components, esp. real time pricing engine
  2. fixed income math,
  3. c++ (knowledge rebuild)

I have never looked back since. Fair to say that my family won’t be where we are today, without this Citigroup experience. With this track record I was able to take on relatively high-end programming jobs in U.S. and Singapore. I was able to live in a convenient location, and buy properties and send my kids to mid-range preschools (too pricey in hind sight). Obviously I wanted this kind of job even in 2009. That dream became reality when I passed the superday interview. That interview was one of the turning points in my career.

Fast track to 2017 — I had a 20-minute phone interview with the world’s biggest asset management firm (Let’s call it PP), then I had a 2-hour skype interview. They made an offer. I discussed with the recruiter the proposal —

  • I would relocate to California
  • I would get paid around 200k pretax and possibly with an increment in 6 months. PP usually increase billing rate after 12 months if contractor does well.
  • Unlike investment banks, PP has long-term contracts not subject to trading desk profits, so this contract is (supposedly) more stable. In a few years, I would consider buying a home in California
  • recruitment agency CEO said he would transfer my visa and sponsor green card.

If I were to take this offer, my life would be transformed. (I would also have a better chance to break into the  high tech industry in nearby silicon valley, because I would have local friends in that domain.) Such a big change in my life is now possible because … I did well [1] in the interview.

Stripped to the core, that’s the reality in our world of contract programmers.  Project delivery, debugging, and relationship with boss can get you promoted, but those on-the-job efforts have much lower impact than your performance during an interview. Like an NBA playoff match. A short few hour under the spot light can change your life forever.

This is not a rare experience. There are other high-paying contract job offers that could “change your life”, and you only need to do well in the interviews to make it happen.

I feel this is typical of U.S. market and perhaps London. In Singapore. contract roles can’t pay this much. A permanent role has a few subtle implications so I feel it’s a different game.

[1] The 7 interviewers felt I was strong in c++ (not really), java and sql, and competent in fixed income math (I only worked with it for a year). Unlike other high-end interviews, there are not many tough tech questions like threading, algorithms, or coding tests. I feel they liked my interview mostly because of the combination of c++/java/fixed income math — not a common combination.