socket shared between 2 processes

Common IV question: In what scenarios can a socket be shared between 2 processes

Background — a socket is a special type of file descriptor (at least in unix). Consider an output file handle. By default, this “channel” isn’t shared between 2 processes. Similarly, when a packet (say a price) is delivered to a given network endpoint, the kernel must decide which process to receive the data, usually not to two processes.

To have two processes both listening on the same network /endpoint/, one of them is usually a child of the other. The webpage in [1] shows a short python code illustrating this scenario. I tested. q(lsof) and q(ss) commands both (but not netstat) show the 2 processes listening on the same endpoint. OS delivers the data to A B A B…

https://bintanvictor.wordpress.com/2017/04/29/so_reuseport-socket-option/ shows an advanced kernel feature to let multiple processes bind() to the same endpoint.

For multicast (UDP only) two processes can listen to the same UDP endpoint. See [3] and [2]

A Unix domain socket can be shared between two unrelated processes.

See

[1] http://stackoverflow.com/questions/670891/is-there-a-way-for-multiple-processes-to-share-a-listening-socket

[2] http://stackoverflow.com/questions/1694144/can-two-applications-listen-to-the-same-port

[3] http://www.tldp.org/HOWTO/Multicast-HOWTO-2.html

Advertisements

SO_REUSEPORT socket option – hungry chicks

With SO_REUSEPORT option, multiple TCP server processes could bind() to the same server endpoint. Designed for the busiest multithreaded servers.

http://i.dailymail.co.uk/i/pix/2011/03/03/article-1362552-0D7319F3000005DC-882_634x357.jpg – a bunch of hungry chicks competing to get the next worm the mother delivers. The mother can only give the worm to one chick at a time. SO_REUSEPORT option sets up a chick family. When an incoming connection hits the accept(), kernel picks one of the accepting threads/processes and delivers the data to it alone.

See https://lwn.net/Articles/542629/  + my socket book P102.

joining/leaving a multicast group

Every multicast address is a group address. In other words, a multicast address identifies a group.

Sending a multicast datagram is much simpler than receiving…

[1] http://www.tldp.org/HOWTO/Multicast-HOWTO-2.html is a concise 4-page introduction. Describes joining/leaving.

[2] http://ntrg.cs.tcd.ie/undergrad/4ba2/multicast/antony/ has sample code to send/receive. Note there’s no server/client actually.

 

2 Active connections on one server socket

This is not the most common design, but have a look at the following output:

remote          local        state
*:*           - 4.3.2.1:5000 LISTENING
1.2.3.4:12345 - 4.3.2.1:5000 CONNECTED
4.5.6.7:83247 - 4.3.2.1:5000 CONNECTED

What needs to be unique, is the 5-tuple (protocol, remote-ip, remote-port, local-ip, local-port).

The listening TCP server will start a new thread/task/process on each incoming connection.

The “accept, move the connection to a dedicated server endpoint, then go back to accept()” is probably another design.

http://stackoverflow.com/questions/11129212/tcp-two-different-sockets-sharing-a-port also says

Multiple connections on the same TCP server can share the same server-side IP/Port pair as long as they are associated with different client-side IP/Port pairs

##some@the significant coding IV experiences

Sugg: practice with additional (real!) cpp coding IV. No pain no gain. No shortcuts.

“webex” is shorthand for any form of remote screen sharing tool

Lang onsite? IDE? firm P/F
C webex paper bbg P 2017
J onsite paper BGC P 2017
J home IDE BGC -connected P 2017
J home IDE HSBC P 2017
J home IDE pimco – iterator P 2017
J onsite paper pimco -Zoltan P 2017
c webex paper Citadel 😦 2017 ez question
py webex paper Broadway -hashtable P 2017
J home codility Baml – FX options 😦 2015 too lazy
c home codility Jump 3rd 😦 2015
c home IDE jump – order book P 2012
c home IDE Tetris 😦 2015 coding style poor
c webex paper bbg -London 😦 2015 unable to understand the Q
c onsite paper bbg -2nd time 😦 2011
c# webex paper Reuters – eikon P 2013
swing webex IDE Barclays – Barx P 2012
j onsite IDE RBC – Korean mgr p 2010

append.bashrc.txt

 

<< ______________end_of_comment_1________________
modified [28 Apr 2017]
______________end_of_comment_1________________
[ -z "$(echo $- | grep i)" ] && return # this same line should be on top of append.bashrc_citi.txt
# # # # # # # # # # # # # # # # # #
#
# << var assignment >> should be first
#
# # # # # # # # # # # # # # # # # # # # # #
export PS1='\n\s!\! \u@\h [ \t \d ] \w/ \n\$ '
export EDITOR=vi # primarily for sqlplus
export VIMINIT=":se nu | :map v :w" 
export EXINIT=$VIMINIT # ignored if .vimrc found!
LESS="--ignore-case"
export TMOUT=987654; [ $root ] && export TMOUT=100600
HISTSIZE=900
set -o emacs
set +o nounset
export PATH=/usr/xpg4/bin:$PATH; [ $(id -u) = 0 ] && root=yesur 
set +o noclobber; [ $root ] && set -o noclobber
sav=$LOGNAME
# # # # # # # # # # # # # # # # # # # # # #
#
# PAGER and TERM:
#
# # # # # # # # # # # # # # # # # # # # # #
export PAGER=more
export PAGERE=more
if [ -z "$(which less |grep '^no less')" ]; then
 export PAGER="less" # breaking some x terminals but ok in "console"
 export PAGERE="less -E" # -E needed for dir-viewing, file-viewing but affects man 
fi
export TERM=vt100 # 'linux' breaks vi in gnome-terminal; 'linux' imperfect for solaris
# # # # # # # # # # # # # # # # # # # # # #
#
# << aliases >> ... sortable
#
# # # # # # # # # # # # # # # # # # # # # #
alias %=cd_l # typing trainer
alias ..='cd_l ..'
alias cp='cp -i' # defuse
alias egi='export |grep -i'
alias h=history
alias hgi="history |grep -i"
alias j1="fg %1"
alias j2="fg %2"
alias killthem="perl -pe 's/^\S+(\s+\d+).*/\1/ or s/.*//'|xargs -pt kill" # p str1 | killthem
alias m=$PAGER
alias mv='mv -i' # defuse
alias p=ps_grep
alias path=" echo \$PATH |perl -pe 's/:/\n/g' |sort -u|$PAGERE "
alias rm=myrm
alias s=' source $HOME/.bashrc '
alias t='l -t'
# # # # # # # # # # # # # # # # # # # # # #
#
# << functions >> ... difficult to sort
#
# # # # # # # # # # # # # # # # # # # # # #
cd_l(){
 [ $# -eq 0 ] && l && return
 [ -n "$( file $* | grep directory )" ] && cd $* && l && return
 [ -n "$( file $* | perl -ne 'print if /text|script/' )" ] && m $* &&
/bin/echo -e "\n\n" && l $*
}
d(){
 [ $# -ne 0 ] && cd $*
 [ `pwd` = '/' ] && target=/ && echo I am in /
 echo "In MB :"
 eval du -ks * |sort -n|perl -pe 's|^(\d+)|$1/1000|e'
}
notinuse_g(){ # bug with g -v
 pattern=$1; shift
 cmd=" grep -i \"$pattern\" $* "
 /usr/bin/printf '%s\n' "cmd=__$cmd __"
 eval $cmd |$PAGERE
}
l(){ # can't be completely replaced by 'cd_l' or ']', because "cd_l -tr dir1" is confusing and should be avoided
 /bin/ls -alFs $* |$PAGERE
}
myrm(){
 cmd="mv -i $* /var/tmp/$sav/ "
 /usr/bin/printf '%s\n' "cmd=__$cmd __"
 eval $cmd
}
ps_grep(){
 ## cmd1='/bin/ps -ef' # truncation risk
 # ps auxwww : inconsistent for root vs non-root
 cmd1='/usr/ucb/ps auxwww' # |grep -v grep -- no cos some real commands contain grep
 for f in $*; do
 cmd1="$cmd1 | g $f"
 done
 eval $cmd1
}
sav(){ # proven since 2007
 suffix=$RANDOM$(date +"%H%d%b") # to avoid misleading readers,make the suffix ambiguous
 for f in $*; do
 f=$(echo $f|perl -pe 's|/$||') # sav dir/
 b=`basename $f`
 (
 cd `dirname $f` ; 
 tar cf - $b | (cd /tmp; tar xpf -)
 /bin/mv -i /tmp/$b $b.b4${sav}edit$suffix
 [ -d $f ] && opt=' -d' 
 eval ls -lt $opt $b*
 )
 done
}
# # # # # # # # # # # # # # # # # # #
#
# << exa group of companies >> ... an exercise in grouping
#
# # # # # # # # # # # # # # # # # # # # # #
exa(){ #testing: exa -t ss ~/.ssh
 local ops
 while is_opt=`echo $1 |perl -ne 'print if /^-/' ` && [ "$is_opt" ]; do
 ops="$ops $1"; shift
 done

 if [ $# -eq 1 ]; then
 fullpath=$1
 shortname=`echo $1|perl -pe 's|/$||;s|^.*/||' `
 else
 fullpath=$2
 shortname=$1
 fi

 [ -x $fullpath ] || return
 # export "$shortname"="$fullpath"
 set $shortname="$fullpath" # not working as of 2017
 eval $shortname="$fullpath"
 prj="$fullpath"
 alias "$shortname"="cd $fullpath;l $ops"
 alias prj=$shortname # USE MORE
}
exa /tmp/
# # # # # # # # # # # # # # # # # # #
#
# << path stuff, potentially dangerous >>
#
# # # # # # # # # # # # # # # # # # # # # #
add_path(){ 
 [ -r $1 ] || return
 [ "$(echo $PATH|grep :$1)" ] && return
 # check world write perm, esp for root
 PATH=$PATH:$1
}
set_manpath(){ # very slow on some systems. Run this when u need it.
 for d in `ls -d /[uo]*/*/man /[uo]*/*/*/man /[uo]*/*/*/*/man`; do
 export MANPATH=$MANPATH:$d
 done
}
set_path(){
 add_path /usr/sbin
 add_path /usr/bin
 add_path /sbin
 add_path /usr/local/bin
 add_path /usr/cluster/bin
 add_path /usr/openwin/bin/
 add_path /usr/ucb
 add_path $ORACLE_HOME/bin
 add_path /usr/sfw/bin # pre-installed
 add_path /opt/csw/bin
 [ $root ] && return
 add_path . 
 add_path $HOME/bin
}
set_path

2H life-changing xp# income,home location,industry…

Here’s a real story in 2010 — I was completely hopeless and stuck in despair after my Goldman Sachs internal transfer was blocked in the last stage. I considered moving my whole family back to Singapore with an offer, and start my job search there. I was seriously considering a S$100k job in a back office batch programming job. Absolutely the lowest point in my entire career. After licking the would for 2 months, I started looking for jobs outside Goldman and slowly found my foot hold. Then in early 2010, I passed a phone screening and attended a Citigroup “superday”. I spent half an hour each with 3 interviewers. By end of the day, recruiter said I was the #1 pick. I took the offer, at a 80% increment. In the next 12 months, I built up my track record + knowledge in

  1. real time trading engine components, esp. real time pricing engine
  2. fixed income math,
  3. c++ (knowledge rebuild)

I have never looked back since. Fair to say that my family won’t be where we are today, without this Citigroup experience. With this track record I was able to take on relatively high-end programming jobs in U.S. and Singapore. I was able to live in a convenient location, and buy properties and send my kids to mid-range preschools (too pricey in hind sight). Obviously I wanted this kind of job even in 2009. That dream became reality when I passed the superday interview. That interview was one of the turning points in my career.

Fast track to 2017 — I had a 20-minute phone interview with the world’s biggest asset management firm (Let’s call it PP), then I had a 2-hour skype interview. They made an offer. I discussed with the recruiter the proposal —

  • I would relocate to California
  • I would get paid around 200k pretax and possibly with an increment in 6 months. PP usually increase billing rate after 12 months if contractor does well.
  • Unlike investment banks, PP has long-term contracts not subject to trading desk profits, so this contract is (supposedly) more stable. In a few years, I would consider buying a home in California
  • recruitment agency CEO said he would transfer my visa and sponsor green card.

If I were to take this offer, my life would be transformed. (I would also have a better chance to break into the  high tech industry in nearby silicon valley, because I would have local friends in that domain.) Such a big change in my life is now possible because … I did well [1] in the interview.

Stripped to the core, that’s the reality in our world of contract programmers.  Project delivery, debugging, and relationship with boss can get you promoted, but those on-the-job efforts have much lower impact than your performance during an interview. Like an NBA playoff match. A short few hour under the spot light can change your life forever.

This is not a rare experience. There are other high-paying contract job offers that could “change your life”, and you only need to do well in the interviews to make it happen.

I feel this is typical of U.S. market and perhaps London. In Singapore. contract roles can’t pay this much. A permanent role has a few subtle implications so I feel it’s a different game.

[1] The 7 interviewers felt I was strong in c++ (not really), java and sql, and competent in fixed income math (I only worked with it for a year). Unlike other high-end interviews, there are not many tough tech questions like threading, algorithms, or coding tests. I feel they liked my interview mostly because of the combination of c++/java/fixed income math — not a common combination.

tech advantages of c# over java#le2XR

Hi XR,

Based on whatever little I know, here are some technical advantages of c# over java.

(Master these c# feature and mention them in your next java interview 🙂

  • C# has many more advantages on desktop GUI, but today let’s focus on server side.
  • [L] generics —- c# generics were designed with full knowledge of java/c++ shortcomings. Simpler than c++ (but less powerful), but more complete than java (no type erasure). For example see type constraints.
  • [L] delegates —- Rather useful. Some (but not all) of its functionalities can be emulated in java8.
  • [L] c# can access low-level windows concurrency constructs such as event wait handles. Windows JVM offers a standardized, “reduced-fat” facade. If you want optimal concurrency on windows, use VC++, or c#.
  • [L] reflection —- is more complete than java. Over the years java reflection proved to be extremely powerful. Not sure if c# has the same power, but c# surely added a few features such as Reflection.Emit.
  • concurrency —- dotnet offers many innovative concurrency features. All high level features, so probably achievable in java too.
  • tight integration with COM and MS Office. In fact, there are multiple official and unofficial frameworks to write Excel add-ins in c#
  • tight integration with high-level commercial products from Microsoft like MSSQL, sharepoint
  • tight integration with windows infrastructure like Windows Services (like network daemons), WCF, Windows networking, Windows web server, windows remoting, windows registry, PowerShell, windows software installation etc
  • c# gives programmers more access to low-level windows system API, via unmanaged code (I don’t have examples). In contrast, Java programmers typically use JNI, but I guess the java security policy restricts this access.
  • probably higher performance than JVM on windows
  • CLR offers scripting languages VB.net, F#, IronPython etc, whereas JVM supports scripting languages javascript, scala, groovy, jython etc.

[L = low-level feature]

If you want highest performance on Windows, low-level access to windows OS, but without the complexity of VC++ and MFC, then c# is the language of choice. It is high-level, convenient like java but flexible enough to let you go one level lower when you need to.

Another way to address your question — listen to the the complaints against java. (Put aside the complaints of GUI programmers.)

Even if a (rational, objective) architect doesn’t recognize any of these as important advantages, she may still favor c# over java because she is familiar and competent ONLY in the Microsoft ecosystem. She could point out countless features in Visual Studio and numerous windows development tools that are rather different from the java tool set, so different that it would take months and years to learn.

Also, there are many design trade-off and implementation techniques built on and for Dotnet. If she is reliant on and comfortable in this ecosystem, she would see the java ecosystem as alien, incomplete, inconvenient and unproductive. Remember when we first moved to U.S. — everything inconvenient.

On a more serious note, her design ideas may not be achievable using java. So java would appear to be missing important features and tools. In a nutshell, for her java is a capable and complete ecosystem theoretically, but in practice an incomplete solution.

##%%offers 2017

$c2c co where primary tech other tech domain nlg duration
100 pimco NPB[1] c++11 🙂 🙂 🙂 java, possibly Hadoop 🙂 FI accrual math 🙂 3+
100 pimco NYC java framework 🙂 🙂 flexible
100+ bgc NYC java minimal cpp FX.. trading to perm 😦 😦 😦
below 100 😦 😦 Ravi Chgo 😦 😦 Qz 😦 😦 😦 java FI trading again flexible
perm Nitin NYC java perm

[1] A bit hard to get next job in NY, but helps me get a next job in West Coast. However, in terms of buying a home, I just don’t know.