TCP listening socket shared by2processes #fork

Common IV question: In what scenarios can a listening socket (in memory) be shared between 2 listening processes?

Background — a socket is a special type of file descriptor (at least in unix). Consider an output file handle. By default, this “channel” isn’t shared between 2 processes. Similarly, when a packet (say a price) is delivered to a given network endpoint, the kernel must decide which process to receive the data, usually not to two processes.

To have two processes both listening on the same listening-socket, one of them is usually a child of the other. The webpage in [1] and my code in https://github.com/tiger40490/repo1/blob/py1/py/sock/1sock2server.py show a short python code illustrating this scenario. I tested. q(lsof) and q(ss) commands both (but not netstat) show the 2 processes listening on the same endpoint. OS delivers the data to A B A B…

https://bintanvictor.wordpress.com/2017/04/29/so_reuseport-socket-option/ shows an advanced kernel feature to let multiple processes bind() to the same endpoint.

For multicast (UDP only) two processes can listen to the same UDP endpoint. See [3] and [2]

A Unix domain socket can be shared between two unrelated processes.

See

[1] http://stackoverflow.com/questions/670891/is-there-a-way-for-multiple-processes-to-share-a-listening-socket

[2] http://stackoverflow.com/questions/1694144/can-two-applications-listen-to-the-same-port

[3] http://www.tldp.org/HOWTO/Multicast-HOWTO-2.html

Advertisements

joining/leaving a multicast group

Every multicast address is a group address. In other words, a multicast address identifies a group.

Sending a multicast datagram is much simpler than receiving…

[1] http://www.tldp.org/HOWTO/Multicast-HOWTO-2.html is a concise 4-page introduction. Describes joining/leaving.

[2] http://ntrg.cs.tcd.ie/undergrad/4ba2/multicast/antony/ has sample code to send/receive. Note there’s no server/client actually.

 

2 Active connections on 1 TCP server socket

This is not the most common design, but have a look at the following output:

remote          local        state
*:*           - 4.3.2.1:5000 LISTENING
1.2.3.4:12345 - 4.3.2.1:5000 CONNECTED
4.5.6.7:83247 - 4.3.2.1:5000 CONNECTED

What needs to be unique, is the 5-tuple (protocol, remote-ip, remote-port, local-ip, local-port).

[[tcp/ip sockets in C]] P100 has a full section on this topic.

The listening TCP server will start a new thread/task/process on each incoming connection using a new “worker” socket on the server side.

The “accept, move the connection to a dedicated server socket, then go back to accept()” is probably a more common design?

http://stackoverflow.com/questions/11129212/tcp-two-different-sockets-sharing-a-port also says

Multiple worker-sockets on the same TCP server can share the same server-side IP/Port pair as long as they are associated with different client-side IP/Port pairs

## significant coding IV xp #P/F

See also ##failed c++cod`IV: home^IDE^whiteboard:4 beat-fronts

  • webex — is shorthand for any form of remote screen sharing tool
  • paper — is a shorthand for paper, dumb editor, white-board etc
lang location IDE? firm/person pts Notes category
java home hackerrank Pimco 2 2018 190 minutes
c webex codility 😦 Mako 4 2018 monkey algo
c webex codility 😦 Mako 3 2018 quasconstant algo
c onsite paper p SIG 2 2018 RAII smart ptr std
c onsite paper p SIG 3 2018 push_back std
py onsite IDE p SIG 3 2018 movie query dStruct
c home hackerrank p Promethean 5 2018 2 tough graph problems algo
c/py home hackerrank p Kraken 3 2018 3 problems algo
any onsite paper 😦 Nsdq 2018 best day-trading std
any onsite paper 😦 Nsdq 2018 check array 0-N in-situ algo
java home IDE p Romain 4 2018 drone: medium-level implementation design
c onsite paper p CVA/Dimitri 3 2018 O(N) in-situ filtering algo
c onsite paper 😦 FB 2018 2 short problems std algo
c onsite 😦 Trex 2018 exchange messaging SDI
py onsite IDE p Trex 1 2018 3 short but tricky problems algo
c onsite paper p Quantum 1 2018 virt func not declared in base trivial
c webex paper p bbg 4 2018 single-thr pool QQ
c webex paper p bbg 3 2018 shared ptr ctor/dtor QQ
py onsite paper p bbg 5 2018 continuousSentence algo
c onsite paper p? Wells 2 2017 concurrent queue std
c onsite paper p Wells 1 2017  remove_spaces trivial
c onsite paper p bbg 2 teams 2 2017 array reshuffle
c onsite IDE p bbg 2 teams 2 2017 bst
c onsite paper p bbg 2 teams 1 2017 string
c onsite paper 😦 bbg 2 teams 2017 filters on screen
C home hackerrank 😦 Thesys 2017 3 problems out of 5 algo
C home IDE p Thesys 5 2017 first N prime Fibonacci
c webex IDE p bbg 2 2017 top N active stocks dStruct heavy
c onsite paper p BAML 1 2017 various
c home IDE 😦 GS 2017 TickEngine dStruct heavy
C webex IDE p? bbg 2 2017 free slots dStruct heavy
C webex paper p bbg 1st round 2 2017 tree serialization std algo
Java onsite paper p BGC 2 2017 multiple short programs
Java home IDE p BGC 3 2017 connected or not
Java home IDE p HSBC 4 2017 barely passed
Java home IDE p pimco 5 2017 iterator again
Java onsite paper p pimco-Zoltan 2 2017
c webex IDE 😦 Citadel 2017 array shrinking
py webex paper 😦 Broadway 2017 hashtable std algo
cpp home IDE 😦 iRage 3 2015 order book again dStruct heavy
Java home codility 😦 baml 2015 Qz FX option team. too lazy
c home codility 😦 Jump 3rd 2015
c home IDE p jump 1st 3 2012 order book dStruct heavy
c home IDE 😦 DRW 3 2015 Tetris. coding style
c webex paper 😦 bbg -London 2015 unable to understand Q1 of 4
c# webex paper p eikon 2 2013
java home IDE 😦 MS FIX
swing webex IDE p Barx 2 2012 swing
C home IDE 😦 Mac 2 simple problems
java home IDE 😦 MS-comm too lazy
java onsite paper 😦 Barc SOR
java onsite IDE p RBC 1 2010 30-45 min
java onsite IDE 😦 UBS 2011 Suntec
java onsite IDE 😦 Lab49 2010
java home IDE p Gelber 2 2011 multithreaded
C home IDE 😦 Amazon
C onsite paper 😦 FB regex QQ; std algo
any onsite paper 😦 Goog 2007

append.bashrc.txt

 

<< ______________end_of_comment_1________________
modified [28 Apr 2017]
______________end_of_comment_1________________
[ -z "$(echo $- | grep i)" ] && return # this same line should be on top of append.bashrc_citi.txt
# # # # # # # # # # # # # # # # # #
#
# << var assignment >> should be first
#
# # # # # # # # # # # # # # # # # # # # # #
export PS1='\n\s!\! \u@\h [ \t \d ] \w/ \n\$ '
export EDITOR=vi # primarily for sqlplus
export VIMINIT=":se nu | :map v :w" 
export EXINIT=$VIMINIT # ignored if .vimrc found!
LESS="--ignore-case"
export TMOUT=987654; [ $root ] && export TMOUT=100600
HISTSIZE=900
set -o emacs
set +o nounset
export PATH=/usr/xpg4/bin:$PATH; [ $(id -u) = 0 ] && root=yesur 
set +o noclobber; [ $root ] && set -o noclobber
sav=$LOGNAME
# # # # # # # # # # # # # # # # # # # # # #
#
# PAGER and TERM:
#
# # # # # # # # # # # # # # # # # # # # # #
export PAGER=more
export PAGERE=more
if [ -z "$(which less |grep '^no less')" ]; then
 export PAGER="less" # breaking some x terminals but ok in "console"
 export PAGERE="less -E" # -E needed for dir-viewing, file-viewing but affects man 
fi
export TERM=vt100 # 'linux' breaks vi in gnome-terminal; 'linux' imperfect for solaris
# # # # # # # # # # # # # # # # # # # # # #
#
# << aliases >> ... sortable
#
# # # # # # # # # # # # # # # # # # # # # #
alias %=cd_l # typing trainer
alias ..='cd_l ..'
alias cp='cp -i' # defuse
alias egi='export |grep -i'
alias h=history
alias hgi="history |grep -i"
alias j1="fg %1"
alias j2="fg %2"
alias killthem="perl -pe 's/^\S+(\s+\d+).*/\1/ or s/.*//'|xargs -pt kill" # p str1 | killthem
alias m=$PAGER
alias mv='mv -i' # defuse
alias p=ps_grep
alias path=" echo \$PATH |perl -pe 's/:/\n/g' |sort -u|$PAGERE "
alias rm=myrm
alias s=' source $HOME/.bashrc '
alias t='l -t'
# # # # # # # # # # # # # # # # # # # # # #
#
# << functions >> ... difficult to sort
#
# # # # # # # # # # # # # # # # # # # # # #
cd_l(){
 [ $# -eq 0 ] && l && return
 [ -n "$( file $* | grep directory )" ] && cd $* && l && return
 [ -n "$( file $* | perl -ne 'print if /text|script/' )" ] && m $* &&
/bin/echo -e "\n\n" && l $*
}
d(){
 [ $# -ne 0 ] && cd $*
 [ `pwd` = '/' ] && target=/ && echo I am in /
 echo "In MB :"
 eval du -ks * |sort -n|perl -pe 's|^(\d+)|$1/1000|e'
}
notinuse_g(){ # bug with g -v
 pattern=$1; shift
 cmd=" grep -i \"$pattern\" $* "
 /usr/bin/printf '%s\n' "cmd=__$cmd __"
 eval $cmd |$PAGERE
}
l(){ # can't be completely replaced by 'cd_l' or ']', because "cd_l -tr dir1" is confusing and should be avoided
 /bin/ls -alFs $* |$PAGERE
}
myrm(){
 cmd="mv -i $* /var/tmp/$sav/ "
 /usr/bin/printf '%s\n' "cmd=__$cmd __"
 eval $cmd
}
ps_grep(){
 ## cmd1='/bin/ps -ef' # truncation risk
 # ps auxwww : inconsistent for root vs non-root
 cmd1='/usr/ucb/ps auxwww' # |grep -v grep -- no cos some real commands contain grep
 for f in $*; do
 cmd1="$cmd1 | g $f"
 done
 eval $cmd1
}
sav(){ # proven since 2007
 suffix=$RANDOM$(date +"%H%d%b") # to avoid misleading readers,make the suffix ambiguous
 for f in $*; do
 f=$(echo $f|perl -pe 's|/$||') # sav dir/
 b=`basename $f`
 (
 cd `dirname $f` ; 
 tar cf - $b | (cd /tmp; tar xpf -)
 /bin/mv -i /tmp/$b $b.b4${sav}edit$suffix
 [ -d $f ] && opt=' -d' 
 eval ls -lt $opt $b*
 )
 done
}
# # # # # # # # # # # # # # # # # # #
#
# << exa group of companies >> ... an exercise in grouping
#
# # # # # # # # # # # # # # # # # # # # # #
exa(){ #testing: exa -t ss ~/.ssh
 local ops
 while is_opt=`echo $1 |perl -ne 'print if /^-/' ` && [ "$is_opt" ]; do
 ops="$ops $1"; shift
 done

 if [ $# -eq 1 ]; then
 fullpath=$1
 shortname=`echo $1|perl -pe 's|/$||;s|^.*/||' `
 else
 fullpath=$2
 shortname=$1
 fi

 [ -x $fullpath ] || return
 # export "$shortname"="$fullpath"
 set $shortname="$fullpath" # not working as of 2017
 eval $shortname="$fullpath"
 prj="$fullpath"
 alias "$shortname"="cd $fullpath;l $ops"
 alias prj=$shortname # USE MORE
}
exa /tmp/
# # # # # # # # # # # # # # # # # # #
#
# << path stuff, potentially dangerous >>
#
# # # # # # # # # # # # # # # # # # # # # #
add_path(){ 
 [ -r $1 ] || return
 [ "$(echo $PATH|grep :$1)" ] && return
 # check world write perm, esp for root
 PATH=$PATH:$1
}
set_manpath(){ # very slow on some systems. Run this when u need it.
 for d in `ls -d /[uo]*/*/man /[uo]*/*/*/man /[uo]*/*/*/*/man`; do
 export MANPATH=$MANPATH:$d
 done
}
set_path(){
 add_path /usr/sbin
 add_path /usr/bin
 add_path /sbin
 add_path /usr/local/bin
 add_path /usr/cluster/bin
 add_path /usr/openwin/bin/
 add_path /usr/ucb
 add_path $ORACLE_HOME/bin
 add_path /usr/sfw/bin # pre-installed
 add_path /opt/csw/bin
 [ $root ] && return
 add_path . 
 add_path $HOME/bin
}
set_path

##study subjects I enjoyed as student(+professional)

(You don’t have to read this lengthy blog post.)

All my examples below required a long period of persistent effort. My son is unwilling to put in even 30% of my effort, so he wouldn’t find any enjoyment.

After the initial few hours of curiosity, learning any subject was always challenging, boring, or repetitive. The mental workload always wears us down, so we always needed conscious effort to stick to it I.e. stay focused and apply ourselves. The joy comes as a light at end of the tunnel. No exception.

In contrast, watching TV, playing electronic games, eating, sight-seeing, shopping … don’t require any effort. When we get bored with it, we are free to break away — there’s no measurable progress to worry so give-up has absolutely no consequence. We choose to stick to it (say, a game) only because we enjoy it. Therefore no effort required.

If a teenager takes on professional gaming as a paid job, he will invariably find it monotonous, repetitive, uninteresting, dull, tiring, before he can earn any money. Those gamers who do earn money are able to earn money precisely because they put up with the boring stuff — persistent effort.

— Now the study subjects I enjoyed, ranked by amount of effort:

I enjoyed blogging on technical subjects. My QnA interviews directly benefit from my blogging but this is not the biggest reason for the enjoyment. In fact, most of my technical blog content is not required for interviews. I can see my blog improving in quality (and quantity) as I revise and expand each article progressively. Blogging let’s me achieve deeper understanding on many tough topics like sockets, reflection, c++template, move-semantics, quant, complex SQL, … I call it "真本事". I then notice the superficial knowledge in fellow programmers, who studied only 20% of what I read.

I enjoyed memorizing English words, in my teenage years. I developed my own method. I was able to memorize 30 – 100 words a day for a few months (I refresh each word 20 to 40 times spaced out over 6 months). The fast progress was "breathtaking". Self-confidence grew. I actually learned some words my English-educated Singapore classmates didn’t know!

I enjoyed Chinese composition in my secondary Year 2-3, when I found my literary "voice", my style of writing. For about 2 years before that break-through, I was always in the bottom 10% in my class in terms of composition and every assignment was a pain like giving birth to a short, boring baby. The pain, self-despise, self-pity was one of the deepest in my childhood. I felt inadequate almost like crippled. The breakthrough was a liberation. I achieved it by writing daily essays for a few months. My output slowly improved from very dry, simple "流水账" to have some meaning, some color, some personality, some nice expressions. It became creative expression.

(Paradoxically, as a student I was very strong in math/physics, but I never had the same enjoyment as I had learning English and Chinese composition. I think my joy was in overcoming highly visible personal limitations — sense of breakthrough and liberation.)

I enjoyed piano playing after years of repetitive, boring, tough practice. In my life I never again put up with this much practice — 5000 repeated practices x 2 minutes/repetition in some cases.

2H life-changing xp#Pimco#income,home location,industry…

Here’s a real story in 2010 — I was completely hopeless and stuck in despair after my Goldman Sachs internal transfer was blocked in the last stage. I considered moving my whole family back to Singapore without any offer, and start my job search there. I was seriously considering a S$100k job in a back office batch programming job. Absolutely the lowest point in my entire career. After licking the would for 2 months, I started looking for jobs outside Goldman and slowly found my foothold. Then in early 2010, I passed a phone screening and attended a Citigroup “superday”. I spent half an hour each with 3 interviewers. By end of the day, recruiter said I was the #1 pick. I took the offer, at a 80% increment. In the next 12 months, I built up my track record + knowledge in

  1. real time trading engine components, esp. real time pricing engine
  2. fixed income math,
  3. c++ (knowledge rebuild)

I have never looked back since. Fair to say that my family won’t be where we are today, without this Citigroup experience. With this track record I was able to take on relatively high-end programming jobs in U.S. and Singapore. I was able to live in a convenient location, and buy properties and send my kids to mid-range preschools (too pricey in hind sight). Obviously I wanted this kind of job even in 2009. That dream became reality when I passed the superday interview. That interview was one of the turning points in my career.

Fast forward to Apr 2017 — I had a 20-minute phone interview with the world’s biggest asset management firm (Let’s call it PP), then I had a 2-hour skype interview. They made an offer. I discussed with my recruiter their proposal —

  • I would relocate to California
  • I would get paid around 200k pretax and possibly with an increment in 6 months. PP usually increase billing rate after 12 months if contractor does well.
  • recruitment agency CEO said he would transfer my visa and sponsor green card.

If I were to take this offer, my life would be transformed. (I would also have a better chance to break into the  high tech industry in nearby silicon valley, because I would have local friends in that domain.) Such a big change in my life is now possible because … I did well [1] in the interview.

Stripped to the core, that’s the reality in our world of contract programmers.  Project delivery, debugging, and relationship with boss can get you promoted, but those on-the-job efforts have much lower impact than your performance during an interview. Like an NBA playoff match. A short few hour under the spot light can change your life forever.

This is not a rare experience. There are higher-paying contract job offers that could “change your life”, and you only need to do well in the interviews to make it happen.

I feel this is typical of U.S. market and perhaps London. In Singapore. contract roles can’t pay this much. A permanent role has a few subtle implications so I feel it’s a different game.

[1] The 7 interviewers felt I was strong in c++ (not really), java and sql, and competent in fixed income math (I only worked with it for a year). Unlike other high-end interviews, there are not many tough tech questions like threading, algorithms, or coding tests. I feel they liked my interview mostly because of the combination of c++/java/fixed income math — not a common combination.

technical advantages of c# over java#le2XR

Hi XR,

Based on whatever little I know, here are some technical advantages of c# over java.

(Master these c# feature and mention them in your next java interview 🙂

  • C# has many more advantages on desktop GUI, but today let’s focus on server side.
  • [L] generics —- c# generics were designed with full knowledge of java/c++ shortcomings. Simpler than c++ (but less powerful), but more complete than java (no type erasure). For example see type constraints.
  • [L] delegates —- Rather useful. Some (but not all) of its functionalities can be emulated in java8.
  • [L] c# can access low-level windows concurrency constructs such as event wait handles. Windows JVM offers a standardized, “reduced-fat” facade. If you want optimal concurrency on windows, use VC++, or c#.
  • [L] reflection —- is more complete than java. Over the years java reflection proved to be extremely powerful. Not sure if c# has the same power, but c# surely added a few features such as Reflection.Emit.
  • concurrency —- dotnet offers many innovative concurrency features. All high level features, so probably achievable in java too.
  • tight integration with COM and MS Office. In fact, there are multiple official and unofficial frameworks to write Excel add-ins in c#
  • tight integration with high-level commercial products from Microsoft like MSSQL, sharepoint
  • tight integration with windows infrastructure like Windows Services (like network daemons), WCF, Windows networking, Windows web server, windows remoting, windows registry, PowerShell, windows software installation etc
  • c# gives programmers more access to low-level windows system API, via unmanaged code (I don’t have examples). In contrast, Java programmers typically use JNI, but I guess the java security policy restricts this access.
  • probably higher performance than JVM on windows
  • CLR offers scripting languages VB.net, F#, IronPython etc, whereas JVM supports scripting languages javascript, scala, groovy, jython etc.

[L = low-level feature]

If you want highest performance on Windows, low-level access to windows OS, but without the complexity of VC++ and MFC, then c# is the language of choice. It is high-level, convenient like java but flexible enough to let you go one level lower when you need to.

Another way to address your question — listen to the the complaints against java. (Put aside the complaints of GUI programmers.)

Even if a (rational, objective) architect doesn’t recognize any of these as important advantages, she may still favor c# over java because she is familiar and competent ONLY in the Microsoft ecosystem. She could point out countless features in Visual Studio and numerous windows development tools that are rather different from the java tool set, so different that it would take months and years to learn.

Also, there are many design trade-off and implementation techniques built on and for Dotnet. If she is reliant on and comfortable in this ecosystem, she would see the java ecosystem as alien, incomplete, inconvenient and unproductive. Remember when we first moved to U.S. — everything inconvenient.

On a more serious note, her design ideas may not be achievable using java. So java would appear to be missing important features and tools. In a nutshell, for her java is a capable and complete ecosystem theoretically, but in practice an incomplete solution.

%%offers 2017

All confirmed offers.

$c2c co where primary tech other tech domain nlg duration
100 pimco Burak NPB[1] c++11 🙂 🙂 🙂 java, possibly Hadoop 🙂 FI accrual math 🙂 3+
100 Pimco Zoltan NYC java framework 🙂 🙂 flexible
100+ bgc Alexi NYC java minimal cpp FX.. trading to perm 😦 😦 😦
below 100 😦 😦 Ravi Chgo 😦 😦 Qz 😦 😦 😦 java FI trading again flexible
perm Nitin Shanghai java perm
perm Tradeweb JC VC++ FI ECN perm
85 baml NYC VC++ repo 😦 12M?

[1] A bit hard to get next job in NY, but helps me get a next job in West Coast. However, in terms of buying a home, I just don’t know.