git | diff | make file6 appear before file2

The pain — a big commit touches on too many files, 3 of them major changes, while 5 files have cosmetic changes. In git-diff or git-difftool, I want to focus on the 3 major files.

Some developers would want to split the big commit into a “major” and a “cosmetic” commit, but one of the snapshots would be half-cooked and unusable. In such a case, the commit comment would warn “This commit is part of a bigger change. Please do not build this commit alone”.

As an alternative we can make the 3 major files show up first (or last) in diff. You can put the content below in src/diffOrder.txt and use it as instructed. All unspecified files would show up in natural order.

The line-ending issue the most common mistake with -O !

# git diff -O src/diffOrder.txt
# check EACH line below to ensure unix line ending, 
# even if entire file is unix-style.
# wildcards are needed below.

*/ProductCacheUtil*
*/CSV*
*/FutureDataCacheUtil.java
*/RJOBRIENDROPTools.java

conEmu tab-bar customize

— Problem: current active tab is indistinguishable from other tabs. https://github.com/Maximus5/ConEmu/issues/552 describes one solution using “tab templates”. Here’s my own adaptation of this solution:

%m ★ m %s%m ★ m

Explanation: The q[ %m ★ m ] inserts a star iFF the host tab is the active tab.

— Problem: tab label should show only dir name and hostname without noisy clutter.

My solution — use SkipWordsFromTitle feature, with pipe-separated list of bad words

MINGW64:/N/repos/|MI_EPA/|.mlp.com – PuTTY

MSOL zoom by mouse/touchpad/touchscreen

My home laptop screen resolution is such that zoom-in is required when reading outlook messages. How do I use keyboard to easily zoom in on a new message?

— touch screen two-finger pinch works, even though I disabled it in my laptop

— Ctrl + right-scrollUp is similar to the slider.

With an external mouse, this gesture is probably proven and reliable. With a touchpad, I relied on two-finger scroll:

Two-finger up-scroll + Ctrl does zoom in 🙂

— two-finger page scroll without Ctrl key : is unrelated to zooming

Windows gives us two choices. I chose “Down motion means scroll down”. Intuitively, the two-finger feels like holding the vertical scroll-bar on a webpage and in Initellj.

Note Intellij scroll is otherwise impossible with touch-screen or touchpad!

MSOL move mails from DeletedItems #ServerSide

Problem 1: DeletedItems are not achieved in some systems. Those emails are purged.
solution: I usually move items to _deleted folder, which is not purged.

Problem 2: moving hundreds of items this way is very slow and can crash my MSOL
Solution: do it on server side, via browser interface like email.mlp.com. I could select 40 items (sometimes hundreds of items) in DeletedItems, and successfully drag-n-drop to _deleted folder.

Problem 3: sometimes the drag has no effect whatsoever
Clue: perhaps the first item is a meeting invite or another type of special item. Delete those items by hand and retry.

egrep: highlight pattern]file !!filtering !!scrolling

https://github.com/tiger40490/repo1/blob/bash/bash/send.sh shows:

cat some.txt | egrep –color “40490.$rand|“

Note the trailing pipe gets rid of filtering 🙂

— other solutions:

less +/pattern # not ideal as it scrolls

MSOL: Tasks^ToDoList

–To-Do list (is better than Task view)

• lets me drag an item within its category
• shows content of Notes

 

vi on multiple files

— 2019 xp: I find it useful to quickly exit vi, examine another file, then use bash to recycle previous (long) vi-launch command.

I rely heavily on the fact that vi remembers the last cursor (and search pattern) per file.

Whenever I run a vi-launch command, I quickly decide on file1 file2 file3 names, picking and shuffling among the hottest files.

—-

[3/4] means vi receives 3 keystrokes; we hit 4 keys including shift or ctrl …

–“split” solution by Deepak M

vi file1 # load 1st file

• :sp file2 # to show 2nd file upstairs
• :vsp file3 # to show 2nd file side by side
• You end up with  — file2 and file3 side by side upstairs, and file1 downstairs!
• [2/3] ctrl-ww # To move cursor to the “next” file, until it cycles back

–the q( :e ) solution

vi file1 # load 1st file

• :e file2 # to put 2nd file on foreground
• [1/3] ctrl-^ — to switch to “the other file”
• This solution is non-ideal for moving data between files, since you must save active file before switching and you can’t see both files

–editing 3 or more files

1. vi file1 file2 file3
2. q(:n) to switch to next, q(:N) for previous…
3. q(:args) shows all files with current file highlighted

• –Suppose now you are at file2.
• q(:e file4) works. q(^) will toggle between file2 and file4
• However, q(:n :N  :args) only work on the original list, not new files from q(:e)

q(:n :N ^) always shows the current filename in status bar:)

## intellij keyboard accelerators

• alt-8 jump to the panel numbered 8, useful if #8 is a tab on a shared “region”
• shift-escape to hide/dismiss current panel
• ctrl-F4 to clone current file among many in the editor
• ctrl-alt-H to show call hierarchy
• ctrl-Q show javadoc
• ctrl-F12 to show all members of current class

split a 2-file commit into two

https://emmanuelbernard.com/blog/2014/04/14/split-a-commit-in-two-with-git/:

git rebase -i <oldsha>
# bring the bigCommit to the tip
git reset HEAD^ # reset to just before that new tip
git add file1
git commit -m "First file"
git add file2
git commit -m "Second file"

windows: open path start`with double \\ #RUN

Suppose you want to access \\ms\dist\a\b\

You can’t paste that string into MSWE. Instead

winKey -> RUN -> paste the path string -> enter

effectively ignore whitespace change]code review #perl

$ perl -pe ‘s/\s//g’ tcm_creator.py> new
$ git checkout some-commit tcm_creator.py
$ perl -pe ‘s/\s//g’ tcm_creator.py> old
$ diff old new # should show no discrepancy

This worked for me. Entire file is flattened to a single long string.

IIF the whitespace change is only and always confined with a line i.e. no line splitting/joining , then perl  -l -pe is more useful.

 

## bit twiddling hacks

http://www.graphics.stanford.edu/~seander/bithacks.html

• Swapping values with XOR
• Determining if an integer is a power of 2
• Find integer log base 10 of an integer
• Round up to the next highest power of 2

## notepad++ in-place sorting tips

If case-sensitive, the best way is Edit->LineOperations

 

If case-insensitive, then no native support. A few options:

• option: right-click on the file name, then copy the filename to clipboard. Then openContainingFolderInDOS. Run DOS sort fileName /o fileName
• option: git-bash q[sort] but in-place sorting may require a bash function
• option: TextFX plugin

windows stickyNote^MSOL note #folder

Purpose: temp reminders

		visibility	hide without deleting	folder tree	listing of notes	reading pane
sticky note		good	?	no	?	no
outlook note		good	yes	YES	yes	yes
paper note		best	yes

MSOL defer-send rule #3spaces

I have used this type of rules in many companies. My current set-up is —

Apply this rule after I submit a msg marked as Normal importance: defer delivery by 1 minute.

• https://blogs.msdn.microsoft.com/edkatibah/2012/12/17/setting-up-a-delayed-sending-rule-for-email-in-microsoft-outlook/  describes an exception – put three spaces anywhere in the subject as a secret instruction for Outlook to “send without delay”
  • I tested 4 spaces –> immediately sent
  • I tested 2 spaces –> deferred
• Using Normal importance as condition, I can skip the default /deferment/ using low-importance or high-importance.
• Submitted messages are Normal by default and therefore deferred.
• Note 1 minute is the minimum

I wish there’s a “importance = A or B” condition.

right_click in mswe

Scenario: If you left-click (even if to the right side) to select file1.txt, then right click, you would get the per-file context menu showing Copy/Cut/Delete etc.

Scenario: If you directly right click in an empty space (even if to the right side of some file1.txt), you get the per-dir context menu.

Q: Why no file selected?  Suppose you do a right-click to the right of file1.txt. Before you release the right button, file1.txt may be highlighted temporarily but not selected. After you release the right button, you will see no file selected and you get the per-dir context menu.

Scenario: What if you have already selected file1.txt and now you want the per-dir context menu? Confusing… Solution — Just right-click in another empty space, perhaps to the right of file2.txt. Again, you will see no file selected.

perl regex top3 tips #modifier /m /s

https://docstore.mik.ua/orelly/perl3/lperl/ch09_05.htm shows

The part of the (haystack) string that actually matched the (needle) pattern is automatically stored in q[  $& ]

Whatever came before the matched section is in $` and whatever was after it is in $'. Another way to say that is that $` holds whatever the regular expression engine had to skip over before it found the match, and $' has the remainder of the string that the pattern never got to. If you glue these three strings together in order, you’ll always get back the original string.

— /m /s clarification:

1. By default, q($) + q(^) won’t match newline. /m targets q($) and q(^)
2. By default, the dot q(.) won’t match newline. /s targets the dot.
3. The /m and /s both help get newlines matched, in different contexts.

Official doc says:

1. /m  Treat the string being matched against as multiple lines. That is, change "^" and "$" from matching the start of the string’s first line and the end of its last line to matching embedded start and end of each line within the string.
2. /s  Treat the string as single line. That is, change "." to match any character whatsoever, even a newline, which normally it would not match.
3. Used together, as /ms, they let the "." match any character whatsoever, while still allowing "^" and "$" to match, respectively, just after and just before newlines within the string.

q[cannot open shared object file] abc.so

strace -e trace=open myprogram can be used on a working instance to see where all the SO files are successfully located.

— Aug 2018 case: in QA host, I hit “error while loading shared libraries: libEazyToFind.so: … No such file or directory”

I can see this .so file so I used LD_LIBRARY_PATH to resolve it.

Then I get “error while loading shared libraries: libXXXXX.so: … No such file or directory”. I can’t locate this .so, but the same executable is runnable in a separate HostB. (All machines can access the same physical file using the same path.)

I zoomed into the HostB and used “ldd /path/to/executable”. Lo and behold, I can see why HostB is lucky. The .so files are located in places local in HostB … for reasons to be understood.

— May 2018 case:

The wording should be “cannot locate ….”

I fixed this error using $LD_LIBRARY_PATH

The *.so  file is actually specified as a -lthr_gcc34_64 option on the g++ command line, but the file libthr_gcc34_64.so was not found at startup.

I managed to manually locate this file in /a/b/c and added it :

LD_LIBRARY_PATH=$LD_LIBRATY_PATH:/a/b/c/

bash: split long command to multiple lines

I managed to split a huge g++ command line  to about 300 lines… much more readable.

The trick:

• terminate each line with a space, a backslash and … NO trailing space
• my perl one-liner must user four backslashes to insert that single backslash, then another \n
• total there are 5 backslashes in a row.

Here’s the command

perl -pe "s/ -/ \\\\\n-/g" build1file.sh

MSOL q[Received]: space-sav`custom format

–based on http://www.tushar-mehta.com/publish_train/xl_vba_cases/1205%20Outlook%20custom%20columns.shtml

IIf(Date()=int([Received]),Format([Received],”hh:mm”),IIf(Date()-[Received]<7,format([Received],”ddd m/d”),format([Received],”d mmm yy”)))

Warning: need to clean up the double-quotes after copy-paste from this blogpost!

I name it “-myDT3” — 3 types of values

Q: can I share this custom column between Inbox/Deleted/Sent?
A: Unsuccessful try in 2010

Problem — unsortable
Solution — add the original Received field, but keep it razor thin

How to edit the formula: FieldChooser -> User-defined (usually empty) -> drag the new field in -> edit -> drag it out

q[ls symlink_to_a_dir/ ] #trailing slash

How to see the source and target of a symlink without unnecessary details:

Solution — q[ ls symlink_to_a_dir/ ]

• bad alternative: Without the trailing slash, you see only the symlink not the content of the dir
• bad alternative: If you replace the slash with /*, then you do see dir content, but you can get another problem. If the target dir has any subdir, you actually list those subdir contents 😦

git | diff | highlights trailing space

By default there’s a default-enabled highlight (in red) of trailing spaces. I encounter it all the time when running git-diff. Extremely distracting.

$ git config core.whitespace  –trailing-space # the dash before “trailing” disables it

Beware: when you copy-paste into a command window, the dash may become something else 😦

If successful, this command adds a line into .git/config like

whitespace = -trailing-space

You can remove this line to restore the “factory default”.

one-liner to choose cout^cerr

Useful in coding tests and in project

// Usage 1
(success ? cout << "Apportionment succeeded: " : 
           cerr << "Apportionment failed: ") 
<< msg <<endl;

// Usage 2 in my own code
(isLeft? parent.left: parent.right) = newChildNode;

stack 2 windows above/below #win7

based on https://superuser.com/questions/285356/possible-to-snap-top-bottom-instead-of-just-left-right-in-windows-7

Scenario — On my vertical monitor I want two windows A and B stacked; on my other monitor I want one big window; total 3 windows.

Solution —

• minimize all other windows
• right click on taskbar -> showWindowsStacked

Note to show windows side by side there are many options like winKey and snap

side-by-side #tested win7,10

Areo Snap (https://www.howtogeek.com/181681/4-hidden-window-management-tricks-on-the-windows-desktop/) worked on my Win10:

• hold winKey and hit 4 arrow keys
• I think this app treats the desktop as 4 quadrants. Each window can snap to
  • any one quadrant or
  • a full column
  • but not a full row
• You can even drag the boundary between 2 tiled windows 🙂

core file location

Still no definitive answers…

In one machine, q(ulimit -c) returns 0 meaning suppressing core files. I had to run q(ulimit -c unlimited) to get my core files generated.

bash: iterate over an integer range

https://stackoverflow.com/questions/169511/how-do-i-iterate-over-a-range-of-numbers-defined-by-variables-in-bash

notepad++saving real-estate

Survival tip — Alt-t gets to Settings, if you need to unhide …

Motivation — most monitors are too “thin”, so the bars take up vertical space.

• Menu bar? Better hide by default. can toggle using Alt or F10
• Toolbar? Better hide
• Document tab? Vertical is best

JDK + eclipse install: %%preference

• C:/j8 or j801 # I prefer shorter path. I almost never have two versions of java on one machine
  • There’s an embedded ./jre directory for a so-called “private JRE”
• public JRE is optional and should be installed outside JDK directory, but I have not needed it since I started using java.
• JAVA_HOME and PATH
• —-eclipse: I prefer the simple zip file download
• C:/ide/elcipse #worked OK
• c++ide? can use c:/ide/eclipseCDT.

 

hot key; copy current line

First learn the keyboard accelerator to select current line.

Remember shift-HOME selects back till beginning; shift-END selects till the end.

HOME then shift-END

END then shift-HOME

bash: simplest bool flag based on hostname match

• Note the quotes and the star in hostname check
• Note no quotes needed around $isUK

[[ $(hostname) == "rtdevfrh"* ]] && isUK=1
[[ -z $isUK ]] && echo not in UK
[[ -n $isUK ]] && echo in UK

symlink/hardlink: Win7 or later

https://www.howtogeek.com/howto/16226/complete-guide-to-symbolic-links-symlinks-on-windows-or-linux/ is a 2017 article.

The mklink command can create both hard links (known as “hard links” in Windows) and soft links (known as “symbolic links” in Windows).

On Windows XP, I have used “Junction.exe” for years, because mklink is not available.

discover cpu count ] my machine #lscpu

NIC — There’s another blog post https://bintanvictor.wordpress.com/2018/03/19/discover-nic-my-machine-lspci/

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 1
Core(s) per socket: 8
CPU socket(s): 2
NUMA node(s): 2
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K

---

$ less /proc/cpuinfo|egrep 'core id|processor'  # on the same machine shows

16 “processors”, but #0 and #1 have the same core-id! Total 8 core-id values. I think this is because in each (of two) socket, there are 8 cores with unique core-id values.

dmesg | grep rocessor # doesn’t work any more

dxdiag # windows

dxdiag /t c:\deldel.txt
find “CPU” c:\deldel.txt # copy-paste may not work

if q[ print obj1] shows only type name

print obj1 # by default shows limited content like

<testcases.multitest.result.Result object at 0x7fe7f08fc810>

It’s more useful to see the source code of the obj1’s  type definition like “Result” in this case. Here’s how

print testcases.multitest.result.__file__ # assuming …result is a module

std::vector insert by position

myVec.insert(myVec.begin() + INTEGER_OFFSET, DATA);

c++concatenate number #stringstream

–Use stringstream:

stringstream tmpPath;
tmpPath.str(“”); //set to empty string
tmpPath<<tmpFolder << tmpCount++;

cout<<tmpPath.str()<<endl;

python run complex external commands #subprocess

I prefer one single full-feature solution that’s enough for all my needs. The os.system() solution is limited. The subprocess module is clearly superior. One of the simplest features is

>>> subprocess.call([“ls”, “-l”])

If you need redirection and background, then try the single-string version

>>> subprocess.call(‘ls /tmp > /tmp/a.log &’, shell=True) # output goes to STDOUT, hard to capture

from within bash script1 get script1’s dirname full path

https://github.com/tiger40490/repo1/blob/bash/bash/thisScriptDir.sh is latest tested solution

fullpath="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
echo fullpath is $fullpath # fairly reliable one-liner, provided the script itself isn't a symlink

 

## vi (+less) cheatsheet

Note q[less] not q[more[ can open *.gz files 🙂

https://github.com/tiger40490/repo1/blob/bash/bash/vimrc has some tricks including how to make vim remember last edit location, but make sure your ‘vi’ command actually runs viM !

• ~~~~ command mode #roughly ranked
• [2/3] :↑ (i.e. up-arrow) — cycle through previous :commands
• [3] dt — “dta” delete until the next “a”
• [2]: 6x — delete 6 chars
• [2] 9s — wipe out 9 characters (including current) and enter insert-mode. Better than R when you know how many chars (9) to change
  • to delete 5 characters … there is NO simpler keystroke sequence
• R — Overwrite each character one by one until end of line. Useful if the replacement content is similar to original?
• Ctrl-R to re-do
• cw — wipe out from cursor to end of word and puts you into insert mode
  • c2w or 2cw
• :se list (or nolist) to reveal invisible chars
• C — wipe out from cursor to END of line and puts you into insert-mode
• capital O — open new line above cursor
• A — to append at END of current line
• :sy off  — get rid of color scheme
• :syntax off # is the long form
• from inside q(LESS), type a single “v” to launch vi

[3/4] means vi receives 3 keystrokes; we hit 4 keys including shift or ctrl …

–paging commands in vi and less

• jump to end of file: capital G == in both vi and LESS
• jump to head of file: 1G == in both vi and LESS
• page dn: Ctrl-f == in both; LESS also uses space
• page up: Ctrl-b == in both; LESS also uses b

 

cout/printf : print ptr

https://stackoverflow.com/questions/2064692/how-to-print-function-pointers-with-cout says just cast to void pointer

"cout << (void *)pf is " << (void *)pf << endl;

printf("printf(\"%%p\", pf) is %p\n", pf);

bash: run command in dir1 +! changing curr dir

I struggled with subshell but now I prefer this pushd technique. Useful not only in shell scripts:

pushd dir1
runCommand
popd

 

diff two checked-out dir from same branch

Challenge (common problem for developers) : locate a breaking commit in a long history

More than once, I have found  great value in checking out two copies from the same branch. Need to compare the two directories recursively.

diff -w -r -x CVS -x shared -x lib -x *.o -x '*.b4vtanedit*' aquis aquis1 | less -E

converting btw epoch, HHMMSS ..

• Note epoch is second-level, not millisecond level.
• Note epoch is timezone agnostics. It’s always defined in UTC.
• Note struct tm is the workhorse. It can break up a time object into the year/weekday/…/second components

—to convert from epoch to HHMMSS:
https://github.com/tiger40490/repo1/blob/cpp1/cpp/88miscLang/convertEpochInt.cpp

—to convert current time to int epoch number:
https://github.com/tiger40490/repo1/blob/cpp1/cpp/88miscLang/convertEpochInt.cpp

—epoch timestamp is typically in seconds

• 1513961081 — 10 digits, seconds since Epoch
• 1511946930032722000 — 19 digits, nanosec since Epoch

split std::string on Custom delimiter #practice every6M

See also post on csv string parse…

For a longer delimiter, you may need string.find()

https://github.com/tiger40490/repo1/blob/cpp1/cpp/binTree/serialize_bbg.cpp has my own tested solution parsing individual tree node details from a stringstream

ifstream f1(fileName.c_str());
string line;
while(getline(f1, line)){
  for(int i=1; ;++i){
        int pos = line.find_first_of("\t");
        string token = line.substr(0,pos);
        cerr<<i<<" : " <<token<<endl;
        if (line == token) break; //there's no more tab in the line
        line = line.substr(pos + 1);
  }
}

///// a simpler method:
istringstream lineStream("denmark sweden   india us"); //consecutive spaces are Not treated as one
string outputToken;
int main(){
  while (
    getline(lineStream, outputToken, ' ')) // <-- the only thing to remember
        cout << outputToken << endl;
}

custom delimiter for cin operator>> #complicated

Tested but is too hard to remember. Better use the getline() trick in https://bintanvictor.wordpress.com/2017/11/05/simplest-cway-to-split-string-on-custom-delimiter/

struct comma_is_space : std::ctype<char> { //use comma as delimiter
  comma_is_space() : std::ctype<char>(get_table()) {}
  static mask const* get_table() {
    static mask rc[table_size];
    rc[','] = std::ctype_base::space;
    return &rc[0];
  }
};

istringstream iss(line);
iss.imbue(locale(cin.getloc(), new comma_is_space));

binary search in sorted vector of Tick pointer

Note the mismatched args to the comparitor functions.

(I was unable to use a functor class.)

std::vector<Tick const*> vec;
int target;
bool mylessFunc(Tick const * tick, unsigned int target) {
     //cout<<tick->ts<<" against "<<target<<endl; 
     return tick-ts < target;
}
lower_bound(vec.begin(),vec.end(),target, mylessFunc);

bool mygreaterFunc(unsigned int target, Tick const * tick){
     //cout<<a->ts<<" against "<<target<<endl; 
     return tick->ts > target;
}
upper_bound(vec.begin(),vec.end(),target, mygreaterFunc)

WordPad : rich-text note taker

I found WordPad RTF files a good middle ground between ascii text and MSWord files.

• 🙂 basic text effects like background+font colors
• 🙂 footprint is comparable to ascii, much smaller than MSWord files.
  • Q: How about after compression?
• 🙂 how about copying to Linux? tested

Verdict —

1. default general-purpose note-taker remains ascii, but
2. for office work notes with text highlighting, let’s try WordPad more often. But let’s not become dependent. Convert old rtf notes to ascii.

std::string trailing space trim

//size_t or int

int endpos = word.find_last_not_of(” “);
if (endpos < string::npos) word = word.substr(0, endpos1);

std::map initializer: before c++11

map<int, int>{{32,1}}

LD_LIBRARY_PATH ^ objdump RUNPATH

This is related to q[cannot open shared object file] abc.so

See https://amir.rachum.com/blog/2016/09/17/shared-libraries/#rpath-and-runpath for the RUNPATH

q(objdump) can inspect the binary file better than q(ldd) does.

q(ldd) shows the final, resolved path of each .so file, but (AFAIK) doesn’t show how it’s resolved. The full steps of resolution is described in http://man7.org/linux/man-pages/man8/ld.so.8.html

q(objdump) can shed some light … in terms of DT_RUNPATH section of the binary file.

##bash alias : permissible meta-chars

I like “typing tutors”

I also like a family of alias like nx- nx, nx.

		allowed in alias	no shift
-(dash)		good	good
, (comma)		good	good
. (dot)		good	good
+ (plus)		good	good
!		good	bad
= (equals)		bad
backtick		bad
/ or \		bad

 

 

simple c++code: convert current or file mtime to struct tm

least amount of code

struct stat buffer;
stat(m_secInfoFileName.c_str(), &buffer);
cerr<<localtime(&buffer.st_mtime)      ->tm_mday<<endl;
cerr<<localtime(new time_t(time(NULL)))->tm_mday<<endl;

quickly identify PID hold`a port #netstat

netstat has -p flag

netstat -pln |grep 40493

fill a vector with 5 copies of an object

https://github.com/tiger40490/repo1/blob/cpp1/cpp/array/minCostPartition_Flex.cpp shows the vector ctor vector(sz, defVal)

SIMPLEST if-true/if-false ] bash

I often need a manual switch in source code. I can make a one-char change to turn on/off a block.

if (0) and if (1) is good enough in c/python. Java must use if(true)

In bash, I now use

• if [ -d / ] # true
• if [ -f  / ] # false

Locate msg]binary feed #multiple issues solved

Hi guys, thanks to all your help, I managed to locate the very first trading session message in the raw data file.

We hit and overcame multiple obstacles in this long “needle search in a haystack”.

• · Big Obstacle 1: endian-ness. It turned out the raw data is little-endian. For my “needle”, the symbol integer id 15852(in decimal) or 3dec(in hex) is printed swapped as “ec3d” when I finally found it.

Solution: read the exchange spec. It should be mentioned.

• · Big Obstacle 2: my hex viewers (like “xxd”) adds line breaks to the output, so my needle can be missed during my search. (Thanks to Vishal for pointing this out.)

Solution 1: xxd -c 999999 raw/feed/file > tmp.txt; grep $needle tmp.txt

The default xxd column size is 16 so every 16 bytes output will get a line break — unwanted! So I set a very large column size of 999999.

Solution 2: in vi editor after “%!xxd -p” if you see line breaks, then you can still search for “ec\_s*3d”. Basically you need to insert “\_s*” between adjacent bytes.

Here’s a 4-byte string I was able to find. It span across lines: 15\_s*00\_s*21\_s*00

• · Obstacle 3: identify the data file among 20 files. Thanks to this one obstacle, I spent most of my time searching in the wrong files 😉

Solution: remove each file successively, starting from the later hours, and retest, until the needle stops showing. The last removed file must contain our needle. That file is a much smaller haystack.

o one misleading info is the “9.30 am” mentioned in the spec. Actually the message came much earlier.

o Another misleading info is the timestamp passed to my parser function. Not sure where it comes from, but it says 08:00:00.1 am, so I thought the needle must be in the 8am file, but actually, it is in the 4am file. In this feed, the only reliable timestamp I have found is the one in packet header, one level above the messages.

• · Obstacle 4: my “needle” was too short so there are too many useless matches.

Solution: find a longer and more unique needle, such as the SourceTime field, which is a 32-bit integer. When I convert it to hex digits I get 8 hex digits. Then I flip it due to endian-ness. Then I get a more unique needle “008e0959”. I was then able to search across all 14 data files:

for f in arca*0; do

xxd -c999999 -p $f > $f.hex

grep -ioH 008e0959 $f.hex && echo found in $f

done

• · Obstacle 5: I have to find and print the needle using my c++ parser. It’s easy to print out wrong hex representation using C/C++, so for most of this exercise I wasn’t sure if I was looking at correct hex dump in my c++ log.

o If you convert a long byte array to hex and print without whitespace, you could see 15002100ffffe87600,but when I added a space after each byte, it looks like 15 00 21 00 ffffe876 00, so the 3rd byte was overflowing without warning!

o If you forget padding, then you can see a lot of single “0” when you should get “00”. Again, if you don’t include white space you won’t notice.

Solution: I have worked out some simplified code that works. I have a c++ solution and c solution. You can ask me if you need it.

• · Obstacle 6: In some cases, sequence number is not in the raw feed. In this case the sequence number is in the feed, so Nick’s suggestion was valid, but I was blocked by other obstacles.

Tip: If sequence number is in the feed, you would probably spot a pattern of incrementing hex numbers periodically in the hex viewer.

ksh wrap long command #avoid truncate

set -o multiline

Doesn’t work on bash but i don’t know if bash has this problem.

scan entire codebase4given func name #how2

–Challenge: scan a c++ codebase for a given func name

• A script would offer more flexibility.
• find + perl + grep is a crude solution, without support for comments

See also the task in Outlook!

–A related challenge: suppose you have the definition of a function, how do you see all the callers?

• Technique using compiler warning — http://stackoverflow.com/questions/1140509/how-to-find-all-callers-of-a-function-in-c . Tested:  able to report the line number of caller:

Csmi.C: In static member function ‘static csmiparser::Csmi& csmiparser::Csmi::getInstance()’:
Csmi.C:14: warning: ‘__comp_ctor ’ is deprecated (declared at /home/vtan/tp/plugins/xtap/csmi/include/Csmi.h:33)

• Technique — Rename the by appending _xxx and rebuild

 

c++simplest file/dir path validation

https://github.com/tiger40490/repo1/blob/cpp1/cpp1/isValidPath.cpp

##windoz ftp client +! admin permission

• experience — in Win7 MSWE, I was able to type

ftp:username@hostname

System prompted and saved my password 🙂

• experience — I installed FileZillaPortable in my own dir and it could connect to a linux site, remembering my credentials

create desktop shortcut to a putty remote host

–Create shortcut on desktop

Copy putty.exe into c:\windows

http://www.htpcbeginner.com/putty-shortcut-to-saved-session/

“putty -load ficsinapd01 -l vtan7”

–adding the shortcut to StartMenu

I was able to copy the shortcut to some part of my startMenu

A new solution — I created a start menu item containing

“\path\to\putty -load Session3”

 

identify unix disk hogs

du -csxPBM * | sort -n

du -cks * |sort -n |tail

du -h –max-depth=1 | tee ~/diskUsage2.log

discover NIC ] my machine #lspci

For cpu, there’s another blog post https://bintanvictor.wordpress.com/2011/11/20/q-discover-cpu-count-my-machine-linuxwin/

lspci | grep -i ‘ethernet’ #shows

• network card bandwidth
• network card manufacturer brand

C++read number from file→variable #briefly

ifstream myfile((tmpDir.str()+"_config").c_str());
if (myfile) // file may not exist
   myfile>>packetDumpLimit;

msol 2010 disable auto-receive at start up

Still no solution

doesn’t work — Select File –> Select Options –> Select Advanced –> With the ‘Send and receive’ section –> Untick the ‘Send immediately when connected’ box

msol mailbox folder size limit

A corporate Exchange server allocates typically 100MB for your account, covering
– data in your calendar
– data in your sent items

NOT covering
– archive folders (usually on your PC)
– personal folders (usually on your PC)

— To investigate
Choose your mailbox (or archive / personal folders) -> right-click -> properties -> folder size
Choose your mailbox (or archive / personal folders) -> right-click -> properties -> Advanced You can see storage locations.

— to “shrink” your data
choose your mailbox -> right click -> advanced find -> more choices -> size greater than 999KB -> advanced tab -> choose Received -> on or before -> 2008/12/31 or 2008-12-31 formats

You can
– remove attachments and add a comment [attachment removed tanbin 9feb]
– permanently delete them or
– move them into personal folders on your PC

Excel formula to get current cell location

An empty q[=row()]  would return current cell’s row number

 

dump multiple files: cat^more

q(more) dumps each filename before the content 🙂 Useful when you need to copy paste multiple file contents to an email.

c++timestamp string includ`microsec

        struct timeval timeValNow;
        gettimeofday (&timeValNow,NULL); //populates the struct
        struct tm * ptm = gmtime( &timeValNow.tv_sec );
        timestampNow.str("");
        timestampNow<<ptm->tm_year +1900;
        timestampNow<<setfill('0')<<setw(2)<<ptm->tm_mon+1;
        timestampNow<<setw(2)<<ptm->tm_mday<<'-';
        timestampNow<<setw(2)<<ptm->tm_hour<<':';
        timestampNow<<setw(2)<<ptm->tm_min<<':';
        timestampNow<<setw(2)<<ptm->tm_sec<<'.';
        timestampNow<<setw(6)<<timeValNow.tv_usec;

printf formatter for a float +! trailing zeros

I think the simple “%f” may just work, without any numeric specifiers.

## tips: search in q[less]

• after you have searched for “needle1”, how do you expand on the pattern? You can hit 2 keys
  • [2]  /↑ (i.e. <upArrow>) to load “needle1.” Now you an edit it or add an alternative like
  • [2+]/↑ (i.e. <upArrow>) |needled2|needle3
• In a q(less) screen, You can highlight not only the same AAA occurrences, but AAA or BBB. I used

/Reset|Msg_|Warn_|Primary|Secondary

• [2/3] /ctrl-R — search without metacharacters
• -i, then /needle — case insensitive

[3/4] means vi/less receives 3 keystrokes; we hit 4 keys including shift or ctrl …

–search can fail with super-long lines. See also http://www.greenwoodsoftware.com/less/bugs.html

• Symptom — if you navigate to the correct region of the big text file, then the search does hit the target word.
• Solution — use grep to confirm

 

navigating large files in q[less]

https://superuser.com/questions/113039/less-quickly-jump-to-line-number-in-large-file

5% .. is extremely useful.

q[diff] adapting to screen resize

alias diff=’diff -W $(( $(tput cols) – 2 ))’ # make use of full scren width

 

tput cols # does change as screen resizes 🙂

win10 system+user settings GUI

I prefer the controlPanel rather than the Settings screen… ControlPanel can show all the apps without the messy categories. Also, the winKey->search is unreliable when searching for a controlPanel app. Once again, Microsoft did a sloppy job.

To change any setting, I believe we always need one of those apps. Windows GUI try to hide the boundaries between the apps, and make it very very confusing.

read` caret notation@ascii characters

https://en.wikipedia.org/wiki/Caret_notation explains the caret notation.

q(less) and many other unix tools prints “funny characters” in caret notation.

MSExcel: OpenFileLocation to launch MSWE

MenuBar -> File -> left pane -> Info -> far right -> OpenFileLocation

sh-script: basic user input prompt

Other posts mention this topic as part of bigger discussions. This post is more focused.

printf "\nContinue? [ y /any_other_key] " ### easier than echo -en
unset REPLY; read $REPLY
[ "$REPLY" = "y" ] || return ### single equal sign!
unset REPLY ### remove any residual effect

vi: locate a long sentence that may span2lines

Suppose you know where the line break, if any, would be.

/abcd\_s*efgh — Finds abcd followed by any whitespace or newlines then efgh.

linux command elapse time

–q(time) command

output can be a bit confusing.

Q: does it work with redirection?

–$SECONDS variable

• See https://stackoverflow.com/questions/16908084/linux-bash-script-to-calculate-time-elapsed?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa
• This technique doesn’t interfere with redirection
• This technique can measure a group of commands

DOS replacement, again

See http://stackoverflow.com/questions/19772/cmd-exe-replacement

–console
not yet tested

http://www.maketecheasier.com/console-2-windows-command-prompt-alternative/
http://www.makeuseof.com/tag/sick-command-prompt-windows-7-upgrade-console/

–%% wishlist
copy/paste
edit command history
shell integration

—

🙂 conEmu can search throughout console output
🙂 conEmu here
🙂 conEmu double-click to select text
🙂 console2 requires research (no time!) on tweaking to fix copy/paste.
🙂 console2 here
😦 PowerCmd requires admin right every time to use

recursively run dos2unix on text files only

for file in `find . -type f`; do

file $file |grep CRLF

[ $? -eq 0 ] && dos2unix $file

done

recursive dir diff on windows

BeyondCompare3 can compare 2 “deep” directory trees. To make it compare file contents, you need to select all files (Ctrl-A) then hit the button “Compare Content”.

Other diff tools? No experience.

bookmarking in vi

— based on http://www.yolinux.com/TUTORIALS/LinuxTutorialAdvanced_vi.html

Any line can be “Book Marked” for a quick cursor return. Type the letter “m” and any other letter to identify the line. This “marked” line can be referenced by the keystroke sequence “‘” and the identifying letter.

Example: “mt” will mark a line by the identifier “t”. “‘t” will return the cursor to this line at any time. I prefer mm and ‘m

A block of text may be referred to by its marked lines. i.e.’t,’b

if unix log file includes no timestamp

http://unix.stackexchange.com/questions/26728/prepending-a-timestamp-to-each-line-of-output-from-a-command

— Not yet tested:
your_command | ts
What if your_command was started by someone else?
Tail -f the.log |ts

–Tested:
tail -f ,trash.log | perl -MPOSIX -pe ‘print strftime (“[%H:%M:%S] “, localtime)’

set up dependency between two c++ projects

I will mostly talk about MSVS but will mention linux build too.

The common methods to set up the dependency, all done inside project BB

1) specify project dependency in MSVS

2) define include path to cover the header from AA

3) define library path to cover the object file from AA

Suppose project BB uses “things” in project AA. Say a file in BB #-includes a header in AA. I think it's not necessary to have any. This is important when you debug the dependency. You may fail to find any trace of the dependency in any of those places. The file in BB can simply #-include the full/relative path to the header from AA.

http://blogs.msdn.com/b/vcblog/archive/2010/02/16/project-settings-changes-with-vs2010.aspx describes project reference vs project dependency….

….. much more to say.

in /bin/sh: print full path of any file #slinks

readlink -f {any file such as ../scripts/mvn.sh}

Usable in sh-script or interactively

See also http://stackoverflow.com/questions/242538/unix-shell-script-find-out-which-directory-the-script-file-resides

DOS FINDSTR/FIND, like grep

1) FINDSTR? I use findstr more

2) FIND?
Note Search-string is case sensitive.

!Must double-quote search-string. (Single quote doesn’t work.)

ftype | find “Python”
assoc | find “pyw”

net use E: \ficsindevnfsacwork1vtan7

label: timeSaver

switch`between LESS,vi,TAIL..#F

less -> vi : “v”

tail -> less? Nothing found.

–from within vi,
switch to readonly mode: qq( :se ro )
switch to readwrite mode: qq( :se noro )

—less to replace tail -f
$ less /var/log/messages

Common-mode -> Follow-mode: F
Follow-mode -> Common-mode: Ctrl-c

To start less in the Follow-mode,
$ less +F /var/log/messages

q[less] to handle color code q[32;1m ] or ESC[….

http://stackoverflow.com/questions/8220256/bash-color-escapes-codes shows

less -R your.file

hot key: open c:/a.txt #win7-10

https://1330152open.wordpress.com/wp-admin/post.php?post=19546&action=edit

q[screen] command cheatsheet

For each feature, let’s verify it and give links to only good web pages

Even after the “parent shell” of the screen is killed, the screen is still alive and “attached”!

–[ Essential] How to tell if I’m in a screen

echo $STY

http://serverfault.com/questions/377221/how-do-i-know-im-running-inside-a-linux-screen-or-not

–[ Essential ] reattach
http://www.gnu.org/software/screen/manual/screen.html#Invoking-Screen

— [ Essential ] list all child screens
screen -R # lists all or reattach if only one.
screen -ls # just list

— [ Essential ] detach i.e. put current screen session to the background
Ctrl-A then ‘d’

–feature
keep a session alive after I log out. Almost like windows mstsc.exe

–feature – log the screen output
See http://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/
The log file is created in the current dir …

–feature – Run a long running process without maintaining an active shell session
See http://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/ — using top as a long command

–(untested) feature – monitor a screen for inactivity as a sign of job done.
See http://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/

—– A real example:

Motivation — I need to run some daemon process in “background” but ideally inside a separate container/shell, so I can switch between that environment and my original environment. However, I don’t want to create another putty session. That’s indeed a more “separate” environment, but comes with higher cost like a lengthy log-in procedure.

[in console] screen # to create a new screen
[in screen-1] ./startDaemon # Note the command doesn’t return
[in screen-1] Ctrl-A d #to switch back to console
[in console] screen -ls # shows screen-1
[in console] screen -R # to switch into screen-1
[in screen-1] # now we are back in the running command which has not returned !
[in screen-1] Ctrl-Z # to suspend this stubborn command
[in screen-1] bg %1 # to put the command into proper background of screen-1

 

 

trailing comment in DOS batch

–http://stackoverflow.com/questions/12407800/which-comment-style-should-i-use-in-batch-files

set "var=3"     %= This is a comment in the same line=%

dir junk >nul 2>&1 && %= If found =% echo found || %= else =% echo not found

The leading = is not necessary, but I like if for the symmetry.
There are two restrictions:
1) the comment cannot contain %
2) the comment cannot contain :


			

[15] %% bashrc

append.bashrc.txt

<![CDATA[ << ______________end_of_comment_1________________ J4 etc: to reduce section spacing modified [11 Jan 2007] ______________end_of_comment_1________________ [ -z “$(echo $- | grep i)” ] && return # # # # # # # # # # # # # # # # # # # # <> should be first # # # # # # # # # # # # # # # # # # # # # # # export PS1=’\n\s!\! \u@\h [ \t \d ] \w/ \n\$ ‘ export EDITOR=vi # primarily for sqlplus export EXINIT=” :se nu | :map v :w ” # must export LESS=”–ignore-case” export TMOUT=987654; [ $root ] && export TMOUT=100600 HISTSIZE=900 set -o emacs set +o nounset export PATH=/usr/xpg4/bin:$PATH; [ $(id -u) = 0 ] && root=yesur set +o noclobber; [ $root ] && set -o noclobber sav=$LOGNAME # # PAGER and TERM issues: # # “vt100 less -E” is ok on linux # TERM ‘linux’ breaks vi in gnome-terminal; ‘linux’ imperfect for solaris # for “less”, -E needed for dir-viewing, file-viewing but affects man # export PAGER=more export PAGERE=more if [ -z “$(which less |grep ‘^no less’)” ]; then export PAGER=”less” # breaking some x terminals but ok in “console” export PAGERE=”less -E” # -E needed for dir-viewing, file-viewing but affects man fi echo TERM=$TERM initially export TERM=vt100 # ‘linux’ breaks vi in gnome-terminal; ‘linux’ imperfect for solaris # # # # # # # # # # # # # # # # # # # # # # # # <># # # # # # # # # # # # # # # # # # # # # # # alias killthem=”perl -pe ‘s/^\S+(\s+\d+).*/\1/ or s/.*//’|xargs -pt kill” # p str1 | killthem # alias killthem=”perl -pe ‘s/\S+(\s+\S+).*/\1/s’|xargs -pt kill” # p str1 | killthem # sortable : alias %=cd_l # typing trainer alias ..=’cd_l ..’ alias cp=’cp -i’ # defuse alias egi=’export |grep -i’ alias h=history alias hgi=”history |grep -i” alias j1=”fg %1″ alias j2=”fg %2″ alias m=$PAGERE alias mv=’mv -i’ # defuse alias p=ps_grep alias path=” echo \$PATH |perl -pe ‘s/:/\n/g’ |sort -u|m ” alias rm=myrm alias s=’ source $HOME/.bashrc ‘ # .profile absent alias t=’l -t’ alias top=’prstat’ # # # # # # # # # # # # # # # # # # # # # # # # <> difficult to sort # # # # # # # # # # # # # # # # # # # # # # # cd_l(){ [ $# -eq 0 ] && l && return [ -n “$( file $* | grep directory )” ] && cd $* && l && return [ -n “$( file $* | perl -ne ‘print if /text|script/’ )” ] && m $* && /bin/echo “\n\n” && l $* } d(){ [ $# -ne 0 ] && cd $* [ `pwd` = ‘/’ ] && target=/ && echo I am in / echo “In MB :” eval du -ks * |sort -n|perl -pe ‘s|^(\d+)|$1/1000|e’ } g(){ # bug with g -v pattern=$1; shift cmd=” grep -i \”$pattern\” $* ” /usr/bin/printf ‘%s\n’ “cmd=__$cmd __” eval $cmd |$PAGERE } l(){ # can’t be completely replaced by ‘cd_l’ or ‘]’, because “cd_l -tr dir1″ is confusing and should be avoided /bin/ls -alFs $* |$PAGERE } myrm(){ cmd=”mv -i $* /var/tmp/$sav/ ” /usr/bin/printf ‘%s\n’ “cmd=__$cmd __” eval $cmd } ps_grep(){ ## cmd1=’/bin/ps -ef’ # truncation risk # ps auxwww : inconsistent for root vs non-root cmd1=’/usr/ucb/ps auxwww’ # |grep -v grep — no cos some real commands contain grep for f in $*; do cmd1=”$cmd1 | g $f” done eval $cmd1 } sav(){ suffix=$RANDOM$(date +”%H%d%b”) # to avoid misleading readers,make the suffix ambiguous for f in $*; do f=$(echo $f|perl -pe ‘s|/$||’) # sav dir/ b=`basename $f` ( cd `dirname $f` ; tar cf – $b | (cd /tmp; tar xpf -) /bin/mv -i /tmp/$b $b.b4${sav}edit$suffix [ -d $f ] && opt=’ -d’ eval ls -lt $opt $b* ) done } # # # # # # # # # # # # # # # # # # # # # <>, an exercise in grouping # # # # # # # # # # # # # # # # # # # # # # # exa(){ #testing: exa -t ss ~/.ssh local ops while is_opt=`echo $1 |perl -ne ‘print if /^-/’ ` && [ “$is_opt” ]; do ops=”$ops $1″; shift done if [ $# -eq 1 ]; then fullpath=$1 shortname=`echo $1|perl -pe ‘s|/$||;s|^.*/||’ ` else fullpath=$2 shortname=$1 fi [ -x $fullpath ] || return # export “$shortname”=”$fullpath” set $shortname=”$fullpath” prj=”$fullpath” alias “$shortname”=”cd $fullpath;l $ops” alias prj=$shortname # USE MORE } exa /tmp/ exa /etc # # # # # # # # # # # # # # # # # # # # # <># # # # # # # # # # # # # # # # # # # # # # # add_path(){ [ -r $1 ] || return [ “$(echo $PATH|grep :$1)” ] && return # check world write perm, esp for root PATH=$PATH:$1 } set_manpath(){ # very slow on some systems. Run this when u need it. for d in `ls -d /[uo]*/*/man /[uo]*/*/*/man /[uo]*/*/*/*/man`; do export MANPATH=$MANPATH:$d done } set_path(){ add_path /usr/sbin add_path /usr/bin add_path /sbin add_path /usr/local/bin add_path /usr/cluster/bin add_path /usr/openwin/bin/ add_path /usr/ucb add_path $ORACLE_HOME/bin add_path /usr/sfw/bin # pre-installed add_path /opt/csw/bin [ $root ] && return add_path . add_path $HOME/bin } set_path ]]>

python print() 2tips

tip: print someStr, someInt # avoids str(someInt)

tip: print ‘  ‘.join([str1,  str2, str3]) # packed tighter than ..

print [str1,  str2, str3] # has space in between

print # empty would print a newline

locate a class in many jar files

for %i in (*.jar) do @jar tvf %i | find “TXDocument.class”

whereis/which on windows

for %i in (java.exe) do @echo. %~$PATH:i

DOS: Temporarily change working directory for a single command

pushd myjava & java -Djava.rmi.server.codebase=http://10.203.111.166:20380/murex.download.guiclient.download -classpath %CP% com.ocbc.quest.murexgateway.MurexServer 19673  & popd

 

This DOS command line does 3 things

1)      temporarily chdir into “myjava” folder and

2)      run a (long) java command line, and then

3)      restore the previous directory.

 

Actually, the java process blocks the script forever. If you use ctrl-C to terminate the blocking java process, you still get back into the previous directory J

 

See

https://www.google.com/url?q=http://superuser.com/questions/316431/temporarily-change-directory-for-single-batch-file-command&sa=U&ei=QE9GU6a0AYaErAev64CoBQ&ved=0CB4QFjAA&sig2=rynk5GiLWNBUreRFdwTTLw&usg=AFQjCNGuhZaXsvB2BIxXBvylAo79NcFRTA

 

log4net line number (%line) requirements

If you don’t satisfy all the conditions, then you get “0” as the dummy line number.

1) deploy the pdb files of all of my own DLL and EXE. If you put in some of them you may see some line numbers.
2) pdb file and the DLL/EXE should come from the same build. Version mismatch will trigger no exception. Just “0”.

Obviously you need something like this to see any logging at all —

log4net.Config.XmlConfigurator.Configure();

get class file physical location of a java object #home made

<![CDATA[ private static String getMetaData4Obj(Object o) { return getMetaData4Class(o.getClass()); } private static String getMetaData4Class(Class clazz) { String className = clazz.getName(); className = className.replace(“.”, “/”); className += “.class”; URL location = clazz.getClassLoader().getResource(className); return clazz + ” loaded from ” + location; } ]]>

c# programmatically retrieve file path of runtime objects

From a Method object you can get the host Type object.

From a Type object you can get the host Assembly object.

From Assembly object you can get the physical assembly file.

1) (tested) Assembly.GetExecutingAssembly().Location; // gives the dll fullpath
1c) new T().GetType().Assembly; //returns the Assembly object. Works even though T is a template dummy type!

2) Process.GetCurrentProcess().MainModule.FileName

3) AppDomain.CurrentDomain.BaseDirectory; // Similar to 1)

c# regex-match backslashes in strings

My suggestion — First find a “safe” character that’s guaranteed not to show up in the original string, like “_”. Replace all back slashes. Then proceed.

Problem with backslashes is the unnecessary complications. Here I want to match “one-or-more backslashes”. In the end I need to put 4 bachslashes in the pattern to represent that “one”.

var ret = Regex.Replace(@”any number of\\\backslashes”, “(.+\\\\+)?(.+)”, “$1 – $2”);

Alternatively, I could use @ to reduce the complexity @”(.+\\+)?(.+)”

Disappointingly the @ does a partial job. We still need 2 strokes — Confusing! I’d rather just remember one simple rule and avoid the @ altogether

%errorlevel% in DOS, briefly

Errorlevel is like a “dynamic readonly” whiteboard (NOT an environment variable — see below). The whiteboard is wiped clean and overwritten by DOS after each command executed. This happens whether you run the commands interactively or in batch using *.bat files.

You can say this whiteboard shows _transient_ values. For a crude analogy, consider the variable behind LED temperature display. It keeps changing.

If you want to capture a particular transient value of this object, save it *immediately* in your own variable (before echoing %errorlevel%). See http://www.coretekservices.com/2012/06/28/dos-batch-file-error-level-checking-tricks/

Q: What’s %ERRORLEVEL% vs bare ERRORLEVEL ?
A: %ERRORLEVEL% will expand into a String representation of the current value of ERRORLEVEL, provided that there is not already an environment variable with the name ERRORLEVEL, in which case you will get its value instead. 
A: To see the value, you must use %xxx%. Consistent with other DOS variables such as %OS%

cygwin to access a remote shared folder

http://superuser.com/questions/102233/how-to-go-to-remote-directory-in-cygwin explains

Windows' builtin pushd will auto-mount the share to a temporary drive, which is accessible from Cygwin.

C:> pushd \myserverphotosoriginal

Z:original>

mailx – send attachment + "body"

uuencode path/to/file file_name2display | mailx -s “subject1” bin.tan@xxx.com

…any body text….

^D # to end keyboard input

–tested in 2019:

any_command|mail -s ‘my subject’ bint

##5 win32 hot keys

alt + F4 — close active window
winKey + Break — to see environment variables

q[whereis] mysterious executable – locate it on %PATH%

for %i in (gcc.exe) do @echo %~$PATH:i

This shows where exactly the executable is loaded from.

(Tested in DOS)

–in win7, try “where”

xml pretty print simple routine using javax.xmltransform.*

&& and || together

[ -f unixfile ] && rm unixfile || print “unixfile was not found, or is not a regular file” — to be tested

Finally, multiple commands can be executed based on the result of command1 by incorporating braces and semi-colons:

command1 &

If the exit status of command1 is true (zero), commands 2, 3, and 4 will be performed.

See other post(s) on exit status

perl – too many ways to create array ref

Simple and clean — $ref = \@existingList

Avoid the other tricks until you find your footing.

\(100, 9900,2500) is misleading.

Anonymous arrayref is confusing.

perl one-liner : sum across lines

A common perl-n/perl-p challenge is to sum over loops. Solution is the END{} block.

This example below is from http://www.theperlreview.com/Articles/v0i1/one-liners.pdf —

ls -lAF | perl -ne ‘   next if /^d/; $sum += (split)[4]; END{ print $sum }   ‘

(I guess this sums up all the files’… sizes?)

Note you need not pre-declare $sum as a static-local (in C lingo). I guess it’s static-local by default.

was my unix cmd successful@@

This post focuses on one question:

Q: was previous cmd successful@@
A: (based on [[ teach yourself shell programming ]])

if [ $? -eq 0 ] ….

Background: i wasn’t sure if my own solutions were reliable. Now this author also believed in [ $? -eq 0 ]

cmd1 && … # tests 0. [[ learning the bash shell ]]
cmd1 || .. # tests 0.

“0 means OK” is standard convention. A script or command can break the convention, but you are allowed to assume no one does.

custom logger to show src line num, caller, parent caller and grandparent caller

<![CDATA[ /** based on http://javablog.co.uk/2008/07/12/logging-with-javautillogging/* Class name is mentioned in C:\Program Files\Java\jre1.6.0_06\lib\logging.properties */ public class JDKLoggingForamtter extends SimpleFormatter { private final String lineSeparator; public JDKLoggingForamtter(){ lineSeparator = java.security.AccessController.doPrivileged( new sun.security.action.GetPropertyAction(“line.separator”)); } @Override public synchronized String format(LogRecord record) { String sb = super.format(record); String lineNum = getEclipseFormat(); return sb.replaceFirst(lineSeparator, ” ” + lineNum + lineSeparator); } /** * Returns caller location information in eclipse format eg (Filename.java:23) * WARNING Generating caller location information is extremely slow. * It's use should be avoided unless execution speed is not an issue. * * @return the eclipse format */ private static String getEclipseFormat() { // getStackTrace can be expensive StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace(); String upstairs = “”; int lineNumber = 0; //StringBuilder sb = new StringBuilder(); // Here is an example of the typical stack trace we get back. // level 0 (Thread.java:1436) getStackTrace // level 1 (CustomFormatter.java:139) getFileLineNumber // level 2 (CustomFormatter.java:128) format // level 3 (StreamHandler.java:179) publish // level 4 (ConsoleHandler.java:88) publish // level 5 (Logger.java:458) log // level 6 (Logger.java:480) doLog // level 7 (Logger.java:503) log // level 8 (YourCodeHere.java:26) someMethod if (stackTrace.length >= 9) { String fileName = stackTrace[8].getFileName(); lineNumber = stackTrace[8].getLineNumber(); // Each of these calls back into logger.logp with the appropriate // level and message. This adds one extra level to the stack trace // logger.finest(“finest message”); // logger.finer(“finer message”); // logger.fine(“fine message”); // logger.info(“info message”); // logger.warning(“warning message”); // logger.severe(“severe message”); // logger.entering(“SomeClass”, “aMethod”); // logger.exiting(“SomeClass”, “aMethod”); // logger.exiting(“SomeClass”, “aMethod”, “returnval”); // logger.config(“config message”); // // Here is an example stack trace from a call to exiting // level 0 (Thread.java:1436) getStackTrace // level 1 (CustomFormatter.java:139) getFileLineNumber // level 2 (CustomFormatter.java:128) format // level 3 (StreamHandler.java:179) publish // level 4 (ConsoleHandler.java:88) publish // level 5 (Logger.java:458) log // level 6 (Logger.java:480) doLog // level 7 (Logger.java:623) logp // level 8 (Logger.java:938) exiting // level 9 (YourCodeHere.java:27) someMethod // // We could check the name of the method we are in and only go one level deeper // if it is one of “finest”, “finer”, “fine”, “info”, “warning”, // “severe”, “config”, “entering” or “exiting” but that would be too much trouble. // If the stack is in Logger.java and we can go one level deeper – do it. if (stackTrace.length >= 10 && “Logger.java”.equals(fileName)) { fileName = stackTrace[9].getFileName(); lineNumber = stackTrace[9].getLineNumber(); } if (stackTrace.length >= 10){ upstairs = ” = 11){ upstairs += “

pushd popd – G5 tips

• pushd dirA ###  1) push current dir, 2) push dirA, 3) Enter (i.e. cd into) dir1 i.e. the new top
• popd ### without arg — means 1) pop 2) Enter the new top in the stack. I guess the item “removed” is forgotten completely.
• pushd ### without arg — means 1) swap top two on the stack. 2) “Enter” (i.e. cd into) new top

 –Gory details
### I feel top item is always, always kept in sync with current dir. An understanding of the stack is essential. The stack is shared by cd, pushd, popd.

• cd – ### a single dash — means “Enter” previous directory (in history, not the stack). Overwrite top.
• pushd +2 ### means 1) identify the 3rd (not 2nd) on the stack. Rotate it to top. 2) Enter new top
• cd dir1 # replace top item on the stack with dir1
• dirs -v # print stack. Readonly operation.
• popd +2 # identify the 3rd (not 2nd) on the stack and remove it. Don’t change current directory, since top is unaffected.
• popd +0 # Probably pop the top. Enter the new top.

wildcard classpath — windows Java 6

java -classpath “C:/jars/*” your.main.class

 

Without double quote, you get funny errors. No such problem on linux.

java format numbers and headers for tabular output

       String formatNumber = “%10.4f “;  //good if numbers don't need
more space than “xxxx.yyyy”
       String formatHeader = “%10.10s “; //fixed 10 char columns,
truncating if necessary

java formatter tricks

“%.8s” — will copy only first 8 characters of the original string.

q{FIND} | perl,grep,notepad++

q(grep -r –color) is a simple solution but I still worry about lack of control.

# use grep inside perl, without xargs
find . -type f|perl -nle 'print "$_ --\n$a" if /\.(C|h)/ and $a=qx(grep -i "btmodels" "$_") '
find . -type f|perl -nle 'print "$_ --\n$a" if !/\.git/ and $a=qx(grep -i "btmodels" "$_") '

—-windows
MSWE search is unreliable for full-text search. Ditto for MSVS search. Don’t waste time on them!
Try notepad++. You can click the search result, and get keyword highlight, like in google!
Try findstr in http://stackoverflow.com/questions/698038/windows-recursive-grep-command-line

simplest unbuffered log file stream – java

new PrintWriter(new FileWriter(fileNameStr, isAppend), isAutoFlush);

defeated candidates:
* PrintStream is nicely unbuffered but not idea for text output.
* the nested FileWriter thing is … sadly buffered.

easiest mswe-style ftp interface

ftp://btan@mts-as1-u.apps.ml.com

svn folder comparison

Tortisesvn has s nice recursive check-for-modifications. It shows my
local changes, in an offline fashion.

No such feature in subclipse.

exceed copy-paste between win32 and X-windows

Using exceed, it can be a challenge to set up copy-paste between win32
and X windows. I know 2 options.

Note I always enable auto-copy-x-selection and
auto-paste-to-x-selection.

— option: X-selection-page -> X Selection Associated With Edit
Operations set to Primary —
Lesson? “Primary” is the default. In this mode, don't use the xwin
context-menu.

* Simple-Select (without middle-button or context menu) to copy from
unix, paste to win32? Yes
* Simple-Select (without middle-button or context menu) to copy from
unix, middle-button to paste to unix? yes
* Select from win32, middle-button to paste in unix? Yes
* Select from win32, context-menu->edit->paste in unix? no

— option: X-selection-page -> X Selection Associated With Edit
Operations set to Clipboard —
This is suggested on some webpage. It also enables copy-paste between
unix and windows.

Formatter.java – replacing StringBuilder

In most cases, I can replace StringBuilder with a Formatter, but of course Formatter offers more!

new Formatter(new StringBuilder()) painlessly appends!

import java.util.Formatter;
import static org.junit.Assert.*;
import org.junit.Before;
import org.junit.Test;

public class FormatterTest {
    private Formatter formatter;
    private StringBuilder sb;
    @Before
    public void setUp() {
        this.formatter = new Formatter(new StringBuilder());
        this.sb = new StringBuilder();
    }
    @Test
    public void testArray() {
        char[] var= new char[] {‘a’, ‘b’, ‘c’};
        sb.append(var+””);
        formatter.format(var+””, “”);
        assertEquals(sb.toString(), formatter.toString());
    }
    @Test
    public void testInt() {
        int var=902;
        sb.append(var);
        formatter.format(var+””, “”);
        assertEquals(sb.toString(), formatter.toString());
    }
    @Test
    public void testFloat() {
        float var=902;
        sb.append(var);
        formatter.format(var+””, “”);
        assertEquals(sb.toString(), formatter.toString());
    }
}

java getResourceAsStream() – easist(@@) way to read any file

You can easily read a file in the same dir (esp. in a jar) as your class or anywhere(?) on file system. Methods below are usually conveniently called on the Class object, like

1) simplest usage:
InputStream is = this.getClass().getResourceAsStream(“any file such as a.txt”);
InputStream is = AnyClass.class.getResourceAsStream(“any file such as a.txt”);

InputStream is = GetResourceTest.class.getResourceAsStream(“../../test.txt”);
if (is==null) System.err.println(“file not found”);

Behind the scene, these methods delegate to the classloader’s getResourceAsStream()
http://mindprod.com/jgloss/getresourceasstream.html has good examples.

putty — save host-specific config

A host-specific config is saved in a so-called saved-session. To access the list of saved-sessions, go to, paradoxically, _New_Session.._

To see or modify a particular saved-session, you must first Load it. You may need to hit Load several times. Once you are in the “editor” mode on a saved-session,

• * Navigate to a particular tree branch, like Window
• * change a particular value there
• * There’s no Save button. You must navigate back to the original Session page, and hit Save (repeatedly:). This Save button
  ** saves all the settings in the editor
  ** saves them under the session name.
• * If a parameter has an unsavable value, system will _silently_ drop the change and rollback.

http://tartarus.org/~simon/putty-snapshots/htmldoc/Chapter4.html#config-saving seems to be a good FAQ for this.

How about the config file for each connection? Apparently putty uses registry…

–DefaultSettings as a template: save it if you repeatedly use this template.

launch -> select DefaultSettings -> LOAD -> change any setting -> come back and (click to) highlight DefaultSettings -> SAVE

file descriptor redirection, exec

–Annotations on http://tldp.org/LDP/abs/html/ioredirintro.html —

bash$ lsof -a -p $$ -d0,1,2
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
bash 363 bozo 0u CHR 136,1 3 /dev/pts/1
bash 363 bozo 1u CHR 136,1 3 /dev/pts/1
bash 363 bozo 2u CHR 136,1 3 /dev/pts/1

bash$ exec 2> /dev/null
bash$ lsof -a -p $$ -d0,1,2
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
bash 371 bozo 0u CHR 136,1 3 /dev/pts/1
bash 371 bozo 1u CHR 136,1 3 /dev/pts/1
bash 371 bozo 2w CHR 1,3 120 /dev/null <—

—http://tldp.org/LDP/abs/html/x17601.html#USINGEXECREF shows —

exec 6>&1 # Link file descriptor #6 with stdout.
# I think this creates a new file descriptor FD#6 as alias of FD#1. FD#6 is probably a **pointer** to the in-memory object FD#1. The object IS the original file descriptor.

exec > $LOGFILE 2>&1 # stdout replaced with file “logfile.txt”.
# the object is not discarded. FD#6 still points to it, but the current process no longer uses that object.

#### this is a useful thing to put into your script, if someone calls your script.

# now the current process will use the original “object” from now on.
exec 1>&6 6>&- # Restore stdout and close file descriptor #6.

pretty-print xml c# one-liner

XDocument.Parse(_lastResponseNonNull.Content).ToString()

To handle possible exceptions, wrap it in try/catch — http://stackoverflow.com/questions/1123718/format-xml-string-to-print-friendly-xml-string

unix subshell — tips

http://www.ibm.com/developerworks/aix/tutorials/au-unixtips4/section5.html —

To run a group of commands in a subshell, enclose them in parentheses. You can use redirection to send input to the subshell's standard input or to send its *collective* output to a file or a pipeline.

Since the subshell environment is a duplicate of its parent, it inherits all of the variables of the parent. But the parent shell never sees any changes that are made in the subshell environment, and the subshell, in turn, never sees any changes that are made in the parent *after* the subshell is spawned.

To save return code of a command (among many) from a subshell,

echo $? > /tmp/hamper1390848908.$$
# other commands….
) > ${LOG_FILE} 2>&1
read GMI_RET_CODE </tmp/hamper1390848908.$$

Here's a simpler alternative

(
someJavaJob
javaStatus=$?
## any other command
[ $javaStatus -ne 0 ] && exit $javaStatus
) > ${LOG_FILE} 2>&1
subshellStatus=$?
echo $subshellStatus
exit $subshellStatus

practical – how many ways to pass config parameters to java

-Ddir=/tmp ######## my favorite

Trailing command line args

MY_DIR=/tmp ######## env var in OS

Properties file ##### must include in class path

Spring xml file

diff tool to show moved lines

windiff supports “moved lines”. when u click on a line in fileA, system jumps to the line in fileB.

Also, the left pane shows slanted lines only for moved lines

boolean tests of return status — perl^shell

– shell: if cmd1; then # evaluates cmd1 as OK if 0, NOK otherwise.
– Perl: if (&sub1(…)) # evalutes sub1 as NOK if 0 or undef or empty list

Perl convention is rooted in perl’s Truth.

bash – test a command is available

Based on [[ teach yourself shell programming ]]
Background — i was looking for a 1)reliable and 2) efficient solution.

type yourCmd > /dev/null 2>&1
echo $? #### should always echo a zero (ok) or non-zero (nok)

type cd > /dev/null 2>&1 && echo ok
type cdddd > /dev/null 2>&1 || echo nok

These were tested personally but some readers (see comment below) reported they don’t always work.

date arithmetic ] java

Problem: calendar solution has too many lines of code — clutter

Solution: long dayBefore = date1.getTime() – 24 * 60 * 60 * 1000;

exit status @ commands ] pipe

http://www.networkcomputing.com/unixworld/answers/003.html says:

“You are correct that the exit status of a pipeline is reported as the exit status of the last command in the pipeline. What happens to the exit status of the first (or other) command? It is ignored by the shell. However, you should be able to capture it with a little ingenuity.”

how to instantiate a new java Date object

Q: how to instantiate a new java Date object with 3 numbers for year, month and date?

I guess Calendar could be one standard solution, but not a one-liner.

Practical solution for the time being: use the deprecated constructors

— new java.util.Date?

Only deprecated constructors do that.
java.sql.Date.valueOf(“2008-04-02” )

–new java.sql.Date?

Only deprecated constructors do that.
java.util.Date.getTime() can feed a long to a java.sql.Date() constructor

exit status, if/while, test, &&, ||

I think most if not all shell boolean-constructs evaluate an exit-status. Here are an incomplete list of boolean-constructs — ie constructs about yes/no. Unifying them all, the 2 unifying factors are boolean and exit-status, right?

autosys failure/success? yes
the shell’s IF construct? This is the simplest, base case.
WHILE? same as IF
TEST? same as []
[]? yes. IF [ something ] evaluates the exit status of [ something ]
&& , || ? yes

See other post(s) on exit status

see major.minor versions of a classfile

http://blogs.sun.com/sundararajan/entry/thou_shall_know_the_class

http://forum.java.sun.com/thread.jspa?threadID=5178593&messageID=9692487 says : Just open the class file with something that can display hexidecimal data and look at bytes 5 – 8 in the file. They contain the minor:major versions in hex.

I used textpad and saw 2E ie 48 for major version. javap reported 0 and 0.

vi corrupting your file

I once had a strange feed file. If I use vi to delete any line, the feed can no longer be processed by the feeder system. I suspect once vi write the file back to disk, it’s corrupted.

Solution: In my case split -1 and head -1 can do a good enough job of deleting lines. Both keep the feed file in good condition.

Q: is vi designed to edit a binary file?

Q: can perl keep all unaffected lines unaffected?

Acid test: less can show the unprintable characters and shows that after deleting one line, every other line has something missing.

q[FIND] → ls

find … > /tmp/,341 ; ls -tr `cat /tmp/,341 ` # most powerful

find … -exec ls -l {} \; # space after {}

I think you can also xargs ….

… or make your own xargs with perl -ne “print qx(….)”

ls — sort by size

ls -S                # GNU/FSF version. Unsupported in a few unix flavors.

ls -s | sort         # useful even if imperfect
ls -s | sort -n 

ls -al | sort -n +4

q[SET]^q[EXPORT]^plain assignment #no-arg SET

(Not sure about csh. Let’s focus on /bin/sh and bash for now. I bet everything here also applies to ksh.)

“set” is mostly used to set options. I see it less often used to manipulate variables, which is possible but unnecessarily complicated. In fact

set a=1; echo $a # shows nothing

• “set” alone displays variables, but “set” doesn’t modify/create variables. [1]
• “export” has a single meaning and you see it used consistently (whereas “set” isn’t).

Are you managing variables in bash? A simple “binary” advice is
– use export for ENV variables
– use “nothing” to manage shell variable — Nothing but “a=1”

How about “let”? Only needed for arithmetic. I would say don’t let this stranger join the club and confuse yourself.

[1] In DOS, “set” command creates variables (http://www.easydos.com/set.html)

perl one-liners for xml

based on http://www.xml.com/pub/a/2002/04/17/perl-xml.html.

Find all section titles in a DocBook XML:

# cat files/mybook.xml | xpath //section/title

Retrieve just the significant text (not including nodes containing all-whitespace) from a given document:

# cat files/mybook.xml | xpath “//text()[string-length(normalize-space(.)) > 0 ]”

Save the entire data stored in the ‘users’ table as a huge file users.xml:

# sql2xml.pl -sn myserver -driver Oracle -uid user -pwd seekrit -table user -output users.xml

Pretty-print a bad xml file:

# cat overwrought.xml | xmlpretty > new.xml

Use the built-in HTML parser to convert ill-formed HTML to XML before further processing:

# xmllint –html khampton_perl_xml_17.html | xpath “//a[@href]”

AND MORE….

if / while in bash

if [[[ some condition ]]]

elif [[[ some condition ]]]

while [[[ some condition ]]]

2 types of things can replace [[[ some condition ]]]
A) a boolean test like ” [ -r /tmp ] ”
B) any other command, since every command returns an exit status

The common characteristic? Both A and B provide an exit-status to the if/elif/while

bash search`unix command #precedence

Q: what if you have an alias, a shell func, a shell built-in (like “echo”), a unix executable file, and your own script all sharing the same name?

Focus on 2 simple rules for now:

1) Your own aliases override everything else (such as shell functions)

2) $PATH is the very last (after things like shell built-in commands) place searched by the shell. Executables in $PATH are external to the shell.

I think the same precedence holdes for any shell.

easiest way to compare 2 files in a windows box

I wanted to compare the old version of myClass.java with a new version but the two files live on 2 drives.

Now I just browse to the first myClass.java, right click to mark it for “compare later”. Then i browse to the 2nd file, and right click to compare them.

Immediately the 2 files are loaded in a GUI side-by-side with differences highlighted in color. You can scroll both panes at the same time.

More visual/convenient than WinDiff or CVS-diff in Netbeans.

* The GUI tool is winMerge (http://prdownloads.sourceforge.net/winmerge/WinMerge-2.6.8-Setup.exe)
* The WinExplorer context-menu plug-in is Diff-ext (http://diff-ext.sourceforge.net/downloads.shtml)

Diff-ext also lets u compare 3 files or directories.

StringBuffer / StringBuilder offers no prepend()

Use insert(). Example —

   myStringBuffer.insert (0, “prefix string”); // C# and java are similar. See API links below

This technique is rather important to job interview coding tests. (I encountered this in the ITA 50 billion quiz)

Warning — http://stackoverflow.com/questions/738950/c-sharp-or-java-prepend-strings-with-stringbuilder points out that prepend is often less efficient as append

myBuf.insert(5, …) means “insert after 5“, that is, insert after first 5 characters of myBuf.
myBuf.insert(0, …) means insert after 0 character, which is the geeky way of saying prepend

http://docs.oracle.com/javase/6/docs/api/java/lang/StringBuffer.html
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/StringBuilder.html — faster but thread unsafe

http://msdn.microsoft.com/en-us/library/7tyt41s0(v=vs.110).aspx

TIME command ok for background job

$ time java some_class &

Command will return immediately, but u get the timing statistics along with the [Done] message.

putty bat file for auto-login

start putty -v -load “nbc210.193” -l … -pw
: “start”? close dos window asap
: “load”? to load proxy saved in the session

full-screen cmd window

full-screen may hide the mouse, the start menu …

Solution: alt-tab. Then right click on the tast-bar icon for the offending cmd window

log4j.xml quick guide

q: threshold?
A: The “threshold” attribute takes a level value such that all
logging statements with a level equal or below this value are
disabled.

Q: root?
A: root of the xml tree is CATEGORY, which typicall references
1 or more appenders, 2nd level node in the xml tree.

Q: eg@appenders?
A: console, snmp, syslog,

q: log4j.xml changes need restart?
A: no

Q: category vs logger?
A: say “category” in xml; say “logger” in src code
#1 tip: child logger inherits from parent logger

Q: Category names are FQCN?
A: yes. “Engine” shown in console but “com.domain.Engine” shown in log file.