Memo

I often spend lots of time on setting up the same software over and over again for different purposes. In order to save time, I decide to write down some useful procedures or hints here. Hope that they could be helpful to myself and others who visit here.

Build OpenCV with SIFT

posted Sep 27, 2019, 12:37 AM by Teng-Yok Lee

Since version 3.0, OpenCV source code remove the parts for feature matching with SIFT/SURF. This guide is about how to build OpenCV with them. It will indicate how to build the python binding.

Windows

  1. Create a folder to hold both source code and build folder. For instance, G:/src/opencv/.
  2. Clone OpenCV source code from https://github.com/Itseez/opencv.git to G:/src/opencv/src.
  3. Check out the tag to 3.2.0.
  4. Clone extra module's source code from https://github.com/opencv/opencv_contrib to G:/src/opencv/contrib.
  5. Check out the tag to 3.2.0.
  6. Open cmake.
    1. Set the source folder as G:/src/opencv/src.
    2. Set the binary folder as  G:/src/opencv/build/win32/vs2013/x64.
  7. Click Configure, and then
    1. Turn OFF: BUILD_WITH_STATIC_CRT
    2. Turn OFF: BUILD_SHARED_LIBS
    3. Turn OFF: BUILD_PERF_TESTS
    4. Turn OFF: BUILD_TESTS
    5. Turn OFF: WITH_CUDA
    6. Turn OFF: WITH_CUFFT
    7. Turn OFF: WITH_LAPACK
    8. Turn OFF: BUILD_opencv_bioinspired
    9. Turn ON: BUILD_ZLIB
    10. Set OPENCV_EXTRA_MODULES_PATH: G:/src/opencv/contrib/modules
    11. Set CMAKE_INSTALL_PREFIX to where you want to install OpenCV
  8. Also configure with Python2. I am using WinPython, which is installed under G:/usr/win32/WinPython-64bit-2.7.10.3/.
    1. Set PYTHON2_EXECUTABLE: G:/usr/win32/WinPython-64bit-2.7.10.3/python-2.7.10.amd64/python.exe
    2. Set PYTHON2_INCLUDE_DIR: G:/usr/win32/WinPython-64bit-2.7.10.3/python-2.7.10.amd64/include
    3. Set PYTHON2_LIBRARY: G:/usr/win32/WinPython-64bit-2.7.10.3/python-2.7.10.amd64/libs/python27.lib
    4. Set PYTHON2_PACKAGES_PATH: G:/usr/win32/WinPython-64bit-2.7.10.3/python-2.7.10.amd64/Lib/site-packages
    5. Set PYTHON2_NUMPY_INCLUDE_DIRS: G:/usr/win32/WinPython-64bit-2.7.10.3/python-2.7.10.amd64/Lib/site-packages/numpy/core/include
  9. Configure.
  10. Generate the Visual Studio projects/solutions.
  11. Open H:/src/opencv/build/vs2013/x64/opencv.sln.
  12. Build all projects.
  13. Build INSTALL. Note that it will try to overwrite cv2.pyd in the site-package folder of python. Backup first if needed.

Ubuntu

  1. Create a folder to hold both source code and build folder. For instance, ~/src/opencv/.
  2. Clone OpenCV source code from https://github.com/Itseez/opencv.git to ~/src/opencv/src.
  3. Check out the tag to 3.0.0.
  4. Clone extra module's source code from https://github.com/opencv/opencv_contrib to ~/src/opencv/contrib.
  5. Check out the tag to 3.0.0.
  6. See commands below.
# Assume that current folder is ~/src/opencv/build] $ mkdir linux $ cd linux # Use cmake to configure the files. # Note: Do not use ccmake here! # Otherwise it cannot configure python correctly # and thus will not build cv2.so. $ cmake \ -DBUILD_SHARED_LIBS=OFF \ -DBUILD_PERF_TESTS=OFF \ -DBUILD_TESTS=OFF \ -DWITH_CUDA=OFF \ -DWITH_CUFFT=OFF \ -DWITH_OPENCL=OFF \ -DBUILD_ZLIB=ON \ -DOPENCV_EXTRA_MODULES_PATH=/homes/tlee/src/opencv/contrib/modules \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/homes/tlee/usr/linux/opencv \ -DWITH_LAPACK=OFF \ ../../src/ $ make -j8 $ make install # NOTE: Different from Windows, # cv2.pyd will be copied to /homes/tlee/usr/linux/opencv/lib/python2.7/dist-packages # because CMAKE_INSTALL_PREFIX is /homes/tlee/usr/linux/opencv.

Test?

# First set PYTHONPATH # by putting the path to the new-built cv2.so # before original paths. $ export PYTHONPATH=$PWD/lib:$PYTHONPATH $ python >>> import cv2; # Confirm the path. >>> cv2.__file__ # Confirm that SIFT can be used. >>> sift = cv2.xfeatures2d.SIFT_create(); # Also confirm that videos can be opened. # Replace the string by the file path of any video on your computer. >>> cap = cv2.VideoCapture('/data/datasets/tmp/RecFile_6_20170122_153530_PointGrey2_output.mp4') >>> cap.isOpened() True

My notes abotu the procedure to work in Japan

posted Sep 1, 2019, 5:29 PM by Teng-Yok Lee   [ updated Sep 1, 2019, 5:33 PM ]

Terminologies

在留(ざいりゅう)カード: Resident card
転入届(てんにゅうとどけ): Move-in registration
健康保険(けんこうほけん): Health insurance
銀行口座(ぎんこうこうざ): Bank account
自己負担(じこふたん): self-pay
補助(ほじょ): Support
請求書(せいきゅうしょ):invoice

Visa

First, a COE (certificate of eligibility) is needed from the employer.

Once the COE is received, apply the Japan working visa in the local embassy. Note:
  • Different embassy might have different requirements of documents. e.g.
  • A legal immigration status in US is typically needed .
  • The arrival time/date can be flexible.
  • The working visa is for 1-time entry. Later will use 在留カード to enter Japan.
  • The visa requires 3 working day to obtain.

Shipping

The shipper might need to know the following information.
  • Need to check with the employer about the detailed upper limits (e.g. how big for the oversea shipment and how heavy for the air cargo).
  • Need to know whether I should pay first or directly bill to the employer.
The shipper will provide two customer declaration forms, which should list the number of boxes for flights and ships. On the airport.
  • Hand in the 2 completed sheet to the custom officer to get the stamp.
  • One will be returned with the stamp.
  • This returned one should be brought to the shipper's counter at the arrival lobby of the airport.

Flight

While the employer should reimburse this, the rules for the source and destination are more restricted. For instance, originally I want to fly to Taiwan first and enter Japan one week later, which is actually way cheaper than directly flying to Japan, but my Japanese colleagues worry that the accounting department might reject because of the 1-week gap. Thus at the end I decide to enter Japan first, settle down everything, and then go to Taiwan.

在留カード

在留カード is crucial for foreigners to stay in Japan. Without 在留カード, it might be not possible to create bank account and apply for phones.
在留カード can be  obtain in some international airports. Once the address in Japan is confirmed, go to the local government to register (転入届)and add the address to the back of 在留カード.

Housing

In my case, my employer rents an apartment for me so the entire procedure is simpler.

On the other hand, one thing I learned during the process is that the employee should provide a concrete plan, including the date, rent range, and size range. The date should be a single choice for the HR to search. At the beginning I provided two possible dates to the HR. According to my Japanese colleague, it is better that I can reduce the date to a single & confirmed one so it is easier for HR to search.

Others

  • 健康保険/Health insurance: Can purchase from the local government, but employers might have their own plan.
  • 銀行口座/Bank account: Most banks would have rules for foreigner. Meanwhile, employers might have their preferred bank to deposit the salary.
  • 携帯: TO ADD







Let matplotlib.pyplot.show() to pause with pydev

posted Aug 23, 2019, 7:29 PM by Teng-Yok Lee

After matplotlib.pyplot.show(), add the following to pause the script:
 
    try:
        # This is to pause the script
        # when debugging with PyDev.
        # It should be put after plt.show()
        # so it will be only executed in debug mode.
        import pydevd;
        dummy_fig = plt.figure();
        dummy_fig.canvas.start_event_loop_default();
    except:
        pass;

My procedure to enable disk quota

posted Aug 4, 2019, 9:40 AM by Teng-Yok Lee   [ updated Sep 23, 2019, 10:02 PM ]

Mount disk with quota options

The options are usrquota and grpquota.

# In /etc/fstab:

/dev/sdb1       /data ext4      defaults,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv1      0       0



Remount the filesystem to make it effective:
$ sudo mount -vo remount /data

Check the quota

NOTE: sudo is needed in order to temporally make the file system read-only. The option -cug will check both users & groups (REF).
$ sudo quotacheck -cug /data

Enable quota

$ sudo quotaon /data

Check quota

To check user-wise quota:
$ sudo repquota /data
To check group-wise quota, add the option -g:
$ sudo repquota -g /data

Edit the quota

Option 1: Use setquota for command-line based editing. The usage is:

$ setquota <user> <soft-inode-limit> <hard-inode-limit> <soft-file-limit> <hard-file-limit> <file-system>

Option 2: Use edquota for interactive editing

Set the preferred editor (REF):
$ export EDITOR="emacs -nw"
Edit the quota for a specific user. Note the the env. var. should be passed via -E.
$ sudo -E edquota tlee

Disable quota check when booting

Editing /etc/default/grub:

$ sudo emacs -nw /etc/default/grub

Append "quotacheck.mode=skip" to GRUB_CMDLINE_LINUX.

Update grub:

$ sudo update-grub

Once reboot, check /proc/cmdline to verify the kernel options.

References

https://wiki.archlinux.org/index.php/Disk_quota
https://web.mit.edu/rhel-doc/5/RHEL-5-manual/Deployment_Guide-en-US/ch-disk-quotas.html
https://www.linuxsecrets.com/archlinux-wiki/wiki.archlinux.org/index.php/Disk_quota.html#Usage

Some slurm tricks

posted Jul 19, 2019, 7:31 AM by Teng-Yok Lee   [ updated Jul 19, 2019, 7:31 AM ]

How to get the stdout of a slurm job?

function slurm_get_job_stdout()
{
    jobid=$1
    scontrol show jobid ${jobid} | grep StdOut | cut -b 11-
}

How to list the CPU/GPU resources of all nodes?

function slurm_list_resources()
{
    extra_args=$@

    # This is a workaround for scontrol to query hidden nodes.
    w_hidden=""
    for a in ${extra_args}
    do
        if [ "-a" == "$a" ]
        then
            w_hidden=$a
        fi
    done

    for h in $(sinfo ${extra_args} -N -h | cut -d " " -f 1)
    do
        n_used_gpus=$(squeue ${extra_args} -t R -h -o "%8b" -w $h | grep gpu | cut -d : -f 2 | paste -s -d + - | bc)
        scontrol_output=$(scontrol ${w_hidden} show node $h | tr ' ' '\n')
        n_gpus=$(printf "%s" "${scontrol_output}" | grep Gres | cut -d : -f 2)
        n_used_cpus=$(printf "%s" "${scontrol_output}" | grep CPUAlloc | cut -d = -f 2)
        n_cpus=$(printf "%s" "${scontrol_output}" | grep CPUTot | cut -d = -f 2)
        cpu_load=$(printf "%s" "${scontrol_output}" | grep CPULoad | cut -d = -f 2)
        printf "[%s]\tGPU: %d/%d\tCPU: %3d/%3d\tLoad: %2.2f%%\n" "$h" "${n_used_gpus}" "${n_gpus}" "${n_used_cpus}" "${n_cpus}" ${cpu_load}
    done
}


Notes for nvidia-smi

posted Jul 12, 2019, 10:42 AM by Teng-Yok Lee

Check P2P status

# REF: https://devtalk.nvidia.com/default/topic/1031758/p2p-not-working-for-p600s-/

$ nvidia-smi topo -p2p w




Use a 1-liner of gnuplot to plot loss in Caffe's log

posted Jul 10, 2019, 5:25 AM by Teng-Yok Lee

This can be very helpful when analyzing logs so need to write extra script.

$ gnuplot -p -e "plot '<cat' with lines"
It will plot the values from stdin. Typically I used it when analyzing the logs from Caffe. For instance, In the example  below, I extract loss #0 from the training network from file caffe.bin.INFO, use cut to filter the value, and then pass it to gnuplot.

$ grep "Train net output #0" caffe.bin.INFO | cut -d = -f 2 | cut -d " " -f 2 | gnuplot -p -e "plot '<cat' with lines"

Comparison of X11 servers on Windows

posted May 11, 2019, 6:52 AM by Teng-Yok Lee

NOTE: The column Easy so use is based on the amount of configurations needed after installation. More * means more effort. So far it seems that VcXsrv is the best one in terms of all main features I need.

Software Installation Price/ license Clipboard Resizing of windows? OpenGL Easy to use Notes
VcXsrv A single installer. FREE GPL3 Yes Yes Yes w/o native opengl. **
  • Might be blocked by old Micro Trend.
  • Should disable native opengl.
MobaXTerm A single installer. FREE Failed? Yes Yes w/o X extensions. ***
  • For heavy PowerPoint users, it would be better to remove the hotkey Ctrl+M. Ctrl+M is also used to insert new slides in PowerPoint, but by default MobaXTerm grab this hotkey.
  • Better to disable X extensions for OpenGL.
  • See more limits of the free version HERE


$69/yr per user ? ? ? ?
Xming A single installer. FREE Yes No Yes *
  • Or try older versions (REF)?


$20/yr ? ? ? ?
cygwin/X Multiple cygwin packages

FREE

X11

Yes Yes Yes with limited & incorrect color depth *****
  • See installation guide HERE.
  • More complicated to configure (e.g. for putty, need to manually specify the path t .Xauthority for each configuration).
  • Might be blocked by old Micro Trend.
  • If the X server restarts, current SSH session will fail and need to re-connect.

OpenGL with X11 servers on Windows

posted May 11, 2019, 6:47 AM by Teng-Yok Lee   [ updated May 11, 2019, 6:53 AM ]

I typically use Windows with ssh -X to access Linux machines so I can avoid dual boot. If I need to run remote applications with X windows, I run VcXsrv (REF) on windows as the X11 server. This typically works except programs that needs OpenGL. If software-based rendering is affordable, my workaround is as follows:

  • On the local windows: Run VcXsrc without native OpenGL supports.
  • ssh to the the remote Linux:
    • Make sure that libgl1-mesa-dev is installed.
      • NOTE: If the nvidia driver is install via .run, it might override libgl1-mesa-dev.
        • To verify, run the following to see whether it is using libGL under mesa or not:
          $ ldd $(which glxinfo) | grep libGL
        • If not, the symlink of /usr/lib/x86_64-linux-gnu/mesa might be broken. In such a case, re-installation of libgl1-mesa-devis needed.
    • Setup env. var. to use mesa's libGL.so:
      $ export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/mesa:$LD_LIBRARY_PATH
    • To confirm, run glxinfo again to see whether direct rendering is Yes:
      $ glxinfo | grep -i direct

New Google Sites

posted Apr 18, 2019, 9:33 PM by Teng-Yok Lee

I did a quick test with the new Google Sites, but I decide to stay with the classic one because:

1. No complete history to rollback. This is the biggest reason.
2. Lose the banner in the left side (I used this to show weather of my current city).
3. The format of my publication list is destroyed.
4. Unsure whether I can post announcement, like what I am doing now.

A related question is: How to re-do the conversion? I created a draft but I disliked it.

The solution is in the page Manage Site/Convert to new site. After entering the stage Review and Publish, open the advanced pane, and choose to disconnect from the current page. Then it will restart from stage 1.

1-10 of 120