I often spend lots of time on setting up the same software over and over again for different purposes. In order to save time, I decide to write down some useful procedures or hints here. Hope that they could be helpful to myself and others who visit here.

Log to both stdout and file in a bash script

posted Aug 7, 2020, 6:12 AM by Teng-Yok Lee

LOG_FILE=/tmp/both.log exec > >(tee ${LOG_FILE}) 2>&1

Tips for tensorflow

posted Mar 9, 2020, 8:30 AM by Teng-Yok Lee

This is my note for various tips about tensorflow.

How lo list all variables in a checkpoint?



How to list all tensors in a graph?

[ for n in tf.get_default_graph().as_graph_def().node]

Use slurm to run mpi jobs

posted Mar 9, 2020, 7:21 AM by Teng-Yok Lee   [ updated Mar 9, 2020, 8:34 AM ]

MPI is a library to run parallel jobs on multiple computers, which is used in some distributed machine learning libraries like horovod. This page will briefly explain how to run programs with MPI on a slurm cluster:

To interactively run jobs, use salloc:
$ salloc <resources> mpirun -n <resources> <cmd> <arg> ...
For instance, the following command will launch 2 nodes with 2 CPUs each, and thus 2 host names should be printed:
$ salloc -n 4 --ntasks-per-node 2 mpirun -n 4 hostname
NOTE This is different from srun, which use -c  to specify #CPU per task.
To request generic consumable resources like GPU, --gres is used to specify the resources per node. For instance, the following command will request 2 nodes with 2 GPUs each.
$ salloc -n 4 --gres gpu:2 --tasks-per-node 2 mpirun -n 4 hostname
To submit a mpi job, use sbatch:
$ sbatch <resources> mpirun -n <resources> <cmd> <arg> ...


Build OpenCV with SIFT

posted Sep 27, 2019, 12:37 AM by Teng-Yok Lee

Since version 3.0, OpenCV source code remove the parts for feature matching with SIFT/SURF. This guide is about how to build OpenCV with them. It will indicate how to build the python binding.


  1. Create a folder to hold both source code and build folder. For instance, G:/src/opencv/.
  2. Clone OpenCV source code from to G:/src/opencv/src.
  3. Check out the tag to 3.2.0.
  4. Clone extra module's source code from to G:/src/opencv/contrib.
  5. Check out the tag to 3.2.0.
  6. Open cmake.
    1. Set the source folder as G:/src/opencv/src.
    2. Set the binary folder as  G:/src/opencv/build/win32/vs2013/x64.
  7. Click Configure, and then
    4. Turn OFF: BUILD_TESTS
    5. Turn OFF: WITH_CUDA
    6. Turn OFF: WITH_CUFFT
    7. Turn OFF: WITH_LAPACK
    8. Turn OFF: BUILD_opencv_bioinspired
    9. Turn ON: BUILD_ZLIB
    10. Set OPENCV_EXTRA_MODULES_PATH: G:/src/opencv/contrib/modules
    11. Set CMAKE_INSTALL_PREFIX to where you want to install OpenCV
  8. Also configure with Python2. I am using WinPython, which is installed under G:/usr/win32/WinPython-64bit-
    1. Set PYTHON2_EXECUTABLE: G:/usr/win32/WinPython-64bit-
    2. Set PYTHON2_INCLUDE_DIR: G:/usr/win32/WinPython-64bit-
    3. Set PYTHON2_LIBRARY: G:/usr/win32/WinPython-64bit-
    4. Set PYTHON2_PACKAGES_PATH: G:/usr/win32/WinPython-64bit-
    5. Set PYTHON2_NUMPY_INCLUDE_DIRS: G:/usr/win32/WinPython-64bit-
  9. Configure.
  10. Generate the Visual Studio projects/solutions.
  11. Open H:/src/opencv/build/vs2013/x64/opencv.sln.
  12. Build all projects.
  13. Build INSTALL. Note that it will try to overwrite cv2.pyd in the site-package folder of python. Backup first if needed.


  1. Create a folder to hold both source code and build folder. For instance, ~/src/opencv/.
  2. Clone OpenCV source code from to ~/src/opencv/src.
  3. Check out the tag to 3.0.0.
  4. Clone extra module's source code from to ~/src/opencv/contrib.
  5. Check out the tag to 3.0.0.
  6. See commands below.
# Assume that current folder is ~/src/opencv/build] $ mkdir linux $ cd linux # Use cmake to configure the files. # Note: Do not use ccmake here! # Otherwise it cannot configure python correctly # and thus will not build $ cmake \ -DBUILD_SHARED_LIBS=OFF \ -DBUILD_PERF_TESTS=OFF \ -DBUILD_TESTS=OFF \ -DWITH_CUDA=OFF \ -DWITH_CUFFT=OFF \ -DWITH_OPENCL=OFF \ -DBUILD_ZLIB=ON \ -DOPENCV_EXTRA_MODULES_PATH=/homes/tlee/src/opencv/contrib/modules \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/homes/tlee/usr/linux/opencv \ -DWITH_LAPACK=OFF \ ../../src/ $ make -j8 $ make install # NOTE: Different from Windows, # cv2.pyd will be copied to /homes/tlee/usr/linux/opencv/lib/python2.7/dist-packages # because CMAKE_INSTALL_PREFIX is /homes/tlee/usr/linux/opencv.


# First set PYTHONPATH # by putting the path to the new-built # before original paths. $ export PYTHONPATH=$PWD/lib:$PYTHONPATH $ python >>> import cv2; # Confirm the path. >>> cv2.__file__ # Confirm that SIFT can be used. >>> sift = cv2.xfeatures2d.SIFT_create(); # Also confirm that videos can be opened. # Replace the string by the file path of any video on your computer. >>> cap = cv2.VideoCapture('/data/datasets/tmp/RecFile_6_20170122_153530_PointGrey2_output.mp4') >>> cap.isOpened() True

My notes abotu the procedure to work in Japan

posted Sep 1, 2019, 5:29 PM by Teng-Yok Lee   [ updated Sep 1, 2019, 5:33 PM ]


在留(ざいりゅう)カード: Resident card
転入届(てんにゅうとどけ): Move-in registration
健康保険(けんこうほけん): Health insurance
銀行口座(ぎんこうこうざ): Bank account
自己負担(じこふたん): self-pay
補助(ほじょ): Support


First, a COE (certificate of eligibility) is needed from the employer.

Once the COE is received, apply the Japan working visa in the local embassy. Note:
  • Different embassy might have different requirements of documents. e.g.
  • A legal immigration status in US is typically needed .
  • The arrival time/date can be flexible.
  • The working visa is for 1-time entry. Later will use 在留カード to enter Japan.
  • The visa requires 3 working day to obtain.


The shipper might need to know the following information.
  • Need to check with the employer about the detailed upper limits (e.g. how big for the oversea shipment and how heavy for the air cargo).
  • Need to know whether I should pay first or directly bill to the employer.
The shipper will provide two customer declaration forms, which should list the number of boxes for flights and ships. On the airport.
  • Hand in the 2 completed sheet to the custom officer to get the stamp.
  • One will be returned with the stamp.
  • This returned one should be brought to the shipper's counter at the arrival lobby of the airport.


While the employer should reimburse this, the rules for the source and destination are more restricted. For instance, originally I want to fly to Taiwan first and enter Japan one week later, which is actually way cheaper than directly flying to Japan, but my Japanese colleagues worry that the accounting department might reject because of the 1-week gap. Thus at the end I decide to enter Japan first, settle down everything, and then go to Taiwan.


在留カード is crucial for foreigners to stay in Japan. Without 在留カード, it might be not possible to create bank account and apply for phones.
在留カード can be  obtain in some international airports. Once the address in Japan is confirmed, go to the local government to register (転入届)and add the address to the back of 在留カード.


In my case, my employer rents an apartment for me so the entire procedure is simpler.

On the other hand, one thing I learned during the process is that the employee should provide a concrete plan, including the date, rent range, and size range. The date should be a single choice for the HR to search. At the beginning I provided two possible dates to the HR. According to my Japanese colleague, it is better that I can reduce the date to a single & confirmed one so it is easier for HR to search.


  • 健康保険/Health insurance: Can purchase from the local government, but employers might have their own plan.
  • 銀行口座/Bank account: Most banks would have rules for foreigner. Meanwhile, employers might have their preferred bank to deposit the salary.
  • 携帯: TO ADD

Let to pause with pydev

posted Aug 23, 2019, 7:29 PM by Teng-Yok Lee

After, add the following to pause the script:
        # This is to pause the script
        # when debugging with PyDev.
        # It should be put after
        # so it will be only executed in debug mode.
        import pydevd;
        dummy_fig = plt.figure();

My procedure to enable disk quota

posted Aug 4, 2019, 9:40 AM by Teng-Yok Lee   [ updated Sep 23, 2019, 10:02 PM ]

Mount disk with quota options

The options are usrquota and grpquota.

# In /etc/fstab:

/dev/sdb1       /data ext4      defaults,usrjquota=aquota.user,,jqfmt=vfsv1      0       0

Remount the filesystem to make it effective:
$ sudo mount -vo remount /data

Check the quota

NOTE: sudo is needed in order to temporally make the file system read-only. The option -cug will check both users & groups (REF).
$ sudo quotacheck -cug /data

Enable quota

$ sudo quotaon /data

Check quota

To check user-wise quota:
$ sudo repquota /data
To check group-wise quota, add the option -g:
$ sudo repquota -g /data

Edit the quota

Option 1: Use setquota for command-line based editing. The usage is:

$ setquota <user> <soft-inode-limit> <hard-inode-limit> <soft-file-limit> <hard-file-limit> <file-system>

Option 2: Use edquota for interactive editing

Set the preferred editor (REF):
$ export EDITOR="emacs -nw"
Edit the quota for a specific user. Note the the env. var. should be passed via -E.
$ sudo -E edquota tlee

Disable quota check when booting

Editing /etc/default/grub:

$ sudo emacs -nw /etc/default/grub

Append "quotacheck.mode=skip" to GRUB_CMDLINE_LINUX.

Update grub:

$ sudo update-grub

Once reboot, check /proc/cmdline to verify the kernel options.


Some slurm tricks

posted Jul 19, 2019, 7:31 AM by Teng-Yok Lee   [ updated Jul 19, 2019, 7:31 AM ]

How to get the stdout of a slurm job?

function slurm_get_job_stdout()
    scontrol show jobid ${jobid} | grep StdOut | cut -b 11-

How to list the CPU/GPU resources of all nodes?

function slurm_list_resources()

    # This is a workaround for scontrol to query hidden nodes.
    for a in ${extra_args}
        if [ "-a" == "$a" ]

    for h in $(sinfo ${extra_args} -N -h | cut -d " " -f 1)
        n_used_gpus=$(squeue ${extra_args} -t R -h -o "%8b" -w $h | grep gpu | cut -d : -f 2 | paste -s -d + - | bc)
        scontrol_output=$(scontrol ${w_hidden} show node $h | tr ' ' '\n')
        n_gpus=$(printf "%s" "${scontrol_output}" | grep Gres | cut -d : -f 2)
        n_used_cpus=$(printf "%s" "${scontrol_output}" | grep CPUAlloc | cut -d = -f 2)
        n_cpus=$(printf "%s" "${scontrol_output}" | grep CPUTot | cut -d = -f 2)
        cpu_load=$(printf "%s" "${scontrol_output}" | grep CPULoad | cut -d = -f 2)
        printf "[%s]\tGPU: %d/%d\tCPU: %3d/%3d\tLoad: %2.2f%%\n" "$h" "${n_used_gpus}" "${n_gpus}" "${n_used_cpus}" "${n_cpus}" ${cpu_load}

Notes for nvidia-smi

posted Jul 12, 2019, 10:42 AM by Teng-Yok Lee

Check P2P status

# REF:

$ nvidia-smi topo -p2p w

Use a 1-liner of gnuplot to plot loss in Caffe's log

posted Jul 10, 2019, 5:25 AM by Teng-Yok Lee

This can be very helpful when analyzing logs so need to write extra script.

$ gnuplot -p -e "plot '<cat' with lines"
It will plot the values from stdin. Typically I used it when analyzing the logs from Caffe. For instance, In the example  below, I extract loss #0 from the training network from file caffe.bin.INFO, use cut to filter the value, and then pass it to gnuplot.

$ grep "Train net output #0" caffe.bin.INFO | cut -d = -f 2 | cut -d " " -f 2 | gnuplot -p -e "plot '<cat' with lines"

1-10 of 123