Memo

I often spend lots of time on setting up the same software over and over again for different purposes. In order to save time, I decide to write down some useful procedures or hints here. Hope that they could be helpful to myself and others who visit here.

Let matplotlib.pyplot.show() to pause with pydev

posted Aug 23, 2019, 7:29 PM by Teng-Yok Lee

After matplotlib.pyplot.show(), add the following to pause the script:
 
    try:
        # This is to pause the script
        # when debugging with PyDev.
        # It should be put after plt.show()
        # so it will be only executed in debug mode.
        import pydevd;
        dummy_fig = plt.figure();
        dummy_fig.canvas.start_event_loop_default();
    except:
        pass;

My procedure to enable disk quota

posted Aug 4, 2019, 9:40 AM by Teng-Yok Lee   [ updated Aug 4, 2019, 9:56 AM ]

REF: https://web.mit.edu/rhel-doc/5/RHEL-5-manual/Deployment_Guide-en-US/ch-disk-quotas.html

Mount disk with quota options

The options are usrquota and grpquota. There are two ways to mount:

Option 1: Use /etc/fstab:

# In /etc/fstab:

/dev/sdb1       /data ext4      defaults,usrquota,grpquota      0       0


Reboot the computer to make it effective.

Option 2: Directly remount with the option:

$ sudo mount -o usrquota,grpquota /dev/sdb1 /data

Check the quota

NOTE: sudo is needed in order to temporally make the file system read-only. The option -cug will check both users & groups (REF).
$ sudo quotacheck -cug /data

Enable quota

$ sudo quotaon /data

Check quota

To check user-wise quota:
$ sudo repquota /data
To check group-wise quota, add the option -g:
$ sudo repquota -g /data

Edit the quota

Set the preferred editor (REF):
$ export EDITOR="emacs -nw"
Edit the quota for a specific user. Note the the env. var. should be passed via -E.
$ sudo -E edquota tlee

Some slurm tricks

posted Jul 19, 2019, 7:31 AM by Teng-Yok Lee   [ updated Jul 19, 2019, 7:31 AM ]

How to get the stdout of a slurm job?

function slurm_get_job_stdout()
{
    jobid=$1
    scontrol show jobid ${jobid} | grep StdOut | cut -b 11-
}

How to list the CPU/GPU resources of all nodes?

function slurm_list_resources()
{
    extra_args=$@

    # This is a workaround for scontrol to query hidden nodes.
    w_hidden=""
    for a in ${extra_args}
    do
        if [ "-a" == "$a" ]
        then
            w_hidden=$a
        fi
    done

    for h in $(sinfo ${extra_args} -N -h | cut -d " " -f 1)
    do
        n_used_gpus=$(squeue ${extra_args} -t R -h -o "%8b" -w $h | grep gpu | cut -d : -f 2 | paste -s -d + - | bc)
        scontrol_output=$(scontrol ${w_hidden} show node $h | tr ' ' '\n')
        n_gpus=$(printf "%s" "${scontrol_output}" | grep Gres | cut -d : -f 2)
        n_used_cpus=$(printf "%s" "${scontrol_output}" | grep CPUAlloc | cut -d = -f 2)
        n_cpus=$(printf "%s" "${scontrol_output}" | grep CPUTot | cut -d = -f 2)
        cpu_load=$(printf "%s" "${scontrol_output}" | grep CPULoad | cut -d = -f 2)
        printf "[%s]\tGPU: %d/%d\tCPU: %3d/%3d\tLoad: %2.2f%%\n" "$h" "${n_used_gpus}" "${n_gpus}" "${n_used_cpus}" "${n_cpus}" ${cpu_load}
    done
}


Notes for nvidia-smi

posted Jul 12, 2019, 10:42 AM by Teng-Yok Lee

Check P2P status

# REF: https://devtalk.nvidia.com/default/topic/1031758/p2p-not-working-for-p600s-/

$ nvidia-smi topo -p2p w




Use a 1-liner of gnuplot to plot loss in Caffe's log

posted Jul 10, 2019, 5:25 AM by Teng-Yok Lee

This can be very helpful when analyzing logs so need to write extra script.

$ gnuplot -p -e "plot '<cat' with lines"
It will plot the values from stdin. Typically I used it when analyzing the logs from Caffe. For instance, In the example  below, I extract loss #0 from the training network from file caffe.bin.INFO, use cut to filter the value, and then pass it to gnuplot.

$ grep "Train net output #0" caffe.bin.INFO | cut -d = -f 2 | cut -d " " -f 2 | gnuplot -p -e "plot '<cat' with lines"

Comparison of X11 servers on Windows

posted May 11, 2019, 6:52 AM by Teng-Yok Lee

NOTE: The column Easy so use is based on the amount of configurations needed after installation. More * means more effort. So far it seems that VcXsrv is the best one in terms of all main features I need.

Software Installation Price/ license Clipboard Resizing of windows? OpenGL Easy to use Notes
VcXsrv A single installer. FREE GPL3 Yes Yes Yes w/o native opengl. **
  • Might be blocked by old Micro Trend.
  • Should disable native opengl.
MobaXTerm A single installer. FREE Failed? Yes Yes w/o X extensions. ***
  • For heavy PowerPoint users, it would be better to remove the hotkey Ctrl+M. Ctrl+M is also used to insert new slides in PowerPoint, but by default MobaXTerm grab this hotkey.
  • Better to disable X extensions for OpenGL.
  • See more limits of the free version HERE


$69/yr per user ? ? ? ?
Xming A single installer. FREE Yes No Yes *
  • Or try older versions (REF)?


$20/yr ? ? ? ?
cygwin/X Multiple cygwin packages

FREE

X11

Yes Yes Yes with limited & incorrect color depth *****
  • See installation guide HERE.
  • More complicated to configure (e.g. for putty, need to manually specify the path t .Xauthority for each configuration).
  • Might be blocked by old Micro Trend.
  • If the X server restarts, current SSH session will fail and need to re-connect.

OpenGL with X11 servers on Windows

posted May 11, 2019, 6:47 AM by Teng-Yok Lee   [ updated May 11, 2019, 6:53 AM ]

I typically use Windows with ssh -X to access Linux machines so I can avoid dual boot. If I need to run remote applications with X windows, I run VcXsrv (REF) on windows as the X11 server. This typically works except programs that needs OpenGL. If software-based rendering is affordable, my workaround is as follows:

  • On the local windows: Run VcXsrc without native OpenGL supports.
  • ssh to the the remote Linux:
    • Make sure that libgl1-mesa-dev is installed.
      • NOTE: If the nvidia driver is install via .run, it might override libgl1-mesa-dev.
        • To verify, run the following to see whether it is using libGL under mesa or not:
          $ ldd $(which glxinfo) | grep libGL
        • If not, the symlink of /usr/lib/x86_64-linux-gnu/mesa might be broken. In such a case, re-installation of libgl1-mesa-devis needed.
    • Setup env. var. to use mesa's libGL.so:
      $ export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/mesa:$LD_LIBRARY_PATH
    • To confirm, run glxinfo again to see whether direct rendering is Yes:
      $ glxinfo | grep -i direct

New Google Sites

posted Apr 18, 2019, 9:33 PM by Teng-Yok Lee

I did a quick test with the new Google Sites, but I decide to stay with the classic one because:

1. No complete history to rollback. This is the biggest reason.
2. Lose the banner in the left side (I used this to show weather of my current city).
3. The format of my publication list is destroyed.
4. Unsure whether I can post announcement, like what I am doing now.

A related question is: How to re-do the conversion? I created a draft but I disliked it.

The solution is in the page Manage Site/Convert to new site. After entering the stage Review and Publish, open the advanced pane, and choose to disconnect from the current page. Then it will restart from stage 1.

Save the subplot of matplotlib

posted Apr 18, 2019, 7:46 PM by Teng-Yok Lee   [ updated Apr 18, 2019, 7:47 PM ]

REF: https://stackoverflow.com/questions/4325733/save-a-subplot-in-matplotlib

The solution is calling ax.get_tightbox() to get the entire region and converting per the figure's DPI. Based on the answer above, I wrote a short function to combine multiple axes:

def SaveAxes(fig, axes, fig_filepath):
    extents = [];
    for ax in axes:
        extents.append(ax.get_tightbbox(fig.canvas.get_renderer()).transformed(fig.dpi_scale_trans.inverted()))
    extent = Bbox.union(extents);
    plt.savefig(fig_filepath, bbox_inches = extent);

How to check a python module path before importing?

posted Jun 24, 2018, 7:01 PM by Teng-Yok Lee

REF: https://stackoverflow.com/questions/247770/retrieving-python-module-path

Use imp. For instance,
import cv2 imp.find_module("cv2")

1-10 of 118