Memo

I often spend lots of time on setting up the same software over and over again for different purposes. In order to save time, I decide to write down some useful procedures or hints here. Hope that they could be helpful to myself and others who visit here.

Use libSDF to read SDF format on Windows

posted Mar 20, 2015, 9:46 AM by Teng-Yok Lee   [ updated Mar 20, 2015, 9:47 AM ]

This is my memo to read the data for IEEE SciVis 2015 Contest. The file is in SDF format. There is a C/C++ library libSDF to open the files, but some instructions are unclear, especially for the porting on Windows. Also, I cannot find examples about its usage so I write one:

Prerequisites

To build libSDF for Visual Studio 2010, I use cygwin and mingw x64. The procedure to install them can be seen here:

http://www.recheliu.org/memo/suggestionstoavoidarpackcompilationerrors

Build libSDF for Windows x64 platform

  • Edit SDFfuncs.c: Remove the preprocessor USE_ALLOCA.
  • Edit utils.c: Change the function MPMU_Fopen() to open the file as binary (otherwise not all bytes can be read):
MPMYFile *MPMY_Fopen(const char *path, int mpmy_flags)
{
    MPMYFile *fp;
    int iomode = MPMY_SINGL;
    char mode[8] = {[0] = 'r'};

    Msgf("Fopen %s\n", path);
    if (mpmy_flags & MPMY_RDONLY) mode[0] = 'r'; /* MPMY_RDONLY is 0 since O_RDONLY is (stupidly) 0 */
    if (mpmy_flags & MPMY_WRONLY) mode[0] = 'w';
    if (mpmy_flags & MPMY_APPEND) mode[0] = 'a';
    // TEST-ADD-BEGIN
    switch(mode[0]) {
    case 'r':
        strcpy(mode, "r+b");
        break;
    case 'w':
        strcpy(mode, "w+b");
        break;
    case 'a':
        strcpy(mode, "a+b");
        break;
    }
    // TEST-ADD-END
    ...

  • Edit Makefile: Change CC from gcc to the one for mingw.
CC=/usr/bin/x86_64-w64-mingw32-gcc.exe
  • Use cygwin to build the library.
$ make libSDF.a
$ /usr/bin/x86_64-w64-mingw32-dllwrap.exe --export-all-symbols *.o -lm --output-def libSDF_x64.def -o libSDF_x64.dll

  • Use Visual Studio (64-bit) Command Prompt to build the .dll

lib.exe /machine:X64 /def:libSDF_x64.def

My quick example to read the array "x"

    #define LOG_VAR(x) cout<<x<<endl;

    char* szSdfFilepath = "F:/data/viscontest/scivis2015/ds14_scivis_0128_e4_dt04_0.0200";
    LOG_VAR(SDFissdf(szSdfFilepath));

    SDF *sdf = SDFopen(szSdfFilepath, "");
    SDFdebug(1); // Put it 0 to disable debug information.

    int64_t uNrOfRecs = SDFnrecs("x", sdf);
    LOG_VAR(uNrOfRecs);
    vector<float> vfData(uNrOfRecs);
    SDFrdvecs(sdf, "x", uNrOfRecs, vfData.data(), 0, NULL);
    SDFclose(sdf);

    for(size_t i = 0; i < uNrOfRecs; i++)
    {
        if( 0.0f != vfData[i] )
        {
            LOG_VAR(vfData[i]);
        }
    }


Suggestions to avoid ARPACK++ compilation errors

posted Jan 21, 2015, 7:24 AM by Teng-Yok Lee   [ updated Jan 21, 2015, 6:42 PM ]

I plan to put my patches to ARPACK++(http://www.ime.unicamp.br/~chico/arpack++/) to my Google Code repo at the end of Jan. 2015. Here I manually list the fixes for my applications. Note that my applications only use ardsmat.h and ardssym.h so there could be more errors, but I guess that the fixes for other parts should be similar.

arch.h

Comment arcomp.h (Otherwise, arcomple<float>/arcomple<double> will be declared inside a extern "C" closure, which is not allowed since C does not understand C++ template).

// #include "arcomp.h"

Also, replace the generic.h

#include <generic.h>

by the only needed macro name2:

// REF: http://www-d0.fnal.gov/KAI/doc/migrate/gnu_generic.h
#define name2(a,b) gEnErIc2(a,b)
#define gEnErIc2(a,b) a ## b

ardssym.h

Replace
DefineParameters(A.ncols(), nevp, &A, ARdsSymMatrix<FLOAT>::MultMv,
by
DefineParameters(A.ncols(), nevp, &A, &ARdsSymMatrix<FLOAT>::MultMv,

arerror.h

Replace
#include <iostream.h>

by
#include <iostream>

arpackf.h

Change the sentences at the end (Because arcomp.h is not included, '}' will be ignored and thus the entern closure is not completed).

}
#endif // ARCOMP_H

to
#endif // ARCOMP_H
} // extern "C" {


Build ARPACK on Windows for Visual Studio

posted Jan 19, 2015, 11:33 PM by Teng-Yok Lee   [ updated Jan 21, 2015, 7:27 AM ]

This is a revision of Jernej Barbic's steps to build ARPACK for Visual Studio. The original steps are in the URL below:

http://www-bcf.usc.edu/~jbarbic/arpack.html

I met issues when executing step 3 so I decide to put my modified procedure here. For 32-bit, you can try both gfortran or g77. For 64-bit, only gfortran can work.

My recommendation is to use the MinGW64 in CYGWIN64.

Use gfortran (Win32)

  1. Modify ARmake.inc in the source code root. I extracted the source code of ARPACK to D:\src\ARPACK. Also I cannot find f77 in the latest MinGW so I used gfortran instead. Thus the following variables should be changed accordingly:
    home = /d/src/ARPACK
    PLAT = win32
    FC      = gfortran
    FFLAGS    = -O
  2. Open a MSYS window and:
    cd /d/src/ARPACK.
  3. Compile the .f to .o:
    make lib
  4. Wrap the *.o to .dll. I need to add the library gfortran:
    dllwrap --export-all-symbols BLAS/*.o LAPACK/*.o SRC/*.o UTIL/*.o -lg2c -lgfortran --output-def arpack_win32.def -o arpack_win32.dll
  5. Open a Visual Studio Command Prompt and:
    cd d:\src\ARPACK
  6. Generate the library:
    lib.exe /machine:i386 /def:arpack_win32.def

Use g77 (Win32)

  1. Modify ARmake.inc in the source code root. I extracted the source code of ARPACK to D:\src\ARPACK. Also I cannot find f77 in the latest MinGW so I used gfortran instead. Thus the following variables should be changed accordingly:
    home = /d/src/ARPACK
    PLAT = win32
    FC      = g77
    FFLAGS    = -O
  2. Open a MSYS window and:
    cd /d/src/ARPACK.
  3. Compile the .f to .o:
    make lib
  4. Wrap the *.o to .dll.
    dllwrap --export-all-symbols BLAS/*.o LAPACK/*.o SRC/*.o UTIL/*.o -lg2c --output-def arpack_win32.def -o arpack_win32.dll
  5. Open a Visual Studio Command Prompt and:
    cd d:\src\ARPACK
  6. Generate the library:
    lib.exe /machine:i386 /def:arpack_win32.def

Use gfortran (Win 64) in MSYS

Before the procedure, install MinGW 64. In my case, it is installed to C:\Program Files (x86)\mingw-w64, and gfortran.exe is in C:\Program Files (x86)\mingw-w64\i686-4.9.2-posix-dwarf-rt_v3-rev1\mingw32\bin.
  1. Modify ARmake.inc in the source code root. I extracted the source code of ARPACK to D:\src\ARPACK. Also I cannot find f77 in the latest MinGW so I used gfortran instead. Thus the following variables should be changed accordingly:
    home = /d/src/ARPACK
    PLAT = x64
    FC = /c/Program\ Files\ \(x86\)/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/bin/gfortran.exe
    FFLAGS    = -O
    RANLIB = /c/Program\ Files\ \(x86\)/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/bin/ranlib.exe

  2. Open a MSYS window and:
    cd /d/src/ARPACK
  3. Extend PATH:
    export PATH=$PATH:/c/Program\ Files\ \(x86\)/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/opt/bin
  4. Change UTIL/second. Remove the sentence: EXTERNAL           ETIME.
  5. Compile the .f to .o:
    make lib
  6. Wrap the *.o to .dll:
    /c/Program\ Files\ \(x86\)/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/bin/dllwrap.exe --export-all-symbols BLAS/*.o LAPACK/*.o SRC/*.o UTIL/*.o -lgfortran --output-def arpack_x64.def -o arpack_x64.dll
  7. Open a Visual Studio (64-bit) Command Prompt and:
    cd d:\src\ARPACK
  8. Generate the library:
    lib.exe /machine:X64 /def:arpack_x64.def

Use gfortran (Win 64) in cygwin

Later I found later MinGW is also available in CYGWIN, which make the commands shorter.

Before the procedure, install mingw64-x86_64-gcc in CYGWIN 64.

  1. Modify ARmake.inc in the source code root. I extracted the source code of ARPACK to D:\src\ARPACK. Also I cannot find f77 in the latest MinGW so I used gfortran instead. Thus the following variables should be changed accordingly:
    home = /cygdrive/d/src/ARPACK
    PLAT = x64
    FC = /usr/bin/x86_64-w64-mingw32-gfortran.exe
    FFLAGS    = -O
    RANLIB = /usr/bin/x86_64-w64-mingw32-ranlib.exe

  2. Open a MSYS window and:
    cd /cygdrive/d/src/ARPACK
  3. Change UTIL/second. Remove the sentence: EXTERNAL           ETIME.
  4. Compile the .f to .o:
    make lib
  5. Wrap the *.o to .dll:
    /usr/bin/x86_64-w64-mingw32-dllwrap.exe --export-all-symbols BLAS/*.o LAPACK/*.o SRC/*.o UTIL/*.o -lm -lgfortran --output-def arpack_x64.def -o arpack_x64.dll
  6. Open a Visual Studio (64-bit) Command Prompt and:
    cd d:\src\ARPACK
  7. Generate the library:
    lib.exe /machine:X64 /def:arpack_x64.def

Link the lib

After building .lib and .dll, adding the directories of related dll. to your path. For CYGWIN, the path is

C:\cygwin64\usr\x86_64-w64-mingw32\sys-root\mingw\bin

For MinGW, it should be the path to the 64-bit.

If the .dll does not match to the linked library, it might show error The application was unable to start correctly (0xc000007b). I saw it when my system had both 32-bit and 64-bit versions of MinGW. After I uninstalled both and just kept CYGWIN 64, the problem was solved.

References

  1. External issue of etime: https://gcc.gnu.org/ml/fortran/2007-03/msg00305.html
  2. Use CYGWIN and MinGW64 to build Visual Studio libraries: https://github.com/arrayfire/arrayfire/wiki/CBLAS-for-Windows
  3. Cannot start my application: http://stackoverflow.com/questions/25124182/mingw-gcc-the-application-was-unable-to-start-correctly-0xc000007b

visual studio 2008 C++ no x64 platform

posted Jan 27, 2013, 5:21 PM by Teng-Yok Lee

http://social.msdn.microsoft.com/Forums/en-US/vssetup/thread/d75db7b8-9f00-4e8b-8be8-7f3ceda77d76

Build NetCDF4 for Visual Studio 2010 x64

posted Dec 22, 2012, 6:39 PM by Teng-Yok Lee   [ updated Feb 10, 2015, 11:25 PM ]

In order to run NetCDF for large data, I plan to build its x64 version. I tried several files from

The official Win32 releases from UCAR or via the official SVN case size_t to (signed) __int64, which can cause linking error.

The unofficial Win32 contributions are either too old or cannot be built.

The closest solution I found was release by Space Research. Their instructions are listed here, including the patched HDF5 and NetCDF 4.2. While their .sln files are for Win32 mode, it is easy to upgrade them to x64 version. The following instruction is my procedure. Note that my procedure make changes to the solution so szip is not required.

  1. Download the patched NetCDF and HDF5.
  2. Build HDF5
    1. Use VS2010 to open the .sln.
    2. Change the configuration from Win32 to x64
    3. Edit H5pubconf.h: Comment H5_HAVE_SZLIB_H, H5_HAVE_FILTER_SZIP, and H5_HAVE_TIMEZONE.
    4. Remove h5szip.c from the project
    5. Build hdf5 and hdf5_hl.
    6. Collect the .h to a folder (e.g. <hdf5_installed_dir>/include) and the .lib to a folder (e.g. <hdf5_installed_dir>/lib).
  3. Build NetCDF
    1. Use VS2010 to open the .sln.
    2. Change the configuration from Win32 to x64
    3. Build libsrc, libsrc4, libdispatch.
    4. I also built ncdump to test whether the libraries work.
    5. Collect the .h to a folder (e.g. <netcdf_installed_dir>/include), the .lib to a folder (e.g. <netcdf_installed_dir>/lib), and ncdump.exe to a folder (e.g. <netcdf_installed_dir>/bin).
  4. Use the files
    1. Include the files in
      • <hdf5_installed_dir>/include
      • <netcdf_installed_dir>/include
    2. Link the files in
      • <hdf5_installed_dir>/lib 
      • <netcdf_installed_dir>/lib
    3. Link the following libraries:
      • (Debug) <hdf5_installed_dir>/lib/hdf5d.lib
      • (Debug) <hdf5_installed_dir>/lib/hdf5_hld.lib
      • (Release) <hdf5_installed_dir>/lib/hdf5.lib
      • (Release) <hdf5_installed_dir>/lib/hdf5_hl.lib
      • <netcdf_installed_dir>/lib/libsrc.lib
      • <netcdf_installed_dir>/lib/libsrc4.lib
      • <netcdf_installed_dir>/lib/libdispatch.lib
PS. If you want to keep szip, you can download the official source code here released by the HDF group. This archive contains a CMakeLists.txt and thus building it should be straightforward.

Building HDF5/NetCDF4 with CMake?


PS. Now HDF5 can be built with CMake, and so does NetCDF 4.3.2 (See the two links below for more detail). So far I can use CMake to build HDF5, but I met issue with NetCDF (It cannot find zlib and there is no CMake entry for ZLIB_DIR).

http://www.hdfgroup.org/HDF5/release/cmakebuild.html
http://www.unidata.ucar.edu/software/netcdf/docs/netCDF-CMake.html


Build NetCDF with the support of NetCDF4 format

posted Nov 9, 2012, 4:47 AM by Teng-Yok Lee   [ updated Jan 31, 2013, 5:03 AM ]

As the configure of netcdf has no option to specify the path of HDF5, this is the command line suggested by the doc INSTALL in netcdf4:

CPPFLAGS=-I<include_path_of_hdf5> LDFLAGS=-L<lib_path_of_hdf5> ./configure

Note:
If HDF5 is built as static linking library, the configure of NetCDF cannot find the library of HDF5. This should be not an issue in most cases. Nevertheless, supercomputers (e.g  SURVEYOR in ANL and HOPPER in NERSC) do not support shared lib, and I am not sure how to build a static version of NetCDF yet.


For Windows, the built installers can be downloaded from here: http://www.unidata.ucar.edu/software/netcdf/win_netcdf/. (For Visual Studio 2010 x64 version, please follow my instructions here: https://sites.google.com/site/recheliunew/memo/buildnetcdf4forvisualstudio2010x64)


1-6 of 6