Welcome to LZN's Blog!

Wind extinguishes a candle but energizes fire.

山东半岛预报系统搭建

2017-12-31

LZN

project

project wrf-sdpwfe

编译WRF
编译WPS
下载geog包并解压。
调整wps设置
执行wps
调整WRF设置
执行real
执行wrf
计划任务

打算做一个山东半岛的预报系统，以后也算投名状。

编译WRF

configure尝试选择20. INTEL (ifort/icc): Xeon (SNB with AVX mods)

之后直接执行编译

./compile -j 8 em_real

执行编译过程卡住。更换15. 不包括avx指令集后成功。

编译WPS

编译wps发现无法编译出ungrib，经过多次测试，分别

yum install libpng-devel
yum install jasper-devel

将devel包安装完整后成功。

下载geog包并解压。

Updated 2017-12-31

调整wps设置

完整namelist.wps

&share
 wrf_core = 'ARW',
 max_dom = 4,
 start_date = '2018-01-05_12:00:00','2018-01-05_12:00:00','2018-01-05_12:00:00','2018-01-05_12:00:00',
 end_date   = '2018-01-10_12:00:00','2018-01-10_12:00:00','2018-01-10_12:00:00','2018-01-10_12:00:00',
 interval_seconds = 21600
 active_grid = .true., .true.,.true.,.true.,
 subgrid_ratio_x = 1
 subgrid_ratio_y = 1
 io_form_geogrid = 2,
 opt_output_from_geogrid_path = './',
 debug_level = 0
/

&geogrid
 parent_id         =   1,   1,   2,   3,
 parent_grid_ratio =   1,   3,   3,   3,
 i_parent_start    =   1, 140,  80, 140,
 j_parent_start    =   1,  80,  50, 110,
 s_we              =   1,   1,   1,   1,
 e_we              = 300, 241, 301, 301,
 s_sn              =   1,   1,   1,   1,
 e_sn              = 220, 202, 283, 301,
 !
 !!!!!!!!!!!!!!!!!!!!!!!!!!!! IMPORTANT NOTE !!!!!!!!!!!!!!!!!!!!!!!!!!!!
 ! The default datasets used to produce the HGT_M, GREENFRAC,
 ! and LU_INDEX/LANDUSEF fields have changed in WPS v3.8. The HGT_M field
 ! is now interpolated from 30-arc-second USGS GMTED2010, the GREENFRAC
 ! field is interpolated from MODIS FPAR, and the LU_INDEX/LANDUSEF fields
 ! are interpolated from 21-class MODIS.
 !
 ! To match the output given by the default namelist.wps.all_options 
 ! in WPS v3.7.1, the following setting for geog_data_res may be used:
 !
 ! geog_data_res = 'gtopo_10m+usgs_10m+nesdis_greenfrac+10m','gtopo_2m+usgs_2m+nesdis_greenfrac+2m',
 !
 !!!!!!!!!!!!!!!!!!!!!!!!!!!! IMPORTANT NOTE !!!!!!!!!!!!!!!!!!!!!!!!!!!!
 !
 geog_data_res = 'default','default','default','default',
 dx        = 27000,
 dy        = 27000,
 map_proj  = 'lambert',
 ref_lat   =  37.00,
 ref_lon   =  110.00,
 ref_x     =  55.0,
 ref_y     =  35.0,
 truelat1  =  30.0,
 truelat2  =  60.0,
 stand_lon =  110.0,
 geog_data_path = '/home/lzhenn/wrf-surface-data/geog'
 opt_geogrid_tbl_path = 'geogrid/'
/
 geog_data_res     = 'modis_lakes+10m','modis_lakes+2m',
 geog_data_res     = 'usgs_lakes+10m','usgs_lakes+2m',

&ungrib
 out_format = 'WPS',
 prefix     = 'GFS',
/

&metgrid
 fg_name         = 'GFS', 'SST'
 constants_name  = './TAVGSFC'
 io_form_metgrid = 2, 
 opt_output_from_metgrid_path = './',
 opt_metgrid_tbl_path         = 'metgrid/',
 process_only_bdy = 5,
/

&mod_levs
 press_pa = 201300 , 200100 , 100000 , 
             95000 ,  90000 , 
             85000 ,  80000 , 
             75000 ,  70000 , 
             65000 ,  60000 , 
             55000 ,  50000 , 
             45000 ,  40000 , 
             35000 ,  30000 , 
             25000 ,  20000 , 
             15000 ,  10000 , 
              5000 ,   1000
/

&plotfmt
 ix = 100
 jx = 100
 ioff = 30
 joff = 30
/

domain设置如下：

执行wps

设置从HKUST ENVR下载GFS输入文件GFS data archive

测试加入gfs的SST数据，参考

http://www2.mmm.ucar.edu/wrf/OnLineTutorial/DATA/SST/index.html

后发现并不能用，出现segfault，之后更改namelist和，对应GFS数据为SST变量Vtable，以及SST前缀，sst问题解决。

met data生成过程一切顺利。

Updated 2018-01-06

调整WRF设置

根据俊文和海洋局的配置调整namelist.input设置，过程中发现V3.9提供了经过测试的参数化suite，果断采用。CONUS代表北美大陆配置。

执行real

调整后real过程出错

 5   --- ERROR: If sst_update /= 0, one of the auxinput4_interval settings must be /= 0
 6   --- Set auxinput4_interval_s to the same value as interval_seconds (usually a pretty good guess).

修改后，执行过程出现段错误。

测试修改环境变量。

# set WRF
export WRFIO_NCD_LARGE_FILE_SUPPORT=1
export WRF_EM_CORE=1
export NETCDF4=0

ulimit -s unlimited
ulimit -c unlimited

通过第一断点。新错误：

-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:     431 
 Problems, we cannot have excluded middle data from WPS 
 -------------------------------------------
 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
 [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
 :
 system msg for write_line failure : Bad file descriptor

将SST Update设置为0后可以顺利执行完real。

执行wrf

执行过程直接出错,有一个gwd_opt设置为1后可行。

顺利执行，但是积分速度奇慢无比，决定采用180s步长，三层嵌套，速度快了很多。最终采用的wrf namelist.input

 &time_control
 run_days                            = 5,
 run_hours                           = 0,
 run_minutes                         = 0,
 run_seconds                         = 0,
 start_year                          = 2018, 2018, 2018, 2018,
 start_month                         = 01,   01,   01,   01,
 start_day                           = 05,   05,   05,   05,
 start_hour                          = 12,   12,   12,   12,
 start_minute                        = 00,   00,   00,   00,
 start_second                        = 00,   00,   00,   00,
 end_year                            = 2018, 2018, 2018, 2018,
 end_month                           = 01,   01,   01,   01,
 end_day                             = 10,   10,   10,   10,
 end_hour                            = 12,   12,   12,   12,
 end_minute                          = 00,   00,   00,   00,
 end_second                          = 00,   00,   00,   00,
 interval_seconds                    = 21600
 input_from_file                     = .true.,.true.,.true.,.true.,
 history_interval                    = 60,  60,   60,   60,
 frames_per_outfile                  = 1000, 1000, 1000, 1000,
 restart                             = .false.,
 restart_interval                    = 5000,
 io_form_history                     = 2
 io_form_restart                     = 2
 io_form_input                       = 2
 io_form_auxinput4                   = 0 
 io_form_boundary                    = 2
 auxinput4_interval                  = 0
 debug_level                         = 0
 /

 &domains
 time_step                           = 180,
 time_step_fract_num                 = 0,
 time_step_fract_den                 = 1,
 max_dom                             = 3,
 e_we                                = 300, 241, 301, 301,
 e_sn                                = 220, 202, 283, 301,
 e_vert                              = 39,   39,  39,  39, 
 p_top_requested                     = 5000,
 num_metgrid_levels                  = 32,
 num_metgrid_soil_levels             = 4,
 eta_levels			     = 1.0000, 0.9979, 0.9956, 0.9931,
				       0.9904, 0.9875, 0.9844, 0.9807,
				       0.9763, 0.9711, 0.9649, 0.9575,
				       0.9488, 0.9385, 0.9263, 0.9120,
				       0.8951, 0.8753, 0.8521, 0.8251,
				       0.7937, 0.7597, 0.7229, 0.6833,
				       0.6410, 0.5960, 0.5484, 0.4985,
				       0.4467, 0.3934, 0.3393, 0.2850,
				       0.2316, 0.1801, 0.1324, 0.0903,
                                       0.0542, 0.0241, 0.0000
 dx                                  = 27000,   9000,   3000,	1000,
 dy                                  = 27000, 	9000,   3000,	1000,
 
 grid_id                             = 1,     2,     3,	   4,
 parent_id                           = 1,     1,     2,	   3,
 i_parent_start                      = 1,   140,    80,  140,
 j_parent_start                      = 1,    80,    50,  110,
 parent_grid_ratio                   = 1,     3,     3,    3,
 parent_time_step_ratio              = 1,     3,     3,    3,
 feedback                            = 0,
 smooth_option                       = 2
 /

 &physics
 physics_suite                       = 'CONUS'
 
 cu_physics                          = 6,     6,     0,    0,

 radt                                = 30,    30,    30,  30,
 bldt                                = 0,     0,     0,    0,
 cudt                                = 5,     5,     5,    5,
 icloud                              = 1,
 num_soil_layers                     = 4,
 num_land_cat                        = 21,
 sf_urban_physics                    = 0,     0,     0,    0,
 sst_update	            		     = 0,
 /

 &fdda
 /

 &dynamics
 w_damping                           = 0,
 diff_opt                            = 1,      1,      1,    1,
 km_opt                              = 4,      4,      4,    4,
 diff_6th_opt                        = 0,      0,      0,    0,
 diff_6th_factor                     = 0.12,   0.12,   0.12, 0.12,
 base_temp                           = 290.                          
 damp_opt                            = 0,                            
 zdamp                               = 5000.,  5000.,  5000.,5000.,
 dampcoef                            = 0.2,    0.2,    0.2   0.2
 khdif                               = 0,      0,      0,    0,
 kvdif                               = 0,      0,      0,    0,
 non_hydrostatic                     = .true., .true., .true..true., ,
 moist_adv_opt                       = 1,      1,      1,    1,       
 scalar_adv_opt                      = 1,      1,      1,    1,       
 gwd_opt                             = 0,
 /

 &bdy_control
 spec_bdy_width                      = 5,
 spec_zone                           = 1,
 relax_zone                          = 4,
 specified                           = .true., .false.,.false.,.false.,
 nested                              = .false., .true., .true., .true.,
 /

 &grib2
 /

 &namelist_quilt
 nio_tasks_per_group = 0,
 nio_groups = 1,
 /

计划任务

之后crontab配合自动运行脚本，每晚0:45开始执行，利用12Z数据，积分步长120s，尝试做未来5天预报。自动运行脚本： ``` bash #!/bin/sh

LID=date -d '1 days ago' +%Y%m%d LID_NLS=date -d '1 days ago' +%Y-%m-%d LID_NLE=date -d '-4 days ago' +%Y-%m-%d # 5 days later, for forecast

WPSDIR=/home/lzhenn/package/WPS WRFDIR=/home/lzhenn/array/lzhenn/WRFV3/run LOGFILE=/home/lzhenn/workspace/wrf-sdpwfe/sys-log/${LID}.log GFSDIR=/home/lzhenn/array/lzhenn/gfs_fcst/$LID

Read All

Ubuntu/K4200机器学习系统搭建

2017-12-30

LZN

technology

ubuntu machine_learning cuda deep_learning python tensorflow
记得装desktop版本的ubuntu，否则没有xserver，比较蛋疼

安装英伟达显卡驱动，要关掉x server……

默认进入命令行启动

http://blog.topspeedsnail.com/archives/4922

注意16.04用这句：

$sudo systemctl set-default multi-user.target

这是每次开机默认进入命令行模式的方法。单词关闭用

sudo /etc/init.d/lightdm stop

然后遇到了坑爹的linux预装驱动……需要关掉。（谁说ubuntu少折腾来着……）

驱动安装过程这个链接内容比较详细（卸载系统驱动后机器无法完全启动，但是ssh没问题）

http://blog.csdn.net/u012759136/article/details/53355781

照着来没问题了吧，错了……build driver的时候直接说unable……我去

差了下说我下的版本不对，赶紧去重新找（为啥查到的帖子都是那个340.32版本）

http://www.nvidia.com/content/DriverDownload-March2009/confirmation.php?url=/XFree86/Linux-x86_64/384.98/NVIDIA-Linux-x86_64-384.98.run&lang=us&type=GeForce

这里下载17.11的版本

安装cuda toolkit，这次吸取教训，直接英伟达官网下载最新cuda9.1版本。

直接运行安装，居然自带了驱动……去死吧……

后面说我有几个库没装好，记录如下

Missing recommended library: libGLU.so

Missing recommended library: libX11.so

Missing recommended library: libXi.so

Missing recommended library: libXmu.so

搜索下直接apt-get搞定

$ sudo apt-getinstall freeglut3-dev build-essential libx11-dev libxmu-dev libxi-dev libgl1-mesa-glx libglu1-mesa libglu1-mesa-dev

官方参考

https://devtalk.nvidia.com/default/topic/827410/cuda-setup-and-installation/new-installation-of-cuda-7-toolkit-in-ubuntu-12-04-question-about-missing-recommended-libraries-/

重装一遍cuda测试，一并安装了连同cuda的显卡驱动。没有问题。

http://blog.csdn.net/shiorioxy/article/details/52652831

http://blog.csdn.net/jhszh418762259/article/details/52958287

之后写几个配置

环境变量设置

sudo vim ~/.bashrc

在末尾添加

export CUDA_HOME=/usr/local/cuda-8.0

export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:$LD_LIBRARY_PATH

export PATH=/usr/local/cuda-8.0/bin:$PATH

保存退出。

然后刷新。

source ~/.bashrc

动态链接库设置

创建文件：

sudo vim /etc/ld.so.conf.d/cuda.conf

写入：/usr/local/cuda/lib64

保存之后使其立即生效：

sudo ldconfig -v

之后是测试sample，网上教程都是进utility编译，表示直接在根目录下编译了，很长时间……但是似乎没有报错退出。

许久之后，安全完成。
Read All
github技术博客搭建

2017-12-30

LZN

technology

github blog jekyll git web
还是要搬家啦～

经历了东校实验室RAID1阵列双盘全坏，我的wordpress站点从2014到前几个月数百篇文章香消玉殒，遑论全校关端口更是让在外访问变得不可能。几个重要的系统搭建流程，目前看来只能靠并不靠谱的直觉和印象了。痛定思痛，为下一处站点做好相对周全的考虑，避免再次出现这类问题。

需求端：
1. 类wordpress，框架成熟，索引方便、样式美观，不需要个人太多调整
2. 足够灵活，有单独页面，比如需要有scp/ftp push实时更新、daily refresh的预报图，以及支持脚本控制
3. 安全，两个方面的，一是数据安全……我就没见过RAID一次坏俩的情况！！！二是防范政策风险，这也是为啥放弃在EC2上开443搭wordpress的原因，目前……22口都已经不稳定了，诶
考虑供给端：
1. EC2+wordpress，足够灵活，10G也够用，但是政策风险太大，目前22访问都不稳定了，放弃
2. 简书，不灵活，放弃
3. csdn，已经被边缘化了，放弃
4. github，嘿嘿，就是你了
思想准备结束，开始搭建 follow这个来
```
$ git checkout --orphan gh-pages
```
语句出错，果然又是git版本问题……坑爹centos

突然发现很多资料都是13-14年的，这段时间应该官方更新了不少，索性还是查官方库吧。

https://pages.github.com/

相当有效。

然后costumize

https://help.github.com/articles/using-jekyll-as-a-static-site-generator-with-github-pages/

模板找到了几个，Jekll安装似乎有点麻烦。
1. 安装ruby，rubygems，安装gems过程中出错，运行中有这么一个提示
YAML safe loading is not available. Please upgrade psych to a version that supports safe loading (>= 2.0).

There are no versions of psych (>= 0) compatible with your Ruby & RubyGems. psych requires Ruby version >= 2.2.2. The current ruby version is 1.8.

真是操蛋的centos，不折腾了还是装在ubuntu下好了。

ubuntu下一条apt-get install ruby直接搞定。

安装jekyll的问题是找不到头文件，apt-get install ruby-dev，注意ubuntu下是dev后缀对应头文件。

can’t find header files for ruby at /usr/lib/ruby/include/ruby.h

之后
```
$jekyll new myblog
```
发现第一次执行非常慢，不过问题不大

然后就是漫长的端口问题。最初以为4000口没有向外开放，折腾了一晚上ufw iptables这些防火墙设置都无效。最后看应用doc，发现应该是在jekyll里设置相应的内容，默认是localhost 127.0.0.1

_config.yml文件下修改host为内网（公网）IP即可。

找到一些不错的介绍防火墙的内容

https://www.rosehosting.com/blog/how-to-set-up-a-firewall-with-iptables-on-ubuntu-and-centos/

搞定内网调试，开始看官方文档调戏github

https://help.github.com/articles/about-jekyll-themes-on-github/

花了一下午时间找了个好模板

https://github.com/Gaohaoyang/gaohaoyang.github.io

直接开服务，出错

Could not locate Gemfile or .bundle/ directory

不管三七二十一，把示例里面的文件丢过来

之后新的错误

Deprecation: The ‘gems’ configuration option has been renamed to ‘plugins’. Please update your config file accordingly.

Dependency Error: Yikes! It looks like you don’t have jekyll-paginate or one of its dependencies installed. In order to use Jekyll as currently configured, you’ll need to install this gem. The full error message from Ruby is: ‘cannot load such file – jekyll-paginate’ If you run into trouble, you can find helpful resources at https://jekyllrb.com/help/!`

按要求安装之后也不成功。google发现如下stackoverflow帖子

https://stackoverflow.com/questions/35401566/dont-have-jekyll-paginate-or-one-of-its-dependencies-installed

居然有github-pages这种东西……

装完之后，拿掉gemfile即可。

摸索修改之后，发现这个评论解决方案

https://imsun.net/posts/gitment-introduction/

问题是评论页面在我这里会混淆，所以单独设计了一个page给评论，就当留言板吧。

参考

http://www.cnfeat.com/blog/2014/05/11/how-to-build-a-blog/

http://www.pchou.info/ssgithubPage/2013-01-03-build-github-blog-page-01.html（重要）
Read All

46/120

Welcome to LZN's Blog!

山东半岛预报系统搭建

编译WRF

编译WPS

下载geog包并解压。

调整wps设置

执行wps

调整WRF设置

执行real

执行wrf

计划任务

Ubuntu/K4200机器学习系统搭建

记得装desktop版本的ubuntu，否则没有xserver，比较蛋疼

安装英伟达显卡驱动，要关掉x server……

安装cuda toolkit，这次吸取教训，直接英伟达官网下载最新cuda9.1版本。

github技术博客搭建