Welcome to LZN's Blog!
Wind extinguishes a candle but energizes fire.-
Start to use Python for Most Work
Start to use python to deal with the WRF output. xarray is a very important package to deal with
NetCDFandHDF5data. As WRF files are not exactly CF compliant: you’ll need a special parser for the timestamp, the coordinate names are a bit exotic and do not correspond to the dimension names, they contain so-called staggered variables (and their correponding coordinates), etc.salem is needed to parser wrf data. This is useful to slice wrf data in xarray.
ds=ds.sel(time=slice('2018-09-15','2018-09-17'))Updated 2020-05-03
-
Object Detection Using Tensorflow Models
Please scroll to the FINAL for Ultimate Solution!
Follow the procedure to install the libs and dependencies.
In the last step, testing the installation:
Running tests under Python 3.6.10: /users/b145872/project-dir/Anaconda3_2020/envs/tf1.14/bin/python [ RUN ] ModelBuilderTest.test_create_experimental_model [ OK ] ModelBuilderTest.test_create_experimental_model [ RUN ] ModelBuilderTest.test_create_faster_rcnn_model_from_config_with_example_miner [ OK ] ModelBuilderTest.test_create_faster_rcnn_model_from_config_with_example_miner [ RUN ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_faster_rcnn_with_matmul [ OK ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_faster_rcnn_with_matmul [ RUN ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_faster_rcnn_without_matmul [ OK ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_faster_rcnn_without_matmul [ RUN ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_mask_rcnn_with_matmul [ OK ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_mask_rcnn_with_matmul [ RUN ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_mask_rcnn_without_matmul [ OK ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_mask_rcnn_without_matmul [ RUN ] ModelBuilderTest.test_create_rfcn_model_from_config [ OK ] ModelBuilderTest.test_create_rfcn_model_from_config [ RUN ] ModelBuilderTest.test_create_ssd_fpn_model_from_config [ OK ] ModelBuilderTest.test_create_ssd_fpn_model_from_config [ RUN ] ModelBuilderTest.test_create_ssd_models_from_config [ OK ] ModelBuilderTest.test_create_ssd_models_from_config [ RUN ] ModelBuilderTest.test_invalid_faster_rcnn_batchnorm_update [ OK ] ModelBuilderTest.test_invalid_faster_rcnn_batchnorm_update [ RUN ] ModelBuilderTest.test_invalid_first_stage_nms_iou_threshold [ OK ] ModelBuilderTest.test_invalid_first_stage_nms_iou_threshold [ RUN ] ModelBuilderTest.test_invalid_model_config_proto [ OK ] ModelBuilderTest.test_invalid_model_config_proto [ RUN ] ModelBuilderTest.test_invalid_second_stage_batch_size [ OK ] ModelBuilderTest.test_invalid_second_stage_batch_size [ RUN ] ModelBuilderTest.test_session [ SKIPPED ] ModelBuilderTest.test_session [ RUN ] ModelBuilderTest.test_unknown_faster_rcnn_feature_extractor [ OK ] ModelBuilderTest.test_unknown_faster_rcnn_feature_extractor [ RUN ] ModelBuilderTest.test_unknown_meta_architecture [ OK ] ModelBuilderTest.test_unknown_meta_architecture [ RUN ] ModelBuilderTest.test_unknown_ssd_feature_extractor [ OK ] ModelBuilderTest.test_unknown_ssd_feature_extractor ---------------------------------------------------------------------- Ran 17 tests in 0.427sNow we test the object detection script. As the model need to run on GPU cluster, we cannot simply use jupyter notebook. So convert to plain python code.
jupyter nbconvert --to python object_detection_tutorial.ipynbExcecute the python code, got:
ModuleNotFoundError: No module named 'object_detection'Compile protobufs and install the object_detection package:
cd models/research/ protoc object_detection/protos/*.proto --python_out=. pip install .Execute again, got:
tensorflow.python.framework.errors_impl.NotFoundError: models/research/object_detection/data/mscoco_label_map.pbtxt; No such file or directoryThis is the relative path problem caused by changing directory from ipython notebook.
Execute again, got:
File "object_detection_tutorial.py", line 98, in <module> detection_model = load_model(model_name) File "object_detection_tutorial.py", line 62, in load_model model = tf.saved_model.load(str(model_dir), None) File "/users/b145872/project-dir/Anaconda3_2020/envs/tf1.14/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func return func(*args, **kwargs) TypeError: load() missing 1 required positional argument: 'export_dir'Version Change:
model = tf.saved_model.load(export_dir=str(model_dir)) # change to: model = tf.saved_model.load_v2(export_dir=str(model_dir),tags=None)New error:
Traceback (most recent call last): File "object_detection_tutorial.py", line 191, in <module> show_inference(detection_model, image_path) File "object_detection_tutorial.py", line 172, in show_inference output_dict = run_inference_for_single_image(model, image_np) File "object_detection_tutorial.py", line 141, in run_inference_for_single_image num_detections = int(output_dict.pop('num_detections')) TypeError: int() argument must be a string, a bytes-like object or a number, not 'Tensor'…….
Ultimate Solution
After multiple tests, we still cannot run the script on gpu smoothly, tf1.9 1.14 and 2.1 all failed. When I come back to the github page. I found the updated
ipynb… And this time, the tf2.1-based env can run it with GPU! Although there are still errors, we reinstalled the tf2.1 byconda --force-reinstall, everything goes nice!Updated 2020-05-10
-
Scratch Notes for Stanford CS107: Programming Paradigm Ch01-02
L1: Administrative Details
Notes
- C
- Assambly
- C++
- Concurrent Programming
- Scheme
- Python
C is more procedure-oriented, from main to sub function, and sub function to sub-sub function. “Procedure Paradiam”
Concurrent programming issue: The shared resource that two different processes to access to.
Scheme is in functional paradiam: rely on a return from a function to move forward! Thus you can program without side effects.
e.g. side effects in C/C++: Pass a data structure into a function by reference, and change it in the function, when return, it changed.
Python, with good web libraries.
Qustions
- Here talking about concurrent programming in one CPU? Seemingly in a Time-sharing system.
Glossary
handouts 讲义 mileage 里程 pertinent 相关的 synopsis 概要 assembly code 汇编代码 asterisks 星号 ampersands 连字号 miniature 微型L2: C/C++ Data Types - Interpretations
Notes
Low-level memory mechanisms.
C/C++ Data types |Data Type|Length| |—-|—-| |bool|1 byte| |char|1 byte| |short|2 byte| |int|4 byte*| |long|4 byte| |float|4 byte| |double|8 byte|
Technically, a Boolean can be mapped by a single bit in memory. Can engineer the compiler to do so.
Considering sign, the first bit in 1 — minus sign.
When 0000 0000 0000 0111 + 1000 0000 0000 0111, if the first bit is sign, the result is not right. To deal with this problem, use invert + 1 to represent negative values. Two’s complement(补码)
char ch = 'A'; short s = ch; cout << s << endl; OUTPUT: 65ch 1111 65 s 1111 65 int i = 2^21+2^10+2^1 short s = i OUTPUT: 1026short s = -1; int i = s;s 1111 1111 1111 1111 i 1111 1111 1111 1111 1111 1111 1111 1111 Floating things are wierd. See IEEE 754
Single (32 bits/ 4 bytes):
(-1)^sign(1.b22b21…b0)2^(e-127)
Double (64 bits/8 bytes):
Qustions
Precision varies with the magnitude of the represented number:
Single Precision: 10^N~10e(N-8) Double Precision: 10^N-10e(N-16)
int i =5; float f=i; cout<<f<<endl;5=>5.0=1.25*2^e # Totally different!
Be careful with the precision loss when convert
inttofloat!Doublemakes things better, merely better.Glossary
Updated 2020-04-26