python-mipp an introduction¶
mipp is a Meteorological Ingest-Processing Package (http://github.com/loerum/mipp).
It’s a Python library and it’s main task is to convert low level satellite data into a format understood by mpop (http://github.com/mraspaud/mpop). The primary purpose is to support Geostationary satellite data (level 1.5) but there is also support for the reading of some polar orbiting SAR data (see below).
A more sophisticated interface to satellite data objects is supported by mpop.
Currently it handles data from all current Meteosat Second Generation (MSG) satellites, Meteosat 7, GOES 11-15, MTSAT’s, and GOMS, all as retrieved via EUMETCast:
L-000-MTP___-MET7________-00_7_057E-PRO______-201002261600-__
L-000-MTP___-MET7________-00_7_057E-000001___-201002261600-C_
L-000-MTP___-MET7________-00_7_057E-000002___-201002261600-C_
L-000-MTP___-MET7________-00_7_057E-000003___-201002261600-C_
...
...
L-000-MSG2__-GOES11______-00_7_135W-PRO______-201002261600-__
L-000-MSG2__-GOES11______-00_7_135W-000001___-201002261600-C_
L-000-MSG2__-GOES11______-00_7_135W-000002___-201002261600-C_
L-000-MSG2__-GOES11______-00_7_135W-000003___-201002261600-C_
...
...
In addition mipp handles Synthetic Apperture Radar (SAR) data from Terrscan-X, Cosmo-Sky Med, and Radarsat 2.
mipp will:
- Decompress XRIT files (if Eumetsat’s xRITDecompress is available). Software to uncompress HRIT/XRIT can be obtained from EUMETSAT (register and download the Public Wavelet Transform Decompression Library Software). Please be sure to set the environment variable XRIT_DECOMPRESS_PATH to point to the full path to the decompression software, e.g. /usr/bin/xRITDecompress. Also you can specify where the decompressed files should be stored after decompression, using the environment variable XRIT_DECOMPRESS_OUTDIR. If this variable is not set the decompressed files will be found in the same directory as the compressed ones.
- Decode/strip-off (according to [CGMS], [MTP], [SGS]) XRIT headers and collect meta-data.
- Catenate image data into a numpy-array.
- if needed, convert 10 bit data to 16 bit
- if a region is defined (by a slice or center, size), only read what is specified.
Note
- MET7: not calibrated.
- GOES, METSAT: calibration constants to Kelvin or Radiance (not Reflectance).
Code Layout¶
- xrit.py
It knows about the genric HRIT/XRIT format
- headers = read_headers(file_handle)
- MTP.py
It knows about the specific format OpenMTP for MET7
- mda = read_metadata(prologue, image_file)
- SGS.py
It knows about the specific format Support Ground Segments for GOES and MTSAT
- mda = read_metadata(prologue, image_files)
- sat.py
It knows about satellites base on configurations files. It returns a slice-able object (see below).
- image = load('met7', time_stamp, channel, mask=False, calibrated=True)
- image = load_files(prologue, image_files, **kwarg)
- slicer.py
It knows how to slice satellite images (return from load(...)). It returns meta-data and a numpy array.
- mda, image_data = image[1300:1800,220:520]
- mda, image_data = image(center, size)
Utilities
- cfg.py
It knows how to read configuration files, describing satellites (see below).
- convert.py
10 to 16 byte converter (uses a C extension)
- bin_reader.py
It reads binary data (network byte order)
- read_uint1(buf)
- read_uint2(buf)
- read_float4(buf)
- ...
- mda.py
A simple (anonymous) metadata reader and writer
- geosnav.py
It will convert from/to pixel coordinates to/from geographical longitude, latitude coordinates.
Example definition of a satellite¶
# An item like:
# name = value
# is read in python like:
# try:
# name = eval(value)
# except:
# name = str(value)
#
[satellite]
satname = 'meteosat'
number = '07'
instruments = ('mviri',)
projection = 'geos(57.0)'
[mviri-level2]
format = 'mipp'
[mviri-level1]
format = 'xrit/MTP'
dir = '/data/eumetcast/in'
filename = 'L-000-MTP___-MET7________-%(channel)s_057E-%(segment)s-%Y%m%d%H%M-__'
[mviri-1]
name = '00_7'
frequency = (0.5, 0.7, 0.9)
resolution = 2248.49
size = (5000, 5000)
[mviri-2]
name = '06_4'
frequency = (5.7, 6.4, 7.1)
resolution = 4496.98
size = (2500, 2500)
[mviri-3]
name = '11_5'
frequency = (10.5, 11.5, 12.5)
resolution = 4496.98
size = (2500, 2500)
Usage¶
import xrit
image = xrit.sat.load('meteosat07', datetime(2010, 2, 1, 10, 0), '00_7', mask=True)
mda, image_data = image(center=(50., 10.), size=(600, 500))
print mda
fname = './' + mda.product_name + '.dat'
print >>sys.stderr, 'Writing', fname
fp = open(fname, "wb")
image_data.tofile(fp)
fp.close()
Examples of the usage of some lower level tools¶
Here an example how to get the observation times (embedded in the ‘Image Segment Line Quality’ record) of each scanline in a segment:
import mipp.xrit.MSG
segfile = "/local_disk/data/MSG/HRIT/H-000-MSG3__-MSG3________-WV_062___-000002___-201311211300-__"
lineq = mipp.xrit.MSG.get_scanline_quality(segfile)
print lineq[0]
(465, datetime.datetime(2013, 11, 21, 13, 1, 48, 924000), 1, 1, 0)
A script, process_fsd¶
The script is intended for work on other geostationary data than the MSG (Meteosat) data, the so-called Foreign Satellite Data (FSD). That is e.g. GOES, MTSAT and COMS.
process_fsd --check-satellite <prologue-file>
check if we handle this satellite
process_fsd --check [-l] <prologue-file>
check if number of image segments are as planned
-l, list corresponding image segment files
process_fsd --decompress [-o<output-dir>] <file> ... <file>
decompress files to output-dir (default is working directory)
-l, list decompressed files
process_fsd --metadata <prologue-file> <image-segment> ... <image-segment>
print meta-data
process_fsd [-o<output-dir>] <prologue-file> <image-segment> ... <image-segment>
it will binary dump image-data and ascii dump of meta-data)
[CGMS] LRIT/HRIT Global Specification; CGMS 03; Issue 2.6; 12 August 1999 “MSG Ground Segment LRIT/HRIT Mission Specific Implementation” EUM/MSG/SPE/057; Issue 6; 21 June 2006
[MTP] “The Meteosat Archive; Format Guide No. 1; Basic Imagery: OpenMTP Format”; EUM FG 1; Rev 2.1; April 2000
[SGS] “MSG Ground Segment LRIT/HRIT Mission Specific Implementation”; EUM/MSG/SPE/057; Issue 6; 21 June 2006