;+ ; $Id: scc_read_summary.pro,v 1.25 2022/06/06 15:36:49 thompson Exp $ ; ; Project : STEREO - SECCHI ; ; Name : SCC_READ_SUMMARY() ; ; Purpose : Read one or more SECCHI summary files. ; ; Category : SECCHI, Catalog ; ; Explanation : This procedure reads through the SECCHI summary files for a ; given date. Optionally, one can restrict the files read by ; spacecraft and type. One can also pass a time range to search. ; ; Syntax : List = SCC_READ_SUMMARY( [ FILES ], keywords ) ; ; Examples : List = SCC_READ_SUMMARY(DATE='1-Nov-2006') ; List = SCC_READ_SUMMARY(DATE=['1-Nov-2006 12','1-Nov-2006 14']) ; List = SCC_READ_SUMMARY('./sccA200611.seq') ; ; Inputs : None required. ; ; Opt. Inputs : FILES = List of files to read (with paths). If not passed, ; then formed from the observation date and other ; keywords. Passing in an explicit list of filenames ; bypasses many of the keywords. ; ; Outputs : The result of the function is a structure array containing the ; following elements: ; ; FILENAME = Name of the file ; DATE_OBS = Observation date ; TELESCOPE = Telescope (EUVI,COR1,COR2 ; EXPTIME = Exposure time, in seconds ; XSIZE = Array size along X, in pixels ; YSIZE = Array size along Y, in pixels ; FILTER = Stray light filter ; VALUE = Polarization angle or EUVI wavelength ; PROG = Observation program (Dark/Led/Norm/Sequ) ; OSNUM = Observation number ; DEST = Destination (RT/SSR1/SSR2/SW) ; FPS = Fine Pointing System (ON/OFF) ; LED = LED (e.g. RedFPA, BluTel, None) ; COMPR = Type of compression (ICER0..ICER11, RICE, HC0..HC9) ; NMISS = Number of missing blocks or Icer segments ; SPACECRAFT = Either A or B ; TYPE = Type (cal/img/seq) ; ; If no valid entries are found, then the number 0 is returned. ; ; Opt. Outputs: None ; ; Keywords : DATE = Either the date of observation, or a beginning and ; end time to search on. When a single date is passed, ; the entire day is returned. Otherwise, only the ; portion between the beginning and end times are ; returned. ; ; If neither FILES nor DATE is passed, then the default ; is to use today's date. ; ; Note that when two dates are passed, they're treated ; as times. If no specific time of day is specified, ; then 00:00 is assumed. For example, ; ; DATE=['2007-12-01', '2007-12-03'] ; ; would return all data from Dec 1 and Dec 2, but would ; stop at the beginning of Dec 3. ; ; CADENCE = desired miniminum time between images in minutes ; ; SOURCE = Telemetry source, either "rt" (realtime), "pb" ; (playback), or "lz" (level-zero). If passed, and the ; environment variable "secchi" is defined, then ; $secchi is used to form the path. ; ; TYPE = The type of data. Can be "cal", "img", or "seq". If ; not passed, then all types are searched. ; ; BEACON = If set, then the beacon summary files are read. ; ; NOBEACON= If set, then spaceweather images are omitted from ; result. ; ; QUADRANT= The EUVI quadrant of interest; one of 171,195,304,284 ; ; POLAR = The polarizer position of interest ; ; SPACECRAFT = Can be one of the following forms: ; ; 'A' 'B' ; 'STA' 'STB' ; 'Ahead' 'Behind' ; 'STEREO Ahead' 'STEREO Behind' ; 'STEREO-Ahead' 'STEREO-Behind' ; ; If not passed, then both spacecraft are searched. ; ; TELESCOPE = SECCHI telescope. Can be "COR1", "COR2", "EUVI", ; "HI1" ("HI_1"), or "HI2" ("HI_2"). If not passed, ; all telescopes are searched. ; ; TOTALB = If set, the summary file for the _NRL ONLY_ total-B ; archive is read. ; ; CHECK = If set, then SCC_CHECK_BAD is called to cull out any ; known bad images. ; ; DEBRIS = Used together with /CHECK to cull out known debris ; images. ; QUIET = if set, avoid some messages ; ; FINDFITS= If set, then call SCCFINDFITS ; ; ERRMSG = If defined and passed, then any error messages will ; be returned to the user in this parameter rather than ; depending on the MESSAGE routine in IDL. If no ; errors are encountered, then a null string is ; returned. In order to use this feature, ERRMSG must ; be defined first, e.g. ; ; ERRMSG = '' ; Result = SCC_READ_SUMMARY( ERRMSG=ERRMSG, ... ) ; IF ERRMSG NE '' THEN ... ; ; Env. Vars. : Either $secchi or $SECCHI_LZ. See scc_data_path.pro for a ; complete list of supported environment variables. ; ; Calls : ANYTIM2UTC, UTC2TAI, BREAK_FILE, VALID_NUM, SCC_DATA_PATH, ; SSC_CHECK_BAD, SCCFINDFITS ; ; Common : None. ; ; Restrictions: None. ; ; Side effects: None. ; ; Prev. Hist. : None. ; ; History : ; 11-apr-2008 - S.L.Freeland - restore QUADRANT&POLAR(!) ; 9-apr-2008 - S.L.Freeland - "no files" trap, /QUIET ; 2-apr-2008 - S.L.Freeland - vectorized read/struct convert ; Move time selection prior to conversion (string op) ; ; $Log: scc_read_summary.pro,v $ ; Revision 1.25 2022/06/06 15:36:49 thompson ; *** empty log message *** ; ; Revision 1.24 2017/04/21 20:48:25 thompson ; *** empty log message *** ; ; Revision 1.23 2011/11/14 20:13:33 nathan ; add CADENCE= keyword ; ; Revision 1.22 2008/05/23 18:26:56 thompson ; 23-may-2008 - F. Auchere - Bug fixed in case the "secchi" environment variable contain dots ; ; Revision 1.21 2008/04/11 18:47:18 thompson ; 11-apr-2008 - S.L.Freeland - restore QUADRANT&POLAR(!) ; ; Revision 1.20 2008/04/10 16:01:22 thompson ; Restore /NOBEACON keyword ; ; Revision 1.19 2008/04/09 21:28:36 thompson ; Incorporate changes by Sam Freeland ; ; Revision 1.18 2008/04/07 15:53:24 thompson ; *** empty log message *** ; ; Revision 1.17 2008/02/28 15:37:49 thompson ; Added keywords /CHECK and /DEBRIS. ; Normalized all error conditions to return 0 ; ; Revision 1.16 2007/11/13 20:15:48 nathan ; expand filename format from 25 to 26 chars for SECCHI_P0 ; ; Revision 1.15 2007/08/28 21:11:51 nathan ; added kw QUADRANT and POLAR ; ; Revision 1.14 2007/06/29 20:24:05 thompson ; Fixed bug when two dates are entered. ; ; Revision 1.13 2007/05/22 18:54:47 nathan ; Add /NOBEACON option ; ; Revision 1.12 2007/05/21 16:24:35 reduce ; A cut-n-paste error with /totalb reading. Now fixed. Karl B. ; ; Revision 1.11 2007/05/02 15:11:39 euvi ; Bug fix: read all files betw. dates, not just start/end -- JPW ; ; Revision 1.10 2007/04/27 19:54:21 reduce ; left a stop point in... oops! -- KarlB ; ; Revision 1.9 2007/04/27 19:51:32 reduce ; Bug fix where totalB summary file was not being found -- KarlB ; ; Revision 1.8 2007/04/11 18:13:35 nathan ; add nmiss to output structure; search only tel-specific summary files if specified ; ; Revision 1.7 2007/04/10 20:12:50 nathan ; Add compr to output; read new format summaryfilename; add /VERBOSE ; ; Version 1, 12-Jun-2006, William Thompson, GSFC ; Version 2, 13-Jun-2006, William Thompson, GSFC ; Added FPS and LED tags, renamed POLAR to VALUE ; Version 3, 14-Jun-2006, William Thompson, GSFC ; Added SPACECRAFT tag, BEACON keyword ; Use today as default ; Version 4, 13-Nov-2006, William Thompson, GSFC ; Added missing SOURCE keyword ; Sort results, filter out duplicate entries. ; Version 5, 21-Dec-2006, William Thompson, GSFC ; Handle "******" for exposure time. ; Version 6, 08-Jan-2006, William Thompson, GSFC ; Skip over any IO errors ; Version 7, 21-Apr-2017, WTT, add FINDFITS keyword. ; Version 8, 6-Jun-2022, WTT, allow FINDFITS with /BEACON ; ; Contact : WTHOMPSON ;- ; function scc_read_summary, files, date=date_s, type=type_s, quadrant=quadrant, polar=polar, $ spacecraft=spacecraft_s, telescope=telescope_s, $ beacon=beacon, source=source, errmsg=errmsg, $ totalb=totalb, verbose=verbose, nobeacon=nobeacon, $ check=check, debris=debris, quiet=quiet, $ CADENCE=cadence, findfits=findfits if keyword_set(totalb) THEN BEGIN tbpath=GETENV('SECCHI_P0') if tbpath EQ '' THEN BEGIN PRINT,'*** ERROR ***' PRINT,'The totalb keyword is only for NRL-based users.' PRINT,'If you are at NRL, then your environment is not correctly set.' PRINT,'Please check you have $SECCHI_P0 defined' return,0 endif endif loud=1-keyword_set(quiet) ; ; on_error, 2 ; ; Define the structure containing the result. ; filename = '' date_obs = '' telescope = '' sexptime = '' exptime = 0.0 xsize = 0 ysize = 0 filter = '' value = 0.0 svalue = '' prog = '' osnum = 0 dest = '' fps = '' led = '' compr = '' nmiss = -1 ; ; result0 = {filename: filename, $ date_obs: date_obs, $ telescope: telescope, $ exptime: '', $ ; string for now (float later) xsize: xsize, $ ysize: ysize, $ filter: filter, $ value: '', $ ; string for now (float later) prog: prog, $ osnum: osnum, $ dest: dest, $ fps: fps, $ led: led, $ compr: compr, $ nmiss: nmiss, $ spacecraft: '', $ type: ''} ; ; If FILES is not passed, then define it based on the keywords. First, parse ; the DATE keyword. ; if n_elements(files) eq 0 then begin if n_elements(date_s) eq 0 then get_utc, date_s if n_elements(date_s) gt 2 then begin message = 'DATE must have one or two elements' goto, handle_error endif message = '' utc = anytim2utc(date_s, errmsg=message, /external) if message ne '' then goto, handle_error if n_elements(utc) eq 1 then utc = replicate(utc,2) temp = utc2tai(utc,/nocorrect) if temp[0] gt temp[1] then utc = reverse(utc) ; ; Convert the dates into year-month strings. If multimonth, collect all the ; months between the two dates. ; sdate = string(utc.year,format='(I4.4)')+string(utc.month,format='(I2.2)') if sdate[0] ne sdate[1] then begin nyears = utc[1].year - utc[0].year + 1 year = utc[0].year + indgen(12*nyears) / 12 month = (indgen(12*nyears) mod 12) + 1 sdates = string(year,format='(I4.4)')+string(month,format='(I2.2)') w = where((sdates ge sdate[0]) and (sdates le sdate[1])) sdate = sdates[w] end else sdate = sdate[0] ; ; Parse the spacecraft, if passed. ; if n_elements(spacecraft_s) eq 1 then begin sc = strtrim(strupcase(spacecraft_s),2) n = strlen(sc) if (sc eq 'STA') or (sc eq strmid('AHEAD',0,n)) or $ (sc eq strmid('STEREO AHEAD',0,n)) or $ (sc eq strmid('STEREO-AHEAD',0,n)) then sc = 'A' if (sc eq 'STB') or (sc eq strmid('BEHIND',0,n)) or $ (sc eq strmid('STEREO BEHIND',0,n)) or $ (sc eq strmid('STEREO-BEHIND',0,n)) then sc = 'B' end else sc='' ; ; If a telescope was specified, specify summary file suffix. ; tel_s='.??' ; default is all if n_elements(telescope_s) eq 1 then begin inpt= strupcase(telescope_s[0]) case inpt of 'COR1': tel_s='.c1' 'COR2': tel_s='.c2' 'EUVI': tel_s='.eu' 'HI1': tel_s='.h1' 'HI2': tel_s='.h2' 'HI_1': tel_s='.h1' 'HI_2': tel_s='.h2' ELSE: endcase endif ; ; Look for files matching the search criterion. ; if n_elements(type_s) eq 0 then type_s = '*' if (sc eq 'A') or (sc eq '') then begin message = '' if keyword_set(totalb) THEN dir = scc_data_path('a',type='summary',/totalb) ELSE $ dir = scc_data_path('Ahead', type='summary', source=source, $ beacon=beacon, errmsg=message) if message ne '' then goto, handle_error for idate = 0,n_elements(sdate)-1 do begin if keyword_set(totalb) then files0 = file_search(dir, '/sccA' + sdate[idate] + '.' + type_s) else $ files0 = file_search(dir, 'sccA' + sdate[idate] + '.' + type_s +tel_s) IF keyword_set(VERBOSE) THEN print,files0 ;stop if files0[0] ne '' then begin if n_elements(files) eq 0 then begin files = files0 scs = replicate('A',n_elements(files0)) end else begin files = [files, files0] scs = [scs, replicate('A',n_elements(files0))] endelse endif endfor endif if (sc eq 'B') or (sc eq '') then begin message = '' if keyword_set(totalb) THEN dir = scc_data_path('b',type='summary',/totalb) ELSE $ dir = scc_data_path('Behind', type='summary', source=source, $ beacon=beacon, errmsg=message) if message ne '' then goto, handle_error for idate = 0,n_elements(sdate)-1 do begin if keyword_set(totalb) then files0 = file_search(dir, '/sccB' + sdate[idate] + '.' + type_s) else $ files0 = file_search(dir, 'sccB' + sdate[idate] + '.' + type_s +tel_s) IF keyword_set(VERBOSE) THEN print,files0 if files0[0] ne '' then begin if n_elements(files) eq 0 then begin files = files0 scs = replicate('B',n_elements(files0)) end else begin files = [files, files0] scs = [scs, replicate('B',n_elements(files0))] endelse endif endfor endif endif ;Files not passed. ; errmsg=(['','No files for date/instrument'])(n_elements(files) eq 0) message=errmsg if errmsg ne '' then goto, handle_error ; error out on No files ; Define the format code for reading the file. ; format = '(A26,2X,A19,3X,A4,2X,A6,1X,I5,2X,I5,3X,A5,2X,A6,3X,A4,2X,I5,' + $ '3X,A4,3X,A3,3X,A6,3X,A6,3X,I4,1x,A1,1x,A)' ; ; Open the files, and read in the lines, skipping over the two-line header. ; For efficiency, let the structure grow in steps of 1000. ; S.L.Freeland 2-apr-2008 - vecorize read and structure convert segments ; iresult = 0L buffer=rd_tfiles(files,filemap=filemap) ; all files-> string array buffer=buffer+' A ' +ssw_strsplit(files(filemap),'/',/tail) ssdata=where(strpos(buffer,'20') eq 0, okcnt) ; assume data begins w/YYYY if okcnt eq 0 then begin errmsg="No data records??" if loud then box_message,errmsg return,0 ; !! Early Exit endif buffer=temporary(buffer(ssdata)) ; if n_elements(date_s) ne 0 then begin ; do date_s windowing here 2-apr-2008 fdates=strmid(buffer,28,10) ftimes=strmid(buffer,28,19) dcnt=0 case n_elements(date_s) of 1: ssd=where(fdates eq anytim2utc(date_s,/ecs,/date_only),dcnt) 2: ssd=where((ftimes ge anytim2utc(date_s[0],/ecs,/truncate)) and $ (ftimes le anytim2utc(date_s[1],/ecs,/truncate)),dcnt) else: box_message,'DATE must have one or two elements' endcase if dcnt gt 0 then buffer=buffer(ssd) else begin errmsg="No records in your DATE Range" if loud then box_message,errmsg return,0 ; early exit endelse endif result=replicate(result0,n_elements(buffer)) ; structures 1:1 ascii reads,buffer, result, format=format ; buffer->structure vector ; slf stags=str2arr('filename,telescope,filter,prog,dest,fps,led,compr') for i=0,n_elements(stags)-1 do begin int=tag_index(result,stags(i)) result.(tag_index(result,stags(i)))= $ strtrim(result.(tag_index(result,stags(i))),2) ; trim string tags endfor sco=(['A','B'])(strpos(result.filename,'B.') gt 0) result.spacecraft=sco filename = ssw_strsplit(result.type, get_delim(), /tail) result.type=strextract(filename, '.') result=rep_tag_value(result,float(str2number(result.exptime)),'exptime') result=rep_tag_value(result,float(str2number(result.value)),'value') ; ; If a time range was passed, select out those entries within the time range. ; (2-feb-2008 - DATE selection done prior to structure convert) ; ; If a telescope was specified, select out the matching entries. ; ;stop ;if n_elements(telescope_s) eq 1 then begin ; w = where(result.telescope eq strupcase(telescope_s), count) ; if count gt 0 then result = result[w] else result = result0 ;endif ; ; If the /NOBEACON keyword was set, then filter out beacon images. if keyword_set(nobeacon) then begin w = where(result.dest ne 'SW', count) if count eq 0 then return, 0 else result = result[w] endif if keyword_set(quadrant) then begin ; optional EUVI quadrant requested? w = where(result.value eq float(quadrant), count) if count eq 0 then return, 0 else result=result[w] endif if keyword_set(polar) then begin w = where(result.value eq float(polar), count) if count eq 0 then return, 0 else result=result[w] endif ; ; Sort by a combination of filename and record number, and remove duplicate ; records. ; test = result.filename + string(lindgen(n_elements(result)), format='(I10.10)') s = sort(test) result = result[s] u = uniq(result.filename) result = result[u] ; if keyword_set(cadence) then begin tai=anytim2tai(result.date_obs) if temp(0) eq temp(1) then dys=86400.0 else dys=temp(1)-temp(0) ;dys= number of seconds in interval timegrid=findgen(((dys/60) /cadence)+1) *(cadence*60.) +temp[0] z=intarr(n_elements(timegrid)) for n=0,n_elements(timegrid)-1 do z[n]=find_closest(double(timegrid[n]),tai,/less,/quiet) result=result[z] endif ; ; If the /CHECK keyword is set, then call scc_check_bad. ; if keyword_set(check) then begin bad = scc_check_bad(result.filename, debris=debris) w = where(bad eq 0, count) if count eq 0 then return, 0 else result = result[w] endif ; if keyword_set(findfits) then result.filename = $ sccfindfits(result.filename, beacon=beacon) ; return, result ; ; Error handling point. ; handle_error: if n_elements(errmsg) eq 0 then message, message else $ errmsg = 'scc_read_summary: ' + message return, 0 ; end