justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303843.1@dunegpschedd01.fnal.gov

Jobsub ID303843.1@dunegpschedd01.fnal.gov
Workflow ID12725
Stage ID1
User namecalcuttj@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-02-04 03:46:37
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce513
Last heartbeat2026-02-04 04:18:41
From worker nodeHostnameb9p35p0059.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 03:47:57
Input filesvd-protodune:np02vd_raw_run040380_0107_df-s04-d2_dw_0_20251105T184347.hdf5
JobscriptExit code0
Real time30m (1823s)
CPU time28m (1682s = 92%)
Max RSS bytes2399342592 (2288 MiB)
Outputting started2026-02-04 04:18:21
Output files
Finished2026-02-04 04:18:41
Saved logsjustin-logs:303843.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

with light cluster  -1
Cathode? True  with  -1
Anode?  False  exit  [-1, -1, -1]
----


----
Track with ID  1835
match IDs  [[np.int64(5828), np.int64(5846), np.int64(5861)], [-1, -1, -1], [-1, -1, -1], [-1, -1, -1]]  n view matched 3
 From (-34.34, 70.45, -146.60) to (-0.64, 68.30, -195.85)
 module ini  0  module end  0
 Time start  2494  stop  1904
 Timestamp -2244.797411341171 us possibly up to  -2244.797411341171 us
 z-overlap  -151.44388547782447  to  -194.4059472663555
 N Hits  [41, 36, 19]
 theta, phi: [ini] 142.37 ; 2.68  -> [end] 145.17 ; -8.21 
 Straight Lengths :   [58.811202361952915, 56.260739396591056, 51.90283000725994]
 Path Lengths  [72.24612343957838, 64.65610304098341, 53.934278689350045]
 Total charges  [np.float64(357.88316404931965), np.float64(244.35207601850078), np.float64(121.03423962774332)]
 z0  146.60114957359554  t0  -986.1857322640224
 MATCHING DISTANCE SCORE :  1.5352187507208024
 matched with light cluster  -1
Cathode? True  with  -1
Anode?  False  exit  [-1, -1, -1]
----


----
Track with ID  1839
match IDs  [[-1, -1, -1], [np.int64(5875), np.int64(5888), np.int64(5902)], [-1, -1, -1], [-1, -1, -1]]  n view matched 3
 From (-0.78, 66.45, -194.87) to (98.92, 55.08, -339.74)
 module ini  1  module end  1
 Time start  1880  stop  10
 Timestamp -2559.165411341171 us possibly up to  -1253.4916790771485 us
 z-overlap  -203.80731740966095  to  -336.6172640957711
 N Hits  [101, 94, 47]
 theta, phi: [ini] 146.05 ; -6.29  -> [end] 144.15 ; -4.90 
 Straight Lengths :   [164.5342394990322, 171.62985500633312, 172.51478540523883]
 Path Lengths  [194.26495143157942, 182.14302291928615, 178.68171617595738]
 Total charges  [np.float64(721.3500961464313), np.float64(511.5331276089905), np.float64(541.1874103215295)]
 z0  9999.0  t0  9999.0
 MATCHING DISTANCE SCORE :  0.5959024184893058
 matched with light cluster  -1
Cathode? False  with  -1
Anode?  False  exit  [-1, -1, -1]
----

Across modules  [0, 1] merged  1  3D tracks together across module !!! 
Across modules  [2, 3] merged  3  3D tracks together across module !!! 
Found  3  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  18
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  18  TRIGGER  9874 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:43:32 2025  +  454410240.0  ns 
Trigger timestamp  1762368212.4544103

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762368212.4531302
number of samples =  9088
----- Number Of Hits found :  [4223 4106 3321]
---- Number Of 2D tracks found :  [25 19 28]
--- Number of 3D tracks found :  10
-- Found  74  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762368212.4531302
number of samples =  9088
----- Number Of Hits found :  [3547 3303 2558]
---- Number Of 2D tracks found :  [40 35 52]
--- Number of 3D tracks found :  20
-- Found  160  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762368212.453152
number of samples =  9280
----- Number Of Hits found :  [5544 5638 6706]
---- Number Of 2D tracks found :  [105  87 117]
--- Number of 3D tracks found :  53
-- Found  257  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762368212.453152
number of samples =  9280
----- Number Of Hits found :  [2012 1899 2321]
---- Number Of 2D tracks found :  [124 105 147]
--- Number of 3D tracks found :  64
-- Found  318  Single Hits!
Across modules  [0, 1] merged  1  3D tracks together across module !!! 
Across modules  [2, 3] merged  2  3D tracks together across module !!! 
Found  4  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  19
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  19  TRIGGER  9875 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:43:33 2025  +  1044480.0  ns 
Trigger timestamp  1762368213.0010445

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762368212.999766
number of samples =  7872
----- Number Of Hits found :  [2511 2282 1849]
---- Number Of 2D tracks found :  [13 10 15]
--- Number of 3D tracks found :  6
-- Found  59  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762368212.999766
number of samples =  7872
----- Number Of Hits found :  [3098 3133 2272]
---- Number Of 2D tracks found :  [43 41 43]
--- Number of 3D tracks found :  14
-- Found  124  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762368212.999776
number of samples =  8064
----- Number Of Hits found :  [1662 1541 2256]
---- Number Of 2D tracks found :  [58 55 64]
--- Number of 3D tracks found :  20
-- Found  221  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762368212.999776
number of samples =  8064
----- Number Of Hits found :  [2560 2021 3059]
---- Number Of 2D tracks found :  [102  92 125]
--- Number of 3D tracks found :  60
-- Found  280  Single Hits!
WHAAAAT ? too many possibility for cathode stitcher, do not do anything 
Found  2  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  20
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  20  TRIGGER  9878 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:43:33 2025  +  673228032.0  ns 
Trigger timestamp  1762368213.673228

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762368213.671969
number of samples =  8512
----- Number Of Hits found :  [5938 5815 5379]
---- Number Of 2D tracks found :  [60 65 71]
--- Number of 3D tracks found :  11
-- Found  49  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762368213.671969
number of samples =  8512
----- Number Of Hits found :  [4136 3805 3124]
---- Number Of 2D tracks found :  [ 96 105 112]
--- Number of 3D tracks found :  22
-- Found  135  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762368213.6719677
number of samples =  8704
----- Number Of Hits found :  [2133 1845 3325]
---- Number Of 2D tracks found :  [113 127 136]
--- Number of 3D tracks found :  31
-- Found  217  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762368213.6719677
number of samples =  8704
----- Number Of Hits found :  [1440 1500 1917]
---- Number Of 2D tracks found :  [123 137 154]
--- Number of 3D tracks found :  38
-- Found  308  Single Hits!
Across modules  [2, 3] merged  1  3D tracks together across module !!! 
Found  3  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  21
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  21  TRIGGER  9880 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:43:34 2025  +  2311424.0  ns 
Trigger timestamp  1762368214.0023115

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762368214.0010579
number of samples =  7872
----- Number Of Hits found :  [2065 2055 1824]
---- Number Of 2D tracks found :  [12 11 11]
--- Number of 3D tracks found :  5
-- Found  60  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762368214.0010579
number of samples =  7872
----- Number Of Hits found :  [1954 1835 1689]
---- Number Of 2D tracks found :  [18 18 18]
--- Number of 3D tracks found :  7
-- Found  110  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762368214.001056
number of samples =  8064
----- Number Of Hits found :  [ 864 1269 1763]
---- Number Of 2D tracks found :  [24 25 30]
--- Number of 3D tracks found :  11
-- Found  178  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762368214.001056
number of samples =  8064
----- Number Of Hits found :  [1644 1248 2655]
---- Number Of 2D tracks found :  [48 42 52]
--- Number of 3D tracks found :  14
-- Found  250  Single Hits!
Found  1  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  22
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  22  TRIGGER  9882 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:43:34 2025  +  622418432.0  ns 
Trigger timestamp  1762368214.6224186

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762368214.6211596
number of samples =  9152
----- Number Of Hits found :  [7083 6225 6267]
---- Number Of 2D tracks found :  [93 81 97]
3D finalizing track: problem with interpolation
--- Number of 3D tracks found :  120
-- Found  76  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762368214.6211596
number of samples =  9152
----- Number Of Hits found :  [2224 2068  929]
---- Number Of 2D tracks found :  [101  90 111]
--- Number of 3D tracks found :  122
-- Found  142  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762368214.621152
number of samples =  9344
----- Number Of Hits found :  [4291 4209 6138]
---- Number Of 2D tracks found :  [172 169 217]
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
--- Number of 3D tracks found :  246
-- Found  233  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762368214.621152
number of samples =  9344
----- Number Of Hits found :  [1441 1235 1658]
---- Number Of 2D tracks found :  [182 178 232]
--- Number of 3D tracks found :  253
-- Found  317  Single Hits!
Across modules  [0, 1] merged  1  3D tracks together across module !!! 
Across modules  [2, 3] merged  1  3D tracks together across module !!! 
WHAAAAT ? too many possibility for cathode stitcher, do not do anything 
WHAAAAT ? too many possibility for cathode stitcher, do not do anything 
Found  0  cathode crossing tracks 
file closed!
**************
* Reco total *
**************
* Nb of Hits 813561
* Nb of 2D Tracks 7844
* Nb of 3D Tracks 2273
* Nb of Ghosts 0
* Nb of Single Hits 6672
* Nb of PDS Peaks 0
* Nb of PDS Clusters 0
**************
it took 1782.94 s to run 23 events (average of 77.52 per event)
Select jobs to execute...
Select jobs to execute...
Complete log: .snakemake/log/2026-02-04T044830.037657.snakemake.log
Done running snakemake
justIN time: 2026-02-04 06:00:20 UTC       justIN version: 01.06.00