justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303843.12@dunegpschedd01.fnal.gov

Jobsub ID303843.12@dunegpschedd01.fnal.gov
Workflow ID12725
Stage ID1
User namecalcuttj@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-02-04 03:46:37
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce513
Last heartbeat2026-02-04 04:07:39
From worker nodeHostnameb9p35p0059.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 03:48:10
Input filesvd-protodune:np02vd_raw_run040380_0033_df-s04-d0_dw_0_20251105T181130.hdf5
JobscriptExit code0
Real time19m (1152s)
CPU time17m (1044s = 90%)
Max RSS bytes2315751424 (2208 MiB)
Outputting started2026-02-04 04:07:23
Output files
Finished2026-02-04 04:07:39
Saved logsjustin-logs:303843.12-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

s found :  [ 60  80 112]
3D finalizing track: problem with interpolation
--- Number of 3D tracks found :  16
-- Found  189  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762366281.6278718
number of samples =  8064
----- Number Of Hits found :  [4128 6666 7962]
---- Number Of 2D tracks found :  [117 144 205]
3D finalizing track: problem with interpolation
--- Number of 3D tracks found :  27
-- Found  273  Single Hits!
Across modules  [2, 3] merged  1  3D tracks together across module !!! 
Found  2  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  13
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  13  TRIGGER  3070 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:11:21 2025  +  637145344.0  ns 
Trigger timestamp  1762366281.6371455

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762366281.6358893
number of samples =  8448
----- Number Of Hits found :  [2901 2859 2274]
---- Number Of 2D tracks found :  [ 9 14 14]
--- Number of 3D tracks found :  7
-- Found  71  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762366281.6358893
number of samples =  8448
----- Number Of Hits found :  [2165 1834 1261]
---- Number Of 2D tracks found :  [17 21 21]
--- Number of 3D tracks found :  11
-- Found  139  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762366281.635872
number of samples =  8704
----- Number Of Hits found :  [ 6835 11207 11408]
---- Number Of 2D tracks found :  [116 128 175]
--- Number of 3D tracks found :  22
-- Found  223  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762366281.635872
number of samples =  8704
----- Number Of Hits found :  [ 5940  8987 10115]
---- Number Of 2D tracks found :  [192 226 321]
--- Number of 3D tracks found :  32
-- Found  330  Single Hits!
Across modules  [0, 1] merged  1  3D tracks together across module !!! 
WHAAAAT ? too many possibility for cathode stitcher, do not do anything 
Found  1  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  14
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  14  TRIGGER  3071 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:11:21 2025  +  640752128.0  ns 
Trigger timestamp  1762366281.6407523

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762366281.6394937
number of samples =  11264
----- Number Of Hits found :  [3478 3685 2782]
---- Number Of 2D tracks found :  [20 17 26]
--- Number of 3D tracks found :  11
-- Found  78  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762366281.6394937
number of samples =  11264
----- Number Of Hits found :  [4090 3484 3576]
---- Number Of 2D tracks found :  [41 34 47]
--- Number of 3D tracks found :  20
-- Found  192  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762366281.6394882
number of samples =  11520
----- Number Of Hits found :  [13151 20958 21860]
---- Number Of 2D tracks found :  [214 222 320]
3D finalizing track: problem with interpolation
--- Number of 3D tracks found :  50
-- Found  310  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762366281.6394882
number of samples =  11520
----- Number Of Hits found :  [12278 18830 21455]
---- Number Of 2D tracks found :  [362 422 602]
--- Number of 3D tracks found :  75
-- Found  406  Single Hits!
Across modules  [0, 1] merged  2  3D tracks together across module !!! 
Across modules  [2, 3] merged  1  3D tracks together across module !!! 
Found  5  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  15
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  15  TRIGGER  3072 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:11:21 2025  +  642852608.0  ns 
Trigger timestamp  1762366281.6428528

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762366281.6415908
number of samples =  13248
----- Number Of Hits found :  [4528 4613 4018]
---- Number Of 2D tracks found :  [29 24 33]
--- Number of 3D tracks found :  13
-- Found  70  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762366281.6415908
number of samples =  13248
----- Number Of Hits found :  [4353 3983 3646]
---- Number Of 2D tracks found :  [48 47 64]
--- Number of 3D tracks found :  21
-- Found  160  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762366281.6416001
number of samples =  13504
----- Number Of Hits found :  [15515 25868 27059]
---- Number Of 2D tracks found :  [270 297 386]
--- Number of 3D tracks found :  44
-- Found  252  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762366281.6416001
number of samples =  13504
----- Number Of Hits found :  [13542 21470 25117]
---- Number Of 2D tracks found :  [442 534 699]
--- Number of 3D tracks found :  63
-- Found  347  Single Hits!
Across modules  [0, 1] merged  2  3D tracks together across module !!! 
Found  8  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  16
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  16  TRIGGER  3073 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:11:21 2025  +  651931648.0  ns 
Trigger timestamp  1762366281.6519318

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762366281.6506677
number of samples =  8512
----- Number Of Hits found :  [2450 2803 2074]
---- Number Of 2D tracks found :  [12 14 17]
--- Number of 3D tracks found :  4
-- Found  74  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp: /local_software/lardon/src/lardon/track_2d.py:143: UserWarning: 
A theoretically impossible result was found during the iteration
process for finding a smoothing spline with fp = s: s too small.
There is an approximation returned but the corresponding weighted sum
of squared residuals does not satisfy the condition abs(fp-s)/s < tol.
  spline_z = UnivariateSpline(z_r_unique, x_r_uz)
 1762366281.6506677
number of samples =  8512
----- Number Of Hits found :  [3148 2815 2712]
---- Number Of 2D tracks found :  [32 35 41]
--- Number of 3D tracks found :  15
-- Found  159  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762366281.6506557
number of samples =  8704
----- Number Of Hits found :  [4658 7405 8222]
---- Number Of 2D tracks found :  [ 89 117 123]
--- Number of 3D tracks found :  21
-- Found  245  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762366281.6506557
number of samples =  8704
----- Number Of Hits found :  [ 9701 14790 17575]
---- Number Of 2D tracks found :  [214 248 309]
--- Number of 3D tracks found :  30
-- Found  351  Single Hits!
Found  3  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  17
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  17  TRIGGER  3074 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:11:21 2025  +  671866112.0  ns 
Trigger timestamp  1762366281.6718662

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762366281.6705906
number of samples =  11328
----- Number Of Hits found :  [4602 4356 4037]
---- Number Of 2D tracks found :  [27 26 35]
--- Number of 3D tracks found :  8
-- Found  78  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762366281.6705906
number of samples =  11328
----- Number Of Hits found :  [4009 3942 3750]
---- Number Of 2D tracks found :  [49 52 61]
--- Number of 3D tracks found :  14
-- Found  167  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762366281.670592
number of samples =  11584
----- Number Of Hits found :  [ 9896 16164 16985]
---- Number Of 2D tracks found :  [175 208 270]
--- Number of 3D tracks found :  21
-- Found  288  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762366281.670592
number of samples =  11584
----- Number Of Hits found :  [11668 17689 19669]
---- Number Of 2D tracks found :  [308 382 527]
--- Number of 3D tracks found :  36
-- Found  389  Single Hits!
Across modules  [0, 1] merged  2  3D tracks together across module !!! 
Found  6  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  18
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  18  TRIGGER  3075 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:11:21 2025  +  683182592.0  ns 
Trigger timestamp  1762366281.6831827

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762366281.6819284
number of samples =  7872
----- Number Of Hits found :  [3560 3036 2279]
---- Number Of 2D tracks found :  [24 15 31]
--- Number of 3D tracks found :  8
-- Found  79  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762366281.6819284
number of samples =  7872
----- Number Of Hits found :  [2785 2204 1863]
---- Number Of 2D tracks found :  [42 28 49]
--- Number of 3D tracks found :  12
-- Found  143  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762366281.68192
number of samples =  8064
----- Number Of Hits found :  [ 7159 12045 12532]
---- Number Of 2D tracks found :  [125 132 187]
--- Number of 3D tracks found :  18
-- Found  220  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762366281.68192
number of samples =  8064
----- Number Of Hits found :  [ 8062 12320 13962]
---- Number Of 2D tracks found :  [218 249 344]
--- Number of 3D tracks found :  27
-- Found  309  Single Hits!
Found  3  cathode crossing tracks 
file closed!
**************
* Reco total *
**************
* Nb of Hits 1589532
* Nb of 2D Tracks 16301
* Nb of 3D Tracks 1186
* Nb of Ghosts 0
* Nb of Single Hits 6229
* Nb of PDS Peaks 0
* Nb of PDS Clusters 0
**************
it took 1098.01 s to run 19 events (average of 57.79 per event)
Select jobs to execute...
Select jobs to execute...
Complete log: .snakemake/log/2026-02-04T044858.786862.snakemake.log
Done running snakemake
justIN time: 2026-02-04 05:57:55 UTC       justIN version: 01.06.00