justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303843.4@dunegpschedd01.fnal.gov

Jobsub ID303843.4@dunegpschedd01.fnal.gov
Workflow ID12725
Stage ID1
User namecalcuttj@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-02-04 03:46:37
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce508
Last heartbeat2026-02-04 04:08:59
From worker nodeHostnameb9p35p0043.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 03:47:30
Input filesvd-protodune:np02vd_raw_run040380_0011_df-s04-d3_dw_0_20251105T180056.hdf5
JobscriptExit code0
Real time21m (1275s)
CPU time18m (1115s = 87%)
Max RSS bytes2465566720 (2351 MiB)
Outputting started2026-02-04 04:08:45
Output files
Finished2026-02-04 04:08:59
Saved logsjustin-logs:303843.4-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

for cathode stitcher, do not do anything 
Found  5  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  14
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  14  TRIGGER  1031 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:01:02 2025  +  687304448.0  ns 
Trigger timestamp  1762365662.6873045

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762365662.6860526
number of samples =  7872
----- Number Of Hits found :  [1971 1773 1512]
---- Number Of 2D tracks found :  [ 9 10  6]
--- Number of 3D tracks found :  3
-- Found  58  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762365662.6860526
number of samples =  7872
----- Number Of Hits found :  [2121 1732 1076]
---- Number Of 2D tracks found :  [19 21 18]
--- Number of 3D tracks found :  7
-- Found  115  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762365662.6860478
number of samples =  8064
----- Number Of Hits found :  [ 6442 10596 11822]
---- Number Of 2D tracks found :  [ 93 122 188]
--- Number of 3D tracks found :  10
-- Found  211  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762365662.6860478
number of samples =  8064
----- Number Of Hits found :  [ 8970 14443 16602]
---- Number Of 2D tracks found :  [211 256 400]
--- Number of 3D tracks found :  22
-- Found  289  Single Hits!
Found  1  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  15
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  15  TRIGGER  1035 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:01:02 2025  +  699495424.0  ns 
Trigger timestamp  1762365662.6994956

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762365662.6982422
number of samples =  8960
----- Number Of Hits found :  [3675 2852 2500]
---- Number Of 2D tracks found :  [14 13 14]
--- Number of 3D tracks found :  6
-- Found  93  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762365662.6982422
number of samples =  8960
----- Number Of Hits found :  [2791 2811 1886]
---- Number Of 2D tracks found :  [28 28 33]
--- Number of 3D tracks found :  8
-- Found  180  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762365662.69824
number of samples =  9216
----- Number Of Hits found :  [ 7174 11199 11522]
---- Number Of 2D tracks found :  [131 131 184]
--- Number of 3D tracks found :  15
-- Found  282  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762365662.69824
number of samples =  9216
----- Number Of Hits found :  [ 9303 14928 15648]
---- Number Of 2D tracks found :  [248 291 364]
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
--- Number of 3D tracks found :  65
-- Found  383  Single Hits!
Found  3  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  16
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  16  TRIGGER  1039 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:01:02 2025  +  714839040.0  ns 
Trigger timestamp  1762365662.7148392

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762365662.7135775
number of samples =  13120
----- Number Of Hits found :  [4362 4044 3385]
---- Number Of 2D tracks found :  [30 27 36]
--- Number of 3D tracks found :  6
-- Found  117  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762365662.7135775
number of samples =  13120
----- Number Of Hits found :  [3649 3455 2434]
---- Number Of 2D tracks found :  [43 46 51]
--- Number of 3D tracks found :  11
-- Found  196  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762365662.713568
number of samples =  13440
----- Number Of Hits found :  [10996 17951 19093]
---- Number Of 2D tracks found :  [197 215 285]
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation/local_software/lardon/src/lardon/track_2d.py:168: RuntimeWarning: divide by zero encountered in divide
  slope_start = 1. / slope_start
/local_software/lardon/src/lardon/track_2d.py:169: RuntimeWarning: divide by zero encountered in divide
  slope_end = 1. / slope_end
/local_software/lardon/src/lardon/track_2d.py:146: UserWarning: 
The maximal number of iterations maxit (set to 20 by the program)
allowed for finding a smoothing spline with fp=s has been reached: s
too small.
There is an approximation returned but the corresponding weighted sum
of squared residuals does not satisfy the condition abs(fp-s)/s < tol.
  spline_x = UnivariateSpline(x_r_unique, z_r_ux)
/local_software/lardon/src/lardon/track_2d.py:146: UserWarning: 
The maximal number of iterations maxit (set to 20 by the program)
allowed for finding a smoothing spline with fp=s has been reached: s
too small.
There is an approximation returned but the corresponding weighted sum
of squared residuals does not satisfy the condition abs(fp-s)/s < tol.
  spline_x = UnivariateSpline(x_r_unique, z_r_ux)
/local_software/lardon/src/lardon/track_2d.py:168: RuntimeWarning: divide by zero encountered in divide
  slope_start = 1. / slope_start
/local_software/lardon/src/lardon/track_2d.py:169: RuntimeWarning: divide by zero encountered in divide
  slope_end = 1. / slope_end

--- Number of 3D tracks found :  52
-- Found  287  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762365662.713568
number of samples =  13440
----- Number Of Hits found :  [12100 18649 21850]
---- Number Of 2D tracks found :  [361 379 556]
--- Number of 3D tracks found :  78
-- Found  395  Single Hits!
Across modules  [0, 1] merged  1  3D tracks together across module !!! 
Across modules  [2, 3] merged  1  3D tracks together across module !!! 
Found  4  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  17
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  17  TRIGGER  1043 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:01:02 2025  +  734087936.0  ns 
Trigger timestamp  1762365662.734088

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762365662.7328124
number of samples =  9792
----- Number Of Hits found :  [3889 3641 3250]
---- Number Of 2D tracks found :  [25 28 26]
--- Number of 3D tracks found :  9
-- Found  94  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762365662.7328124
number of samples =  9792
----- Number Of Hits found :  [2515 2622 2059]
---- Number Of 2D tracks found :  [40 39 39]
--- Number of 3D tracks found :  14
-- Found  182  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762365662.732832
number of samples =  9984
----- Number Of Hits found :  [13502 22759 24054]
---- Number Of 2D tracks found :  [189 230 312]
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
--- Number of 3D tracks found :  25
-- Found  291  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762365662.732832
number of samples =  9984
----- Number Of Hits found :  [13261 22388 23105]
---- Number Of 2D tracks found :  [362 493 603]
--- Number of 3D tracks found :  30
-- Found  382  Single Hits!
Found  0  cathode crossing tracks 
-*-*-*-*-*-*-*-*-*-*-
 READING EVENT  18
-*-*-*-*-*-*-*-*-*-*-
RUN  40380  of  bot  EVENT  18  TRIGGER  1047 : kADCSimpleWindow ( 6 )
Taken at  Wed Nov  5 19:01:02 2025  +  754601984.0  ns 
Trigger timestamp  1762365662.7546022

## Reading TPC ##

MODULE  0
 Removing  12  broken channels
1st sample timestamp:  1762365662.7533252
number of samples =  17600
----- Number Of Hits found :  [5094 5193 3634]
---- Number Of 2D tracks found :  [27 26 33]
--- Number of 3D tracks found :  10
-- Found  23  Single Hits!

MODULE  1
 Removing  12  broken channels
1st sample timestamp:  1762365662.7533252
number of samples =  17600
----- Number Of Hits found :  [6182 6284 4934]
---- Number Of 2D tracks found :  [53 57 72]
--- Number of 3D tracks found :  19
-- Found  41  Single Hits!

MODULE  2
 Removing  12  broken channels
1st sample timestamp:  1762365662.753344
number of samples =  17984
----- Number Of Hits found :  [22176 35248 38672]
---- Number Of 2D tracks found :  [373 457 599]
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
3D finalizing track: problem with interpolation
not enought point to spline the complete track
--- Number of 3D tracks found :  242
-- Found  104  Single Hits!

MODULE  3
 Removing  12  broken channels
1st sample timestamp:  1762365662.753344
number of samples =  17984
----- Number Of Hits found :  [18473 30214 31870]
---- Number Of 2D tracks found :  [605 762 977]
--- Number of 3D tracks found :  251
-- Found  218  Single Hits!
WHAAAAT ? too many possibility for cathode stitcher, do not do anything 
Found  4  cathode crossing tracks 
file closed!
**************
* Reco total *
**************
* Nb of Hits 1589815
* Nb of 2D Tracks 16756
* Nb of 3D Tracks 1425
* Nb of Ghosts 0
* Nb of Single Hits 5444
* Nb of PDS Peaks 0
* Nb of PDS Clusters 0
**************
it took 1196.56 s to run 19 events (average of 62.98 per event)
Select jobs to execute...
Select jobs to execute...
Complete log: .snakemake/log/2026-02-04T044821.140391.snakemake.log
Done running snakemake
justIN time: 2026-02-04 06:00:27 UTC       justIN version: 01.06.00