justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 259010.14@dunegpschedd01.fnal.gov

Jobsub ID259010.14@dunegpschedd01.fnal.gov
Workflow ID10823
Stage ID1
User namecalcuttj@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes6290407424 (5999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 20:51:00
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2025-11-27 04:36:48
From worker nodeHostnamewn-la-13.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes7864320000 (7500 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-27 02:05:52
Input fileshd-protodune:np04hd_raw_run028799_0000_dataflow6_datawriter_0_20240821T040553.hdf5
JobscriptExit code0
Real time2h (8884s)
CPU time1h (6277s = 70%)
Max RSS bytes5471559680 (5218 MiB)
Outputting started2025-11-27 04:33:57
Output filesusertests:pdhd_prod_data__259010_14_1_20251127T020601Z.root
Finished2025-11-27 04:36:48
Saved logsjustin-logs:259010.14-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

desired window for channel 8685
wclsFrameSaver: no samples within desired window for channel 8685
wclsFrameSaver: no samples within desired window for channel 8686
wclsFrameSaver: no samples within desired window for channel 8686
wclsFrameSaver: no samples within desired window for channel 8687
wclsFrameSaver: no samples within desired window for channel 8687
wclsFrameSaver: no samples within desired window for channel 8688
wclsFrameSaver: no samples within desired window for channel 8688
wclsFrameSaver: no samples within desired window for channel 8689
wclsFrameSaver: no samples within desired window for channel 8690
wclsFrameSaver: no samples within desired window for channel 9209
wclsFrameSaver: no samples within desired window for channel 9561
wclsFrameSaver: no samples within desired window for channel 9581
wclsFrameSaver: no samples within desired window for channel 9582
wclsFrameSaver: no samples within desired window for channel 9582
wclsFrameSaver: no samples within desired window for channel 9583
wclsFrameSaver: no samples within desired window for channel 9584
wclsFrameSaver: no samples within desired window for channel 9585
wclsFrameSaver: no samples within desired window for channel 9585
wclsFrameSaver: no samples within desired window for channel 9586
wclsFrameSaver: no samples within desired window for channel 9586
wclsFrameSaver: no samples within desired window for channel 9587
wclsFrameSaver: no samples within desired window for channel 9587
wclsFrameSaver: no samples within desired window for channel 9588
wclsFrameSaver: no samples within desired window for channel 9589
wclsFrameSaver: no samples within desired window for channel 9589
wclsFrameSaver: no samples within desired window for channel 9590
wclsFrameSaver: no samples within desired window for channel 9590
wclsFrameSaver: no samples within desired window for channel 9591
wclsFrameSaver: no samples within desired window for channel 9591
wclsFrameSaver: no samples within desired window for channel 9591
wclsFrameSaver: no samples within desired window for channel 9592
wclsFrameSaver: no samples within desired window for channel 9592
wclsFrameSaver: no samples within desired window for channel 9592
wclsFrameSaver: no samples within desired window for channel 9593
wclsFrameSaver: no samples within desired window for channel 9593
wclsFrameSaver: no samples within desired window for channel 9594
wclsFrameSaver: no samples within desired window for channel 9594
wclsFrameSaver: no samples within desired window for channel 9594
wclsFrameSaver: no samples within desired window for channel 9595
wclsFrameSaver: no samples within desired window for channel 9595
wclsFrameSaver: no samples within desired window for channel 9596
wclsFrameSaver: no samples within desired window for channel 9596
wclsFrameSaver: no samples within desired window for channel 9597
wclsFrameSaver: no samples within desired window for channel 9598
wclsFrameSaver: no samples within desired window for channel 9599
wclsFrameSaver: no samples within desired window for channel 9599
wclsFrameSaver: no samples within desired window for channel 9600
FrameSaver: q=1.12527e+06 n=773192 tag=wiener
3198 hits in TPC 0
7811 hits in TPC 1
5501 hits in TPC 2
3690 hits in TPC 3
2768 hits in TPC 4
3881 hits in TPC 5
4475 hits in TPC 6
2561 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 198 XUs and 128 XVs -> 0 XUVs
C:0 T:1 58397 XUs and 51498 XVs -> 1568 XUVs
C:0 T:2 19304 XUs and 17235 XVs -> 1883 XUVs
C:0 T:3 80 XUs and 98 XVs -> 2 XUVs
C:0 T:4 66 XUs and 144 XVs -> 7 XUVs
C:0 T:5 16652 XUs and 20651 XVs -> 3939 XUVs
C:0 T:6 13802 XUs and 16248 XVs -> 747 XUVs
C:0 T:7 64 XUs and 96 XVs -> 2 XUVs
8148 XUVs total
1898 collection wire objects
8148 potential space points
Neighbour search...
946642 tests to find 521750 neighbours
Iterating with no regularization...
Begin: 9.43907e+06
0 8.62187e+06
1 8.55281e+06
2 8.54385e+06
3 8.54144e+06
Now with regularization...
Begin: 7.93368e+06
0 7.93118e+06
TestBeamParticleCreationAlgorithm::SetupTestBeamVertex - Test beam particle has no initial vertex
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
27-Nov-2025 05:33:55 CET  Closed output file "pdhd_prod_data__259010_14_1_20251127T020601Z.root"
27-Nov-2025 05:33:55 CET  Closed input file "temp.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              91.0348       162.829       272.996       149.374       42.4342        30     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000439327   0.00286256    0.00654115    0.00238796    0.00175185       30     
produce:fembfilter:PDHDFEMBFilter                      0.776793       0.80298       0.94354      0.798425      0.0279333       30     
produce:wclsdatahd:WireCellToolkit                      59.6519       72.1111       77.9692       73.902        5.06956        30     
produce:gaushit:GausHitFinder                          0.395413      0.795363       1.29005      0.718896      0.214632        30     
produce:nhitsfilter:NumberOfHitsFilter                0.000241581   0.000443256   0.000805484   0.000392006   0.000175956      30     
produce:reco3d:SpacePointSolver                        0.141294      0.998566       3.20703       0.72285      0.755115        30     
produce:hitpdune:DisambigFromSpacePoints               0.151223       0.60623       1.36977      0.538724      0.310912        30     
produce:pandora:StandardPandora                         5.75768       19.2982       49.2381       15.9594       10.8797        30     
produce:pandoraWriter:StandardPandora                  0.0987775     0.193079      0.321491      0.186678      0.0510736       30     
produce:pandoraTrack:LArPandoraTrackCreation           0.649211       4.66747       11.7903       3.86145       2.78443        30     
produce:pandoraShower:LArPandoraShowerCreation         0.578196       5.84613       20.6778       4.8114         4.492         30     
produce:pandoracalo:Calorimetry                        0.299935       1.64862       5.33578       1.34467       1.00005        30     
produce:pandoracalonosce:Calorimetry                   0.266209       1.54308       4.15747       1.2794       0.903335        30     
produce:pandorapid:Chi2ParticleID                     0.00122612    0.00472833     0.0441969    0.00301925    0.00744523       30     
produce:pandoraShowercalo:ShowerCalorimetry            0.421682       4.17272       13.168        3.30098       2.96728        30     
produce:pandoraShowercalonosce:ShowerCalorimetry       0.409086       4.07142       12.5492       3.34049       2.78983        30     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.0757467     0.163824      0.254711       0.16464      0.0454562       30     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0329612     0.067993      0.110639      0.067223      0.0195387       30     
produce:emtrkmichelid:EmTrackMichelId                   12.7373       23.3241       36.1558       22.0727       5.8447         30     
[art]:TriggerResults:TriggerResultInserter            2.5999e-05    5.68431e-05   8.8492e-05    5.65835e-05   1.45917e-05      30     
end_path:out1:RootOutput                               6.53e-06     1.2073e-05    3.1938e-05    1.0581e-05    5.37848e-06      30     
end_path:michelremoving:michelremoving                 0.0234708      0.19624      0.725264      0.138979      0.164139        30     
end_path:michelremovinggnocchi:michelremoving          0.0204865     0.193208      0.769174      0.142011      0.165671        30     
end_path:calibanagnocchi:CalibAnaTree                   5.54619       20.2861       44.4333       17.4282       10.5574        30     
end_path:out1:RootOutput(write)                        0.501553      0.980543       1.52663      0.906459      0.253542        30     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8347.58 MB
  Peak resident set size usage (VmHWM): 5471.56 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
stage2 lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 5393232
-rw-r--r--. 1 dune009 dune    6718014 Nov 27 05:33 jobscript.log
-rw-r--r--. 1 dune009 dune        163 Nov 27 05:33 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune   98285513 Nov 27 05:33 temp_20251127T043356_keepup_hists.root
-rw-r--r--. 1 dune009 dune       3574 Nov 27 05:33 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune     385024 Nov 27 05:33 mem.db
-rw-r--r--. 1 dune009 dune      61440 Nov 27 05:33 time.db
-rw-r--r--. 1 dune009 dune  796948544 Nov 27 05:33 pdhd_prod_data__259010_14_1_20251127T020601Z.root
-rw-r--r--. 1 dune009 dune  123445250 Nov 27 05:32 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune          0 Nov 27 04:11 debugprod.log
-rw-r--r--. 1 dune009 dune  925946155 Nov 27 04:11 pdhd_keepup_decoder.root
-rw-r--r--. 1 dune009 dune 3569310899 Nov 27 04:11 temp.root
-rw-r--r--. 1 dune009 dune         83 Nov 27 03:06 all-input-dids.txt
justIN time: 2026-02-08 05:42:20 UTC       justIN version: 01.06.00