Jobsub ID 48443.13@dunegpschedd01.fnal.gov
| Jobsub ID | 48443.13@dunegpschedd01.fnal.gov | |
| Workflow ID | 2501 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-09-18 11:50:24 | |
| Site | US_UCSD | |
| Entry | CMSHTPC_T2_US_UCSD_gw6 | |
| Last heartbeat | 2025-09-18 13:17:25 | |
| From worker node | Hostname | mh-7662-12.t2.ucsd.edu |
| cpuinfo | AMD EPYC 7662 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | outputting_failed | |
| Started | 2025-09-18 11:50:59 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_299_20231207T072019Z_gen_g4_detsim_hitreco__20240510T063126Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_395_20231120T135034Z_gen_g4_detsim_hitreco__20240503T061452Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 1h (5091s) | |
| CPU time | 46m (2772s = 54%) | |
| Max RSS bytes | 2164895744 (2064 MiB) | |
| Outputting started | 2025-09-18 13:15:51 | |
| Output files | ||
| Finished | 2025-09-18 13:17:25 | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
==================================== Full event 3.34951 14.2981 1061.92 6.9325 69.4466 300 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.149907 0.239822 0.339762 0.180546 0.0763836 300 reco:pandora:StandardPandora 0.905076 8.82062 1016.98 2.88714 60.8757 300 reco:pandoraTrack:LArPandoraTrackCreation 0.000156114 0.0142108 0.812764 0.00183119 0.0653613 300 reco:pandoraShower:LArPandoraModularShowerCreation 0.000197784 0.020092 0.895644 0.00367254 0.0697425 300 reco:pandoracalo:Calorimetry 0.000116703 0.0549583 3.73419 0.00132435 0.343557 300 reco:pandorapid:Chi2ParticleID 4.3961e-05 0.000258576 0.00221056 0.000193549 0.00026241 300 reco:cvnmap:CVNMapper 2.3561e-05 0.0227244 0.0792333 0.0196779 0.0194261 300 reco:cvneva:CVNEvaluator 1.947e-05 0.444931 2.74718 0.589041 0.322574 300 reco:energyrecnumu:EnergyReco 0.000379319 0.323748 8.37201 0.025691 0.777389 300 reco:energyrecnue:EnergyReco 0.000169384 0.00494971 0.281846 0.000813873 0.0216858 300 reco:energyrecnc:EnergyReco 0.00254054 1.39841 277.921 0.622235 15.9955 300 reco:energyrecnumurange:EnergyReco 0.000323267 0.115767 10.0387 0.00424678 0.780045 300 reco:energyrecnumumcs:EnergyReco 0.000818848 0.112698 9.68654 0.00494089 0.751339 300 reco:energyrecnumumcsllhd:EnergyReco 0.000624714 0.111126 8.91479 0.00496371 0.720507 300 reco:opdec:Deconvolution 0.454187 0.802196 1.42193 0.828729 0.172983 300 reco:ophitspe:OpHitFinderDeco 1.341 1.37865 1.43437 1.37577 0.0271635 300 reco:opflash:OpFlashFinder 0.000395778 0.00194302 0.00709923 0.00172711 0.00108896 300 reco:opslicer:OpSlicer 6.4792e-05 0.364885 1.61862 0.273554 0.310811 300 reco:rns:RandomNumberSaver 2.6991e-05 6.25062e-05 0.00042153 5.3181e-05 3.60055e-05 300 reco:anglereconue:NuAngularReco 0.000193714 0.00356468 0.234076 0.000511811 0.0176461 300 reco:anglereconumu:NuAngularReco 9.4722e-05 0.00325021 0.22507 0.000279581 0.0169529 300 reco:anglereconuepfps:NuAngularReco 0.000146003 0.00337884 0.224595 0.00041823 0.0171826 300 reco:anglereconumupfps:NuAngularReco 0.000127813 0.00337298 0.228225 0.000372694 0.0173912 300 reco:anglerecohits:NuAngularReco 0.000140733 0.00445232 0.241106 0.000984522 0.018871 300 [art]:TriggerResults:TriggerResultInserter 1.11e-05 1.66508e-05 0.000120152 1.569e-05 7.35926e-06 300 end_path:cafmaker:CAFMaker 0.00158746 0.0460666 2.49444 0.00789629 0.199754 300 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 3829.04 MB Peak resident set size usage (VmHWM): 2164.9 MB ==================================================================================================== lar exit code 0 === Start last 100 lines of lar log file === Boundary wire vector sizes: 215, 114, 145 minwire 0: 1492 minwire 1: 1315 minwire 2: 1441 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.718616, 0.250489, 0.0308942, Output 1: 0.0191696, 0.916045, 0.0627488, 0.00203639, Output 2: 0.0529003, 0.786458, 0.156396, 0.00424595, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1714! Begin processing the 295th record. run: 6207207 subRun: 1 event: 79195 at 18-Sep-2025 06:14:53 PDT Running Ophitfinder with InputDigiType = 'recob' Found hits: 464! Begin processing the 296th record. run: 6207207 subRun: 1 event: 79196 at 18-Sep-2025 06:15:00 PDT Boundary wire vector sizes: 88, 83, 78 minwire 0: 732 minwire 1: 2740 minwire 2: 182 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.000548996, 0.995957, 0.00349417, Output 1: 0.00381169, 0.989125, 0.00687106, 0.000192431, Output 2: 0.997962, 0.00197595, 6.01728e-05, 2.10621e-06, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1136! Begin processing the 297th record. run: 6207207 subRun: 1 event: 79197 at 18-Sep-2025 06:15:08 PDT Boundary wire vector sizes: 1324, 1042, 954 minwire 0: 119 minwire 1: 2182 minwire 2: 68 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.000962519, 0.9943, 0.00473702, Output 1: 0.0343168, 0.340229, 0.547856, 0.0775987, Output 2: 0.883722, 0.10761, 0.00830877, 0.00035906, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1682! Begin processing the 298th record. run: 6207207 subRun: 1 event: 79198 at 18-Sep-2025 06:15:19 PDT Boundary wire vector sizes: 808, 844, 518 minwire 0: 2516 minwire 1: 627 minwire 2: 2336 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.000511167, 0.992373, 0.00711578, Output 1: 0.411639, 0.56694, 0.0201468, 0.0012741, Output 2: 0.989243, 0.0103754, 0.000358485, 2.3049e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 2113! Begin processing the 299th record. run: 6207207 subRun: 1 event: 79199 at 18-Sep-2025 06:15:30 PDT Boundary wire vector sizes: 45, 20, 38 minwire 0: 1554 minwire 1: 1822 minwire 2: 1280 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.00388248, 0.00248545, 0.993632, Output 1: 0.01353, 0.981026, 0.00529644, 0.000147632, Output 2: 0.998059, 0.00187939, 5.93671e-05, 2.10608e-06, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1522! Begin processing the 300th record. run: 6207207 subRun: 1 event: 79200 at 18-Sep-2025 06:15:39 PDT Boundary wire vector sizes: 735, 1112, 842 minwire 0: 1815 minwire 1: 520 minwire 2: 2358 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.00812495, 0.877228, 0.114647, Output 1: 0.0334888, 0.895007, 0.0693611, 0.00214354, Output 2: 0.846096, 0.149468, 0.0043513, 8.51324e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1957! Art has completed and will exit with status 0. === End last 100 lines of lar log file === processed files root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1e/68/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_299_20231207T072019Z_gen_g4_detsim_hitreco__20240510T063126Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1b/70/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_395_20231120T135034Z_gen_g4_detsim_hitreco__20240503T061452Z_reco2.root .: total 17988 -rw-r--r-- 1 cuser cuser 6801556 Sep 18 06:15 flatcaf.root -rw-r--r-- 1 cuser cuser 5623867 Sep 18 06:15 caf.root -rw-r--r-- 1 cuser cuser 5623867 Sep 18 06:15 caf_fd_hd_atmo_2501_20250918T115108Z.root -rw-r--r-- 1 cuser cuser 187352 Sep 18 06:15 caf_20250918T115108Z.log -rw-r--r-- 1 cuser cuser 73482 Sep 18 06:15 jobscript.log -rw-r--r-- 1 cuser cuser 372 Sep 18 06:15 caf_20250918T115108Z.file -rw-r--r-- 1 cuser cuser 372 Sep 18 06:15 caf_20250918T115108Z.pfns -rw-r--r-- 1 cuser cuser 372 Sep 18 04:51 file.list -rw-r--r-- 1 cuser cuser 372 Sep 18 06:15 justin-processed-pfns.txt -rw-r--r-- 1 cuser cuser 274 Sep 18 04:51 all-input-dids.txt -rw-r--r-- 1 cuser cuser 274 Sep 18 06:15 caf_20250918T115108Z.did -rw-r--r-- 1 cuser cuser 274 Sep 18 04:51 did.list -rw-r--r-- 1 cuser cuser 0 Sep 18 04:51 debugprod.log