Jobsub ID 29295.171@dunegpschedd01.fnal.gov

Jobsub ID	29295.171@dunegpschedd01.fnal.gov
Workflow ID	1657
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-19 23:46:33
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod00
Last heartbeat	2025-08-19 23:51:15
From worker node	Hostname	wg00.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-19 23:48:11
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_716_20231207T150857Z_gen_g4_detsim_hitreco__20240510T062229Z_reco2.root
Jobscript	Exit code	0
	Real time	2m (157s)
	CPU time	1m (70s = 44%)
	Max RSS bytes	1032966144 (985 MiB)
Outputting started	2025-08-19 23:50:49
Output files
Finished	2025-08-19 23:51:15
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

:49:45 BST
Begin processing the 30th record. run: 6457551 subRun: 1 event: 71630 at 20-Aug-2025 00:49:46 BST
Begin processing the 31st record. run: 6457551 subRun: 1 event: 71631 at 20-Aug-2025 00:49:46 BST
Begin processing the 32nd record. run: 6457551 subRun: 1 event: 71632 at 20-Aug-2025 00:49:46 BST
Begin processing the 33rd record. run: 6457551 subRun: 1 event: 71633 at 20-Aug-2025 00:49:47 BST
Begin processing the 34th record. run: 6457551 subRun: 1 event: 71634 at 20-Aug-2025 00:49:51 BST
Begin processing the 35th record. run: 6457551 subRun: 1 event: 71635 at 20-Aug-2025 00:49:52 BST
Begin processing the 36th record. run: 6457551 subRun: 1 event: 71636 at 20-Aug-2025 00:49:52 BST
Begin processing the 37th record. run: 6457551 subRun: 1 event: 71637 at 20-Aug-2025 00:49:53 BST
Begin processing the 38th record. run: 6457551 subRun: 1 event: 71638 at 20-Aug-2025 00:49:53 BST
Begin processing the 39th record. run: 6457551 subRun: 1 event: 71639 at 20-Aug-2025 00:49:59 BST
Begin processing the 40th record. run: 6457551 subRun: 1 event: 71640 at 20-Aug-2025 00:50:00 BST
Begin processing the 41st record. run: 6457551 subRun: 1 event: 71641 at 20-Aug-2025 00:50:00 BST
Begin processing the 42nd record. run: 6457551 subRun: 1 event: 71642 at 20-Aug-2025 00:50:00 BST
Begin processing the 43rd record. run: 6457551 subRun: 1 event: 71643 at 20-Aug-2025 00:50:01 BST
Begin processing the 44th record. run: 6457551 subRun: 1 event: 71644 at 20-Aug-2025 00:50:01 BST
Begin processing the 45th record. run: 6457551 subRun: 1 event: 71645 at 20-Aug-2025 00:50:04 BST
Begin processing the 46th record. run: 6457551 subRun: 1 event: 71646 at 20-Aug-2025 00:50:05 BST
Begin processing the 47th record. run: 6457551 subRun: 1 event: 71647 at 20-Aug-2025 00:50:05 BST
Begin processing the 48th record. run: 6457551 subRun: 1 event: 71648 at 20-Aug-2025 00:50:05 BST
Begin processing the 49th record. run: 6457551 subRun: 1 event: 71649 at 20-Aug-2025 00:50:06 BST
Begin processing the 50th record. run: 6457551 subRun: 1 event: 71650 at 20-Aug-2025 00:50:06 BST
Begin processing the 51st record. run: 6457551 subRun: 1 event: 71651 at 20-Aug-2025 00:50:07 BST
Begin processing the 52nd record. run: 6457551 subRun: 1 event: 71652 at 20-Aug-2025 00:50:07 BST
Begin processing the 53rd record. run: 6457551 subRun: 1 event: 71653 at 20-Aug-2025 00:50:08 BST
Begin processing the 54th record. run: 6457551 subRun: 1 event: 71654 at 20-Aug-2025 00:50:08 BST
Begin processing the 55th record. run: 6457551 subRun: 1 event: 71655 at 20-Aug-2025 00:50:08 BST
Begin processing the 56th record. run: 6457551 subRun: 1 event: 71656 at 20-Aug-2025 00:50:09 BST
Begin processing the 57th record. run: 6457551 subRun: 1 event: 71657 at 20-Aug-2025 00:50:10 BST
Begin processing the 58th record. run: 6457551 subRun: 1 event: 71658 at 20-Aug-2025 00:50:10 BST
Begin processing the 59th record. run: 6457551 subRun: 1 event: 71659 at 20-Aug-2025 00:50:10 BST
Begin processing the 60th record. run: 6457551 subRun: 1 event: 71660 at 20-Aug-2025 00:50:12 BST
Begin processing the 61st record. run: 6457551 subRun: 1 event: 71661 at 20-Aug-2025 00:50:16 BST
Begin processing the 62nd record. run: 6457551 subRun: 1 event: 71662 at 20-Aug-2025 00:50:17 BST
Begin processing the 63rd record. run: 6457551 subRun: 1 event: 71663 at 20-Aug-2025 00:50:20 BST
Begin processing the 64th record. run: 6457551 subRun: 1 event: 71664 at 20-Aug-2025 00:50:21 BST
Begin processing the 65th record. run: 6457551 subRun: 1 event: 71665 at 20-Aug-2025 00:50:21 BST
Begin processing the 66th record. run: 6457551 subRun: 1 event: 71666 at 20-Aug-2025 00:50:22 BST
Begin processing the 67th record. run: 6457551 subRun: 1 event: 71667 at 20-Aug-2025 00:50:22 BST
Begin processing the 68th record. run: 6457551 subRun: 1 event: 71668 at 20-Aug-2025 00:50:23 BST
Begin processing the 69th record. run: 6457551 subRun: 1 event: 71669 at 20-Aug-2025 00:50:23 BST
Begin processing the 70th record. run: 6457551 subRun: 1 event: 71670 at 20-Aug-2025 00:50:24 BST
Begin processing the 71st record. run: 6457551 subRun: 1 event: 71671 at 20-Aug-2025 00:50:25 BST
Begin processing the 72nd record. run: 6457551 subRun: 1 event: 71672 at 20-Aug-2025 00:50:25 BST
Begin processing the 73rd record. run: 6457551 subRun: 1 event: 71673 at 20-Aug-2025 00:50:25 BST
Begin processing the 74th record. run: 6457551 subRun: 1 event: 71674 at 20-Aug-2025 00:50:29 BST
Begin processing the 75th record. run: 6457551 subRun: 1 event: 71675 at 20-Aug-2025 00:50:30 BST
Begin processing the 76th record. run: 6457551 subRun: 1 event: 71676 at 20-Aug-2025 00:50:34 BST
Begin processing the 77th record. run: 6457551 subRun: 1 event: 71677 at 20-Aug-2025 00:50:35 BST
Begin processing the 78th record. run: 6457551 subRun: 1 event: 71678 at 20-Aug-2025 00:50:35 BST
Begin processing the 79th record. run: 6457551 subRun: 1 event: 71679 at 20-Aug-2025 00:50:37 BST
Begin processing the 80th record. run: 6457551 subRun: 1 event: 71680 at 20-Aug-2025 00:50:38 BST
Begin processing the 81st record. run: 6457551 subRun: 1 event: 71681 at 20-Aug-2025 00:50:39 BST
Begin processing the 82nd record. run: 6457551 subRun: 1 event: 71682 at 20-Aug-2025 00:50:40 BST
Begin processing the 83rd record. run: 6457551 subRun: 1 event: 71683 at 20-Aug-2025 00:50:40 BST
Begin processing the 84th record. run: 6457551 subRun: 1 event: 71684 at 20-Aug-2025 00:50:40 BST
Begin processing the 85th record. run: 6457551 subRun: 1 event: 71685 at 20-Aug-2025 00:50:41 BST
Begin processing the 86th record. run: 6457551 subRun: 1 event: 71686 at 20-Aug-2025 00:50:41 BST
Begin processing the 87th record. run: 6457551 subRun: 1 event: 71687 at 20-Aug-2025 00:50:42 BST
Begin processing the 88th record. run: 6457551 subRun: 1 event: 71688 at 20-Aug-2025 00:50:42 BST
Begin processing the 89th record. run: 6457551 subRun: 1 event: 71689 at 20-Aug-2025 00:50:43 BST
Begin processing the 90th record. run: 6457551 subRun: 1 event: 71690 at 20-Aug-2025 00:50:43 BST
Begin processing the 91st record. run: 6457551 subRun: 1 event: 71691 at 20-Aug-2025 00:50:44 BST
Begin processing the 92nd record. run: 6457551 subRun: 1 event: 71692 at 20-Aug-2025 00:50:44 BST
Begin processing the 93rd record. run: 6457551 subRun: 1 event: 71693 at 20-Aug-2025 00:50:45 BST
Begin processing the 94th record. run: 6457551 subRun: 1 event: 71694 at 20-Aug-2025 00:50:45 BST
Begin processing the 95th record. run: 6457551 subRun: 1 event: 71695 at 20-Aug-2025 00:50:45 BST
Begin processing the 96th record. run: 6457551 subRun: 1 event: 71696 at 20-Aug-2025 00:50:46 BST
Begin processing the 97th record. run: 6457551 subRun: 1 event: 71697 at 20-Aug-2025 00:50:46 BST
Begin processing the 98th record. run: 6457551 subRun: 1 event: 71698 at 20-Aug-2025 00:50:47 BST
Begin processing the 99th record. run: 6457551 subRun: 1 event: 71699 at 20-Aug-2025 00:50:47 BST
Begin processing the 100th record. run: 6457551 subRun: 1 event: 71700 at 20-Aug-2025 00:50:47 BST
20-Aug-2025 00:50:47 BST  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/12/2d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_716_20231207T150857Z_gen_g4_detsim_hitreco__20240510T062229Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      0.10481      0.476661       3.97668      0.279886      0.540886        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00405225    0.00861556     0.0352511    0.00748769     0.0056594       100    
prod:pandorapid:Chi2ParticleID                 0.0185246     0.0353291      0.24203      0.0294973     0.024367        100    
prod:energyreconue:EnergyReco                  0.0534887     0.101153      0.425193      0.0779638     0.0636829       100    
prod:energyreconumu:EnergyReco                0.00189486     0.0595326      1.30994      0.0269591     0.140858        100    
prod:energyreconc:EnergyReco                  0.00664335     0.0416119     0.305894      0.0320384     0.0441027       100    
[art]:TriggerResults:TriggerResultInserter    1.8591e-05    2.66881e-05    9.164e-05    2.5096e-05    8.18494e-06      100    
end_path:ana:PIDASelectionAna                  0.0113222      0.22987       1.84711      0.109685      0.328064        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5436.87 MB
  Peak resident set size usage (VmHWM): 1032.97 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 60
-rw-r--r--. 1 job0003 job0003 20607 Aug 20 00:50 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_716_20231207T150857Z_gen_g4_detsim_hitreco__20240510T062229Z_reco2_pida_2025-08-19T_234824Z.root
-rw-r--r--. 1 job0003 job0003 14157 Aug 20 00:50 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_716_20231207T150857Z_gen_g4_detsim_hitreco__20240510T062229Z_reco2_reco_2025-08-19T_234824Z.log
-rw-r--r--. 1 job0003 job0003 11922 Aug 20 00:50 jobscript.log
-rw-r--r--. 1 job0003 job0003   200 Aug 20 00:50 justin-processed-pfns.txt
-rw-r--r--. 1 job0003 job0003   137 Aug 20 00:48 all-input-dids.txt
-rw-r--r--. 1 job0003 job0003     0 Aug 20 00:48 debugprod.log