nl.nikhef.pdp.lrmsutils/doc/lcg-info-dynamic-scheduler.txt

FOR THE IMPATIENT

Just installing the RPM should result in a working system.

You may want to look at the "vomap" section below if your
unix group names are not the same as your VO names.

OK THE REAL DOCS

This is an implementation of per-VO estimated response times and free
slots for the LCG 'generic information provider' framework.

HOW IT IS SUPPOSED TO WORK

There are two parts to the system.

One part (lcg-dynamic-info-scheduler program) contains the algorithm
that computes the response times.  This part doesn't know the details
of the underlying batch system, so that the estimated times are as
independent as possible from the various LRMS.

The second part is the LRMS specific part.  This gathers information
from the LRMS and writes it out in a LRMS-independent format.  There
are two of these critters; one for the LRMS state, and one for the
scheduling policy.

A typical deployment will consist of the generic part, and two LRMS-
and/or site-specific plugins for this second part.  These plugins need
to write their information in a specific format that is the same for
all LRMSes/schedulers.  For the LRMS state information, see the file
lrmsinfo-generic.txt; for the free slot information, see the file
vomaxjobs-generic.txt.

WORKING EXAMPLES

See the doc directory, where a working set of files is included.
YAIM sets these up for your site by default, if you use YAIM.

DEPLOYMENT

The RPM building takes care of making sure that the scripts can find
the support modules (e.g. lrms.py, EstTT.py).  If you decide to move
these modules somewhere else (relocate the RPM) then you will need to
adjust the paths in the script.

Aside from that the dynamic-scheduler plugin needs a configuration
file, which is described below.  Finally, one needs to drop the ERT plugin
(lcg-info-dynamic-scheduler-wrapper) into the GIP plugin directory, 
typically

  /opt/lcg/var/gip/plugin

CONFIG FILE

The format of the config file is like a Windows INI file; for the
exact format, see the documentation for the ConfigParser module in the
Python standard library.  Concisely it looks like this:

[Main]
static_ldif_file : value   # required
vomap :                    # required
  unixgroup1 : vo1
  unixgroup2 : vo2
  [ ... ]

[LRMS]
lrms_backend_cmd : value  # required

[Scheduler]
vo_max_jobs_cmd : value # optional
cycle_time  : value   # optional

Here is an explanation of each of the options.

[Main] static_ldif_file

Set to the full pathname of the static LDIF file used by the GIP
system.  It is used to find out which CEs are in the system, and which
VOs are supported by which CE.  You can find this in a normal
deployment under

  /opt/lcg/var/gip/ldif/static-file-CE.ldif

[Main] vomap

On many systems, control of access to the LRMS queues is done by unix
group names, associating a single unix group to a set of pool accounts
belonging to a particular VO.  Sometimes the unix group names are not
the same as the VO names, "VO name" here being the thing that appears
in the GlueCEAccessControlBaseRule lines.  The vomap is how
information coming from the LRMS backends is modified so that the ERT
and free-slot computations can be done using VO names found in the
static LDIF file, rather than having to make site-specific
customizations to either the LRMS backend plugins or to the ERT
scripts.

Specifically, if all relevant unix group names are identical to the
external VO names (as published in the CEAccessControlBaseRules),
then the vomap section is not needed.  The only entries that are
needed here are those for which the external VO name is different
than the unix group name.  Note that it is allowed to map several
unix group names onto a single "external" name, but each internal
group name must map to at most one external name.

The format looks like

vomap:
   atlsgm : atlas
   biome  : biomed
   LSF_lhcb : lhcb

The leading spaces are significant, they signify that each line is a
continuation of the 'vomap' parameter.  The left hand of each line is
the unix group name, the right hand is the external VO name found in the
GlueCEAccessControlBaseRule field.  If the unix group name and VO name
are the same, there is no need to include it in the vomap construct.
NOTE I have yet to test what happens if vomap is empty.

New in release 2.0 : the vomap can also map unix groups to VOMS FQANs,
like

  aliceprd : /alice/Role=production

which maps the unix group 'aliceprd' to the listed VOMS FQAN.  If
there is a VOView for this FQAN in the static_ldif_file, the relevant
info will be printed for jobs belonging to this unix group.

[LRMS] lrms_backend_cmd

Set this to a string that will run the command producing the queue
state information (first part of the lrms-specific part).  This could
either be a real command like

  /path/to/cmd/lrmsinfo-pbs -c /other/path/to/config_file_if_needed

or it could simply write the contents of some file to standard output
if you choose to generate the queue state information by a mechanism
other than the GIP subsystem:

  cat /place/where/you/find/text_file_containing_the_generic_lrms_state.info

[Scheduler] vo_max_jobs_cmd

This is the same kind of critter as [LRMS] lrms_backend_cmd, except it
will provide for each unix group known to the LRMS, the maximum number
of job slots this group can take.  Mapping from group to VO is handled
by the vomap parameter.  If a VO has no cap on the number of jobs, it
can be left out.  This entire command is optional, in which case the
free slots will only be limited by the number of free CPUs (or will be
set to zero when jobs for a VO are in a waiting state, in which case
there are obviously no free slots at the moment).

[Scheduler] cycle_time

There are various pieces of information that only make sense modulo
the scheduler cycle time.  For example if the ERT prediction is
'zero', it is reset to half the scheduler cycle since the job will
almost never be run immediately, unless it happens to have been
submitted one second before the scheduling cycle starts.  Set this
parameter to the length of your scheduling cycle.  It is optional.
NOTE I have yet to test what happens if you leave this out.

LIMITATIONS

The program assumes that the structure of the static LDIF file is always
such that each block (corresponding to a given DN) is separated from the
next one by a blank line.  If this is not the case, the parser will fail.

The program assumes that dn's it needs to parse are constructed
like

dn: GlueCEUniqueID=lxb...

where the line begins in the first column, and there is exactly one
space between the colon and the 'G' character.

For the purposes of VOView scheduling information, the plugin only
looks at the last AccessControlBaseRule in each VOView.  The assumption
here is that the only reason that two ACBRs would be present in a single VOView
is if they mapped to the same underlying group in the batch system, in which
case it does not matter which one is used.  The last one happens to be easier
for parsing purposes.
1	FOR THE IMPATIENT
2
3	Just installing the RPM should result in a working system.
4
5	You may want to look at the "vomap" section below if your
6	unix group names are not the same as your VO names.
7
8	OK THE REAL DOCS
9
10	This is an implementation of per-VO estimated response times and free
11	slots for the LCG 'generic information provider' framework.
12
13	HOW IT IS SUPPOSED TO WORK
14
15	There are two parts to the system.
16
17	One part (lcg-dynamic-info-scheduler program) contains the algorithm
18	that computes the response times. This part doesn't know the details
19	of the underlying batch system, so that the estimated times are as
20	independent as possible from the various LRMS.
21
22	The second part is the LRMS specific part. This gathers information
23	from the LRMS and writes it out in a LRMS-independent format. There
24	are two of these critters; one for the LRMS state, and one for the
25	scheduling policy.
26
27	A typical deployment will consist of the generic part, and two LRMS-
28	and/or site-specific plugins for this second part. These plugins need
29	to write their information in a specific format that is the same for
30	all LRMSes/schedulers. For the LRMS state information, see the file
31	lrmsinfo-generic.txt; for the free slot information, see the file
32	vomaxjobs-generic.txt.
33
34	WORKING EXAMPLES
35
36	See the doc directory, where a working set of files is included.
37	YAIM sets these up for your site by default, if you use YAIM.
38
39	DEPLOYMENT
40
41	The RPM building takes care of making sure that the scripts can find
42	the support modules (e.g. lrms.py, EstTT.py). If you decide to move
43	these modules somewhere else (relocate the RPM) then you will need to
44	adjust the paths in the script.
45
46	Aside from that the dynamic-scheduler plugin needs a configuration
47	file, which is described below. Finally, one needs to drop the ERT plugin
48	(lcg-info-dynamic-scheduler-wrapper) into the GIP plugin directory,
49	typically
50
51	/opt/lcg/var/gip/plugin
52
53	CONFIG FILE
54
55	The format of the config file is like a Windows INI file; for the
56	exact format, see the documentation for the ConfigParser module in the
57	Python standard library. Concisely it looks like this:
58
59	[Main]
60	static_ldif_file : value # required
61	vomap : # required
62	unixgroup1 : vo1
63	unixgroup2 : vo2
64	[ ... ]
65
66	[LRMS]
67	lrms_backend_cmd : value # required
68
69	[Scheduler]
70	vo_max_jobs_cmd : value # optional
71	cycle_time : value # optional
72
73	Here is an explanation of each of the options.
74
75	[Main] static_ldif_file
76
77	Set to the full pathname of the static LDIF file used by the GIP
78	system. It is used to find out which CEs are in the system, and which
79	VOs are supported by which CE. You can find this in a normal
80	deployment under
81
82	/opt/lcg/var/gip/ldif/static-file-CE.ldif
83
84	[Main] vomap
85
86	On many systems, control of access to the LRMS queues is done by unix
87	group names, associating a single unix group to a set of pool accounts
88	belonging to a particular VO. Sometimes the unix group names are not
89	the same as the VO names, "VO name" here being the thing that appears
90	in the GlueCEAccessControlBaseRule lines. The vomap is how
91	information coming from the LRMS backends is modified so that the ERT
92	and free-slot computations can be done using VO names found in the
93	static LDIF file, rather than having to make site-specific
94	customizations to either the LRMS backend plugins or to the ERT
95	scripts.
96
97	Specifically, if all relevant unix group names are identical to the
98	external VO names (as published in the CEAccessControlBaseRules),
99	then the vomap section is not needed. The only entries that are
100	needed here are those for which the external VO name is different
101	than the unix group name. Note that it is allowed to map several
102	unix group names onto a single "external" name, but each internal
103	group name must map to at most one external name.
104
105	The format looks like
106
107	vomap:
108	atlsgm : atlas
109	biome : biomed
110	LSF_lhcb : lhcb
111
112	The leading spaces are significant, they signify that each line is a
113	continuation of the 'vomap' parameter. The left hand of each line is
114	the unix group name, the right hand is the external VO name found in the
115	GlueCEAccessControlBaseRule field. If the unix group name and VO name
116	are the same, there is no need to include it in the vomap construct.
117	NOTE I have yet to test what happens if vomap is empty.
118
119	New in release 2.0 : the vomap can also map unix groups to VOMS FQANs,
120	like
121
122	aliceprd : /alice/Role=production
123
124	which maps the unix group 'aliceprd' to the listed VOMS FQAN. If
125	there is a VOView for this FQAN in the static_ldif_file, the relevant
126	info will be printed for jobs belonging to this unix group.
127
128	[LRMS] lrms_backend_cmd
129
130	Set this to a string that will run the command producing the queue
131	state information (first part of the lrms-specific part). This could
132	either be a real command like
133
134	/path/to/cmd/lrmsinfo-pbs -c /other/path/to/config_file_if_needed
135
136	or it could simply write the contents of some file to standard output
137	if you choose to generate the queue state information by a mechanism
138	other than the GIP subsystem:
139
140	cat /place/where/you/find/text_file_containing_the_generic_lrms_state.info
141
142	[Scheduler] vo_max_jobs_cmd
143
144	This is the same kind of critter as [LRMS] lrms_backend_cmd, except it
145	will provide for each unix group known to the LRMS, the maximum number
146	of job slots this group can take. Mapping from group to VO is handled
147	by the vomap parameter. If a VO has no cap on the number of jobs, it
148	can be left out. This entire command is optional, in which case the
149	free slots will only be limited by the number of free CPUs (or will be
150	set to zero when jobs for a VO are in a waiting state, in which case
151	there are obviously no free slots at the moment).
152
153	[Scheduler] cycle_time
154
155	There are various pieces of information that only make sense modulo
156	the scheduler cycle time. For example if the ERT prediction is
157	'zero', it is reset to half the scheduler cycle since the job will
158	almost never be run immediately, unless it happens to have been
159	submitted one second before the scheduling cycle starts. Set this
160	parameter to the length of your scheduling cycle. It is optional.
161	NOTE I have yet to test what happens if you leave this out.
162
163	LIMITATIONS
164
165	The program assumes that the structure of the static LDIF file is always
166	such that each block (corresponding to a given DN) is separated from the
167	next one by a blank line. If this is not the case, the parser will fail.
168
169	The program assumes that dn's it needs to parse are constructed
170	like
171
172	dn: GlueCEUniqueID=lxb...
173
174	where the line begins in the first column, and there is exactly one
175	space between the colon and the 'G' character.
176
177	For the purposes of VOView scheduling information, the plugin only
178	looks at the last AccessControlBaseRule in each VOView. The assumption
179	here is that the only reason that two ACBRs would be present in a single VOView
180	is if they mapped to the same underlying group in the batch system, in which
181	case it does not matter which one is used. The last one happens to be easier
182	for parsing purposes.
grid.support@nikhef.nl	ViewVC Help
Powered by ViewVC 1.1.28