next up previous contents index
Next: 4.2 Condor's Checkpoint Mechanism Up: 4. Miscellaneous Concepts Previous: 4. Miscellaneous Concepts   Contents   Index

Subsections


4.1 Condor's ClassAd Mechanism

ClassAds are a flexible mechanism for representing the characteristics and constraints of machines and jobs in the Condor system. ClassAds are used extensively in the Condor system to represent jobs, resources, submitters and other Condor daemons. An understanding of this mechanism is required to harness the full flexibility of the Condor system.

A ClassAd is is a set of uniquely named expressions. Each named expression is called an attribute. Figure 4.1 shows an example of a ClassAd with ten attributes.

Figure 4.1: An example ClassAd
\begin{figure}\footnotesize
\begin{verbatim}MyType = ''Machine''
TargetType = ...
...\vert LoadAvg<=0.3 && KeyboardIdle>15*60\end{verbatim}
\normalsize\end{figure}

ClassAd expressions look very much like expressions in C, and are composed of literals and attribute references composed with operators and functions. The difference between ClassAd expressions and C expressions arise from the fact that ClassAd expressions operate in a much more dynamic environment. For example, an expression from a machine's ClassAd may refer to an attribute in a job's ClassAd, such as TARGET.Owner in the above example. The value and type of the attribute is not known until the expression is evaluated in an environment which pairs a specific job ClassAd with the machine ClassAd.

ClassAd expressions handle these uncertainties by defining all operators to be total operators, which means that they have well defined behavior regardless of supplied operands. This functionality is provided through two distinguished values, UNDEFINED and ERROR, and defining all operators so that they can operate on all possible values in the ClassAd system. For example, the multiplication operator which usually only operates on numbers, has a well defined behavior if supplied with values which are not meaningful to multiply. Thus, the expression 10 * "A string" evaluates to the value ERROR. Most operators are strict with respect to ERROR, which means that they evaluate to ERROR if any of their operands are ERROR. Similarly, most operators are strict with respect to UNDEFINED.


4.1.1 Syntax

ClassAd expressions are formed by composing literals, attribute references and other sub-expressions with operators and functions.


4.1.1.1 Literals

Literals in the ClassAd language may be of integer, real, string, undefined or error types. The syntax of these literals is as follows:
Integer
A sequence of continuous digits (i.e., [0-9]). Additionally, the keywords TRUE and FALSE (case insensitive) are syntactic representations of the integers 1 and 0 respectively.

Real
Two sequences of continuous digits separated by a period (i.e., [0-9]+.[0-9]+).

String
A double quote character, followed by an list of characters terminated by a double quote character. A backslash character inside the string causes the following character to be considered as part of the string, irrespective of what that character is.

Undefined
The keyword UNDEFINED (case insensitive) represents the UNDEFINED value.

Error
The keyword ERROR (case insensitive) represents the ERROR value.


4.1.1.2 Attributes

Every expression in a ClassAd is named by an attribute name. Together, the (name,expression) pair is called an attribute. An attributes may be referred to in other expressions through its attribute name.

Attribute names are sequences of alphabetic characters, digits and underscores, and may not begin with a digit. All characters in the name are significant, but case is not significant. Thus, Memory, memory and MeMoRy all refer to the same attribute.

An attribute reference consists of the name of the attribute being referenced, and an optional scope resolution prefix. The prefixes that may be used are MY. and TARGET.. The case used for these prefixes is not significant. The semantics of supplying a prefix are discussed in Section 4.1.2.


4.1.1.3 Operators

The operators that may be used in ClassAd expressions are similar to those available in C. The available operators and their relative precedence is shown in figure 4.2.
Figure 4.2: Relative precedence of ClassAd expression operators
\begin{figure}\begin{verbatim}- (unary negation) (high precedence)
* /
+ -...
... >= >
== != =?= =!=
&&
\vert\vert (low precedence)\end{verbatim}
\end{figure}
The operator with the highest precedence is the unary minus operator. The only operators which are unfamiliar are the =?= and =!= operators, which are discussed in Section 4.1.2.


4.1.1.4 Predefined Functions

Any ClassAd expression may utilize predefined functions. Function names are case insensitive. Parameters to functions and a return value from a function may be typed (as given) or not. Nested or recursive function calls are allowed.

Here are descriptions of each of these predefined functions. The possible types are the same as itemized in in Section 4.1.1. Where the type may be any of these literal types, it is called out as AnyType. Where the type is Integer, but only returns the value 1 or 0 (implying True or False), it is called out as Boolean. The format of each function is given as

ReturnType FunctionName(ParameterType parameter1, ParameterType parameter2, ...)
Optional parameters are given within square brackets.

AnyType ifThenElse(AnyType IfExpr,AnyType ThenExpr, AnyType ElseExpr)
A conditional expression is described by IfExpr. The following defines return values, when IfExpr evaluates to Where IfExpr evaluates to give a value of type String, the function returns the value ERROR. The implementation uses lazy evaluation, so expressions are only evaluated as defined.

This function returns ERROR if other than exactly 3 arguments are given.

Boolean isUndefined(AnyType Expr)
Returns True, if Expr evaluates to UNDEFINED. Returns False in all other cases.

This function returns ERROR if other than exactly 1 argument is given.

Boolean isError(AnyType Expr)
Returns True, if Expr evaluates to ERROR. Returns False in all other cases.

This function returns ERROR if other than exactly 1 argument is given.

Boolean isString(AnyType Expr)
Returns True, if the evaluation of Expr gives a value of type String. Returns False in all other cases.

This function returns ERROR if other than exactly 1 argument is given.

Boolean isInteger(AnyType Expr)
Returns True, if the evaluation of Expr gives a value of type Integer. Returns False in all other cases.

This function returns ERROR if other than exactly 1 argument is given.

Boolean isReal(AnyType Expr)
Returns True, if the evaluation of Expr gives a value of type Real. Returns False in all other cases.

This function returns ERROR if other than exactly 1 argument is given.

Boolean isBoolean(AnyType Expr)
Returns True, if the evaluation of Expr gives the integer value 0 or 1. Returns False in all other cases.

This function returns ERROR if other than exactly 1 argument is given.

Integer int(AnyType Expr)
Returns the integer value as defined by Expr. Where the type of the evaluated Expr is Real, the value is truncated (round towards zero) to an integer. Where the type of the evaluated Expr is String, the string is converted to an integer using a C-like atoi() function. When this result is not an integer, ERROR is returned. Where the evaluated Expr is ERROR or UNDEFINED, ERROR is returned.

This function returns ERROR if other than exactly 1 argument is given.

Real real(AnyType Expr)
Returns the real value as defined by Expr. Where the type of the evaluated Expr is Integer, the return value is the converted integer. Where the type of the evaluated Expr is String, the string is converted to a real value using a C-like atof() function. When this result is not a real, ERROR is returned. Where the evaluated Expr is ERROR or UNDEFINED, ERROR is returned.

This function returns ERROR if other than exactly 1 argument is given.

String string(AnyType Expr)
Returns the string that results from the evaluation of Expr. Converts a non-string value to a string. Where the evaluated Expr is ERROR or UNDEFINED, ERROR is returned.

This function returns ERROR if other than exactly 1 argument is given.

Integer floor(AnyType Expr)
Returns the integer that results from the evaluation of Expr, where the type of the evaluated Expr is Integer. Where the type of the evaluated Expr is not Integer, function real(Expr) is called. Its return value is then used to return the largest magnitude integer that is not larger than the returned value. Where real(Expr) returns ERROR or UNDEFINED, ERROR is returned.

This function returns ERROR if other than exactly 1 argument is given.

Integer ceiling(AnyType Expr)
Returns the integer that results from the evaluation of Expr, where the type of the evaluated Expr is Integer. Where the type of the evaluated Expr is not Integer, function real(Expr) is called. Its return value is then used to return the smallest magnitude integer that is not less than the returned value. Where real(Expr) returns ERROR or UNDEFINED, ERROR is returned.

This function returns ERROR if other than exactly 1 argument is given.

Integer round(AnyType Expr)
Returns the integer that results from the evaluation of Expr, where the type of the evaluated Expr is Integer. Where the type of the evaluated Expr is not Integer, function real(Expr) is called. Its return value is then used to return the integer that results from a round-to-nearest rounding method. The nearest integer value to the return value is returned, except in the case of the value at the exact midpoint between two integer values. In this case, the even valued integer is returned. Where real(Expr) returns ERROR or UNDEFINED, or the integer value does not fit into 32 bits, ERROR is returned.

This function returns ERROR if other than exactly 1 argument is given.

Integer random([ AnyType Expr ])
Where the optional argument Expr evaluates to type Integer or type Real (and called x), the return value is the integer or real r randomly chosen from the interval 0 <= r < x. With no argument, the return value is chosen with random(1.0). Returns ERROR in all other cases.

This function returns ERROR if greater than 1 argument is given.

String strcat(AnyType Expr1 [ , AnyType Expr2 ... ])
Returns the string which is the concatenation of all arguments, where all arguments are converted to type String by function string(Expr). Returns ERROR if any argument evaluates to UNDEFINED or ERROR.

String substr(String s, Integer offset [ , Integer length ])
Returns the substring of s, from the position indicated by offset, with (optional) length characters. The first character within s is at offset 0. If the optional length argument is not present, the substring extends to the end of the string. If offset is negative, the value (length - offset) is used for the offset. If length is negative, an initial substring is computed, from the offset to the end of the string. Then, the absolute value of length characters are deleted from the right end of the initial substring. Further, where characters of this resulting substring lie outside the original string, the part that lies within the original string is returned. If the substring lies completely outside of the original string, the null string is returned.

This function returns ERROR if greater than 3 or less than 2 arguments are given.

Integer strcmp(AnyType Expr1, AnyType Expr2)
Both arguments are converted to type String by function string(Expr). The return value is an integer that will be Case is significant in the comparison. Where either argument evaluates to ERROR or UNDEFINED, ERROR is returned.

This function returns ERROR if other than 2 arguments are given.

Integer stricmp(AnyType Expr1, AnyType Expr2)
This function is the same as strcmp, except that letter case is not significant.

String toUpper(AnyType Expr)
The single argument is converted to type String by function string(Expr). The return value is this string, with all lower case letters converted to upper case. If the argument evaluates to ERROR or UNDEFINED, ERROR is returned.

This function returns ERROR if greater than 1 argument is given.

String toLower(AnyType Expr)
The single argument is converted to type String by function string(Expr). The return value is this string, with all upper case letters converted to lower case. If the argument evaluates to ERROR or UNDEFINED, ERROR is returned.

This function returns ERROR if other than exactly 1 argument is given.

Integer size(AnyType Expr)
Returns the number of characters in the string, after calling function string(Expr). If the argument evaluates to ERROR or UNDEFINED, ERROR is returned.

This function returns ERROR if other than exactly 1 argument is given.

For the following functions, a delimiter is represented by a string. Each character within the delimiter string delimits individual strings within a list of strings that is given by a single string. The default delimiter contains the comma and space characters. A string within the list is ended (delimited) by one or more characters within the delimiter string.

Integer stringListSize(String list [ , String delimiter ])
Returns the number of elements in the string list, as delimited by the optional delimiter string. Returns ERROR if either argument is not a string.

This function returns ERROR if other than 1 or 2 arguments are given.

Integer stringListSum(String list [ , String delimiter ])
OR Real stringListSum(String list [ , String delimiter ])
Sums and returns the sum of all items in the string list, as delimited by the optional delimiter string. If all items in the list are integers, the return value is also an integer. If any item in the list is a real value (noninteger), the return value is a real. If any item does not represent an integer or real value, the return value is ERROR.

Real stringListAve(String list [ , String delimiter ])
Sums and returns the real-valued average of all items in the string list, as delimited by the optional delimiter string. If any item does not represent an integer or real value, the return value is ERROR. A list with 0 items (the empty list) returns the value 0.0.

Integer stringListMin(String list [ , String delimiter ])
OR Real stringListMin(String list [ , String delimiter ])
Finds and returns the minimum value from all items in the string list, as delimited by the optional delimiter string. If all items in the list are integers, the return value is also an integer. If any item in the list is a real value (noninteger), the return value is a real. If any item does not represent an integer or real value, the return value is ERROR. A list with 0 items (the empty list) returns the value UNDEFINED.

Integer stringListMax(String list [ , String delimiter ])
OR Real stringListMax(String list [ , String delimiter ])
Finds and returns the maximum value from all items in the string list, as delimited by the optional delimiter string. If all items in the list are integers, the return value is also an integer. If any item in the list is a real value (noninteger), the return value is a real. If any item does not represent an integer or real value, the return value is ERROR. A list with 0 items (the empty list) returns the value UNDEFINED.

Boolean stringListMember(String x, String list [ , String delimiter ])
Returns TRUE if item x is in the string list, as delimited by the optional delimiter string. Returns FALSE if item x is not in the string list. Comparison is done with strcmp(). The return value is ERROR, if any of the arguments are not strings.

Boolean stringListIMember(String x, String list [ , String delimiter ])
Same as stringListMember(), but comparison is done with stricmp(), so letter case is not relevant.

The following three functions utilize regular expressions as defined and supported by the PCRE library. See http://www.pcre.org for complete documentation of regular expressions.

The options argument to these functions is a string of special characters that modify the use of the regular expressions. Inclusion of characters other than these as options are ignored.

I or i
Ignore letter case.
M or m
Modifies the interpretation of the carat (^) and dollar sign ($) characters. The carat character matches the start of a string, as well as after each newline character. The dollar sign character matches before a newline character.
S or s
The period matches any character, including the newline character.
X or x
Ignore both white space and comments within the pattern. A comment is defined by starting with the pound sign (#) character, and continuing until the newline character.

Boolean regexp(String pattern, String target [ , String options ])
Returns TRUE if the string target is a regular expression as described by pattern. Returns FALSE otherwise. If any argument is not a string, or if pattern does not describe a valid regular expression, returns ERROR.

String regexps(String pattern, String target, String substitute, [ String options ])
The regular expression pattern is applied to target. If the string target is a regular expression as described by pattern, the string substitute is returned, with backslash expansion performed. The return value is ERROR, if any of the arguments are not strings.

Boolean stringListRegexpMember(String pattern, String list [ , String delimiter ] [ , String options ])
Returns TRUE if any of the strings within the list is a regular expression as described by pattern. Returns FALSE otherwise. If any argument is not a string, or if pattern does not describe a valid regular expression, returns ERROR. To include the fourth (optional) argument options, a third argument of delimiter is required. A default value for a delimiter is " ,".

Integer time()
Returns the current coordinated universal time, which is the same as the ClassAd attribute CurrentTime. This is the time, in seconds, since midnight of January 1, 1970.

String interval(Integer seconds)
Uses seconds to return a string of the form days+hh:mm:ss. This represents an interval of time. Leading values that are zero are omitted from the string. For example, seconds of 67 becomes "1:07". A second example, seconds of 1472523 = 17*24*60*60 + 1*60*60 + 2*60 + 3, results in the string "17+1:02:03".


4.1.2 Evaluation Semantics

The ClassAd mechanism's primary purpose is for matching entities that supply constraints on candidate matches. The mechanism is therefore defined to carry out expression evaluations in the context of two ClassAds that are testing each other for a potential match. For example, the condor_ negotiator evaluates the Requirements expressions of machine and job ClassAds to test if they can be matched. The semantics of evaluating such constraints is defined below.

4.1.2.1 Literals

Literals are self-evaluating, Thus, integer, string, real, undefined and error values evaluate to themselves.


4.1.2.2 Attribute References

Since the expression evaluation is being carried out in the context of two ClassAds, there is a potential for name space ambiguities. The following rules define the semantics of attribute references made by ad A that is being evaluated in a context with another ad B:
  1. If the reference is prefixed by a scope resolution prefix,

  2. If the reference is not prefixed by a scope resolution prefix,

  3. Finally, if the reference refers to an expression that is itself in the process of being evaluated, there is a circular dependency in the evaluation. The value of the reference is ERROR.


4.1.2.3 Operators

All operators in the ClassAd language are total, and thus have well defined behavior regardless of the supplied operands. Furthermore, most operators are strict with respect to ERROR and UNDEFINED, and thus evaluate to ERROR (or UNDEFINED) if either of their operands have these exceptional values.

4.1.3 ClassAds in the Condor System

The simplicity and flexibility of ClassAds is heavily exploited in the Condor system. ClassAds are not only used to represent machines and jobs in the Condor pool, but also other entities that exist in the pool such as checkpoint servers, submitters of jobs and master daemons. Since arbitrary expressions may be supplied and evaluated over these ads, users have a uniform and powerful mechanism to specify constraints over these ads. These constraints can take the form of Requirements expressions in resource and job ads, or queries over other ads.


4.1.3.1 Constraints and Preferences

The requirements and rank expressions within the submit description file are the mechanism by which users specify the constraints and preferences of jobs. For machines, the configuration determines both constraints and preferences of the machines.

For both machine and job, the rank expression specifies the desirability of the match (where higher numbers mean better matches). For example, a job ad may contain the following expressions:

Requirements = Arch=="SUN4u" && OpSys == "SOLARIS251"
Rank         = TARGET.Memory + TARGET.Mips
In this case, the job requires an UltraSparc computer running the Solaris 2.5.1 operating system. Among all such computers, the customer prefers those with large physical memories and high MIPS ratings. Since the Rank is a user-specified metric, any expression may be used to specify the perceived desirability of the match. The condor_ negotiator daemon runs algorithms to deliver the best resource (as defined by the rank expression) while satisfying other required criteria.

Similarly, the machine may place constraints and preferences on the jobs that it will run by setting the machine's configuration. For example,

    Friend        = Owner == "tannenba" || Owner == "wright"
    ResearchGroup = Owner == "jbasney" || Owner == "raman"
    Trusted       = Owner != "rival" && Owner != "riffraff"
    START         = Trusted && ( ResearchGroup || LoadAvg < 0.3 &&
                         KeyboardIdle > 15*60 )
    RANK          = Friend + ResearchGroup*10

The above policy states that the computer will never run jobs owned by users rival and riffraff, while the computer will always run a job submitted by members of the research group. Furthermore, jobs submitted by friends are preferred to other foreign jobs, and jobs submitted by the research group are preferred to jobs submitted by friends.

Note: Because of the dynamic nature of ClassAd expressions, there is no a priori notion of an integer-valued expression, a real-valued expression, etc. However, it is intuitive to think of the Requirements and Rank expressions as integer-valued and real-valued expressions, respectively. If the actual type of the expression is not of the expected type, the value is assumed to be zero.

4.1.3.2 Querying with ClassAd Expressions

The flexibility of this system may also be used when querying ClassAds through the condor_ status and condor_ q tools which allow users to supply ClassAd constraint expressions from the command line.

For example, to find all computers which have had their keyboards idle for more than 20 minutes and have more than 100 MB of memory:

% condor_status -const 'KeyboardIdle > 20*60 && Memory > 100'

Name       Arch     OpSys        State      Activity   LoadAv Mem  ActvtyTime

amul.cs.wi SUN4u    SOLARIS251   Claimed    Busy       1.000  128   0+03:45:01
aura.cs.wi SUN4u    SOLARIS251   Claimed    Busy       1.000  128   0+00:15:01
balder.cs. INTEL    SOLARIS251   Claimed    Busy       1.000  1024  0+01:05:00
beatrice.c INTEL    SOLARIS251   Claimed    Busy       1.000  128   0+01:30:02
...
...
                     Machines Owner Claimed Unclaimed Matched Preempting

    SUN4u/SOLARIS251        3     0       3         0       0          0
    INTEL/SOLARIS251       21     0      21         0       0          0
    SUN4x/SOLARIS251        3     0       3         0       0          0
       INTEL/WINNT51        1     0       0         1       0          0
         INTEL/LINUX        1     0       1         0       0          0

               Total       29     0      28         1       0          0

Here is an example that utilizes a regular expression ClassAd function to list specific information. A file contains ClassAd information. condor_ advertise is used to inject this information, and condor_ status constrains the search with an expression that contains a ClassAd function.

% cat ad
MyType = "Generic"
FauxType = "DBMS"
Name = "random-test"
Machine = "f05.cs.wisc.edu"
MyAddress = "<128.105.149.105:34000>"
DaemonStartTime = 1153192799
UpdateSequenceNumber = 1

% condor_advertise UPDATE_AD_GENERIC ad

% condor_status -any -constraint 'FauxType=="DBMS" && regexp("random.*", Name, "i")'

MyType               TargetType           Name                          

Generic              None                 random-test

Similar flexibility exists in querying job queues in the Condor system.


next up previous contents index
Next: 4.2 Condor's Checkpoint Mechanism Up: 4. Miscellaneous Concepts Previous: 4. Miscellaneous Concepts   Contents   Index
condor-admin@cs.wisc.edu