Competition of Distributed and Multiagent Planners (CoDMAP)

The CoDMAP competition (of the DMAP 2015 Workshop) is meant to be a preliminary version of possible future Multiagent Planning (MAP) track at the IPC (International Planning Competition). The competition focuses on deterministic (STRIPS-compatible), domain-independent, offline planners for cooperative agents. For maximal openness to various multiagent planners, CoDMAP consists of two tracks (see Figure 1): centralized (compatibility track) and distributed (track towards IPC MAP). You can choose which track you will be competing in. If your planner is suitable for the distributed track, it can automatically compete in the centralized track as well, so we kindly suggest to submit it to both tracks.

Figure 1. Architecture of planners in various competition tracks, including the centralized and distributed tracks of CoDMAP. P₁, ..., P_n represent processes of the planner. In centralized CoDMAP, there can be arbitrary number of planner processes (e.g., only P₁ for centralized planners). In distributed CoDMAP, the processes correspond to n agents α₁, ..., α_n running on machines M₁, ..., M_n. Communication among the planner's processes and (shared) memory are denoted as comm and mem respectively. The output plan(s) are ordered lists of k actions a₁, ..., a_k, where in the distributed CoDMAP, an agent's α_i action is denoted as a^α_i. In the case of centralized CoDMAP, the the planner can arbitrarily use unfactored and/or factored input.

Results

See the regular CoDMAP results.

See the additional (Summer Run – September 2015) CoDMAP results.

Rules & Evaluation

General CoDMAP requirements:

• deterministic non-durative model (STRIPS-compatible)
• domain-independent
• cooperative agents (only common and public goals; common metrics; no joint-actions)
• evaluation metrics: coverage of solved problems, planning speed, and plan cost (satisficing planning)
• offline (as in IPC Deterministic Track: input, planning, output (plan), validation)
• source codes of the planner are publishable (at least) from June 7th, 2015 (we accept planners with closed sources as well, but such will be treated specially in the final results)
• the planner should run from command line on *nix-based system (the competition will run on Ubuntu)
• min 2, max 4 pages long descriptive paper in AAAI format has to be submitted for each planner (prospectively describing all submitted configurations)

Centralized CoDMAP Track

motivation:

highly compatible “transitional” track for various multiagent planners

planner architecture:

centralized (monolithic, single-core, ...), parallel or distributed

input:

centralized (global description of one planning problem instance per a planner)

• domain & problem as MA-PDDL with unfactored-privacy (if necessary, custom factorization can be used instead)
• public/private separation in MA-PDDL by MA-STRIPS definition (if necessary, custom separation can be used instead)
• input files: domain.pddl, problem.pddl

alternative input:

distributed (local factored descriptions of one planning problem instance per agents)

• domain & problem as MA-PDDL with factored-privacy (if necessary, custom agent factorization can be used instead)
• public/private separation in MA-PDDL by MA-STRIPS definition (if necessary, custom separation can be used instead)
• either centralized or distributed input variant may be used
• input files: domain-agent_1.pddl, problem-agent_1.pddl, ..., domain-agent_n.pddl, problem-agent_n.pddl

output:

has to be a sound linear plan

soundness validation:

IPC (centralized) validator

plan format:

IPC (VAL compatible)

One action per line: (action-name param1 param2 ...)

Comments can be added after ; to the end of the line

running:

30 minutes limit per planner per problem; one planner on one machine (4cores, 8GB RAM - operational system which fits under 1GB)

communication:

any

evaluation:

• coverage
• IPC score over runtime T (sum over all problems T*/T, where T* is runtime of the fastest planner)
• IPC score over cost (sum over all problems Q*/Q, see IPC rules, evaluation scheme, satisficing track)
• note on how the planner treats privacy, factorization, whether the planner is complete and/or optimal and what is its architecture

the planner description paper has to answer:

• Is the planner complete?
• Is the planner optimal?
• Is the agent factorization presented in the MA-PDDL files used?
• Is the private/public separation presented in the MA-PDDL files used?
• Is the planner using MA-STRIPS private/public separation? (either from MA-PDDL or done by the planner)
• What private information (or its derivative), in what form, and how is communicated in the planner?
• What is the architecture of the planner (centralized or distributed; single or multi-threaded)?

Distributed CoDMAP Track

motivation:

“experimental” proper track for distributed MA planners

planner architecture:

distributed

input:

distributed (local factored descriptions of one planning problem instance per agents)

• domain & problem as MA-PDDL with factored-privacy (the agent factorization has to follow the definition in MA-PDDL)
• public/private separation in MA-PDDL (the separation has to follow the MA-PDDL, it is not necessarily MA-STRIPS)
• input files: domain-agent_1.pddl, problem-agent_1.pddl, …, domain-agent_n.pddl, problem-agent_n.pddl
• all private parts of the input has to be kept local on the agent’s machine (NOTE)
  Strictly speaking, this requirement does not forbid you to exchange private information among the agents i.e., processes of your distributed planner. As you can encrypt it for instance, we would have really hard time to detect it. Honestly, we will not even try, but we require to explain such behavior in the description paper. The fact we are mentioning this here is to communicate our intention that private information should be kept as local as possible.
• IP addresses of all agents will be known in advance in an additional input file with format:
  • <agent_1's name> <agent_1's IPv4 address>
  • ...
  • <agent_n's name> <agent_n's IPv4 address>

output:

has to be a sound set of n parallel linear plans for each agent, each action has a global (integer) timestamp, actions at the same timestamp are considered parallel. All actions are considered to have duration of 1 timestep.

soundness validation:

All plans will be concatenated and the concatenated plan checked with VAL.

Actions at the same time point (parallel) must not be mutually exclusive - that is one action deletes effect or precondition of the other.

plan format:

Timestamp and one action per line per agent

After concatenation VAL compatible:

timestamp: (action-name param1 param2 ...)

Example: 10: (load-truck tru1 pkg1 loc1)

Comments can be added after ; to the end of the line

running:

30 minutes limit per planner per problem; one agent on one machine (4cores, 8GB RAM - operational system which fits under 1GB)

communication:

• TCP/IP network (1GBit switch)
• separate subnet
• each agent's planner will be parameterized with name of the agent

evaluation:

• coverage
• IPC score over runtime T (sum over all problems T*/T, where T* is runtime of the fastest planner)
• IPC score over cost (sum over all problems Q*/Q, see IPC rules, evaluation scheme, satisficing track)
• note on how the planner treats privacy and whether it is complete and/or optimal

the planner description paper has to answer:

• Is the planner complete?
• Is the planner optimal?
• What private information (or its derivative) and in what form is communicated among the agents? (e.g., aggregated heuristic estimates based on private actions, obfuscated private actions, etc.)
• What is the architecture of an agent’s planning process (single or multi-threaded)?

Language

The language of the competition will be MA-PDDL (:factored-privacy and :unfactored-privacy versions). MA-PDDL is an extension of PDDL 3.1.

• We will use only the :strips + action costs subset of PDDL
• Action costs may be ignored (will have effect only on the IPC score over costs)
• Privacy is defined over predicates, constants and objects
  • If a public predicate is grounded with only public constants/objects, the resulting fact is public
  • If a public predicate is grounded with at least one private constant/object, the resulting fact is private (in case of :unfactored-privacy grounded predicates can refer to the private constants/objects of only one agent -- mixed groundings are not allowed)
  • If the predicate is private, the grounded fact is always private
    • In general, the same grounded predicate may be private to multiple agents, which means that it is private to a subset of agents. In the competition domains and problems, we will avoid such situations.
  • A (grounded) action is public if it contains a public fact in :precondition or :effect, otherwise it is private
• The privacy definition is more expressive than MA-STRIPS, but we will stick to MA-STRIPS in most domains.
• In general, the type hierarchy of factored domains can be different for each agent (although must be consistent). In the competition domains, we will keep all type hierarchies the same for all factors.

Centralized CoDMAP Track

The centralized track will use the :unfactored-privacy version of MA-PDDL. An agent is any object/constant associated with any grounded action by an :agent <agent> field. All predicates are defined in the domain, private predicates are enclosed in a (:private <agent> ...) expression. All actions are defined in the domain, with additional :agent <agent> field (which can be used also as an parameter to the action). All objects are defined in the problem, private objects are enclosed in a (:private ...) expression. In the domain, the agents are represented by a lifted typet expression, such as (in the action def.) :agent ?airplane - airplane, which means that all objects of type airplane are agents. In the problem, the respective objects or constants are used instead.

Distributed CoDMAP Track

The distributed track will use a single-agent variant of the :factored-privacy version of MA-PDDL. Each agent has its own (single-agent) domain and problem. Only predicates known to the agent are defined in the domain, private predicates are enclosed in a (:private ...) expression. Only actions known to the agent are defined in the domain. The agents can exchange public information about their public actions (aka public projections) at runtime. Only objects known to the agent are defined in the problem, private objects are enclosed in a (:private ...) expression.

• logistics-00 [unfactored] [factored]
• elevators-08 [unfactored] [factored]
• zenotravel [unfactored] [factored] [last update 24.4.2015]
• rovers [unfactored] [factored] [last update 29.4.2015]
• satellites [unfactored] [factored] [last update 29.4.2015]
• blocksworld [unfactored] [factored]
• driverlog [unfactored] [factored]
• depot [unfactored] [factored] [last update 29.4.2015]
• ma-sokoban [unfactored] [factored] [last update 29.4.2015]
• woodworking-08 [unfactored] [factored]
• taxi [unfactored] [factored]
• wireless [unfactored] [factored] [last update 22.5.2015]

Submission

Each planner (in one particular configuration) will be submitted (uploaded) to the testing environment (a specific SSH-accessible directory on our competition server). You can submit one planner in more configurations. To receive the SSH access, please, send us an email with name of your planner+configuration, track your planner will be competing in (centralized, distribued, or both) and optionally a SSH key. In the reply, we will send you a URL and a username for your planner together with a password if the SSH key is not provided.

The testing environment will contain a shell script with unified input parameters which we will use to run your planner, thus you will be responsible to fill in the script to correctly run and parameterize your planner and gather the resulting plan(s). Additionally, you will be able to run another shell script which will emulate running of your planner on selected (or all) domains and problems available (note: two additional domains will be added after submission deadline; for more details, please, refer to the Domains & Problems section). We will not require your source codes till the submission deadline, so you can use only binaries of your planner in the testing environment.

Pre-submission Poll

• 12 MA planners (1 in design phase), 3 optimal
• 11 support PDDL with agent specification (5 external private/public spec.)
• 9 focus on planning time minimization, 8 focuses on coverage
• 6 support distribution with communication over network
• 3 support no privacy, 6 obfuscation/aggregation, 3 (total) privacy
• 11 are *nix-like OS compatible and all run from command line
• C++ (5), Java (4), Python (2), Lisp (2), C# (1), C (1)

Important Dates

• planner submission open: February 23rd, 2015
• domains+problems and testing environment finalized: March 27th, 2015
• planner submission deadline: April 26th, 2015 23:59 GMT-10, April 29th, 2015
• planner papers deadline: May 31st, 2015
• announcement of results: June 7th, 2015 at the ICAPS/DMAP'15 Workshop in Jerusalem Israel

Questions

Organizers

• Michal Štolba (Czech Technical University in Prague, Czech Republic)
• Antonín Komenda (Czech Technical University in Prague, Czech Republic)
• Dániel Laszlo Kovacs (Budapest University of Technology and Economics, Hungary)

Acknowledgments